Commit graph

754 commits

Author SHA1 Message Date
Vincent Ambo
a99e33a107 chore(ops/sourcegraph): Bump to 3.32.1
Change-Id: I8efdf3dbfc5575f24c8e6996a7716d308f1446df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5752
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-28 11:21:27 +00:00
Vincent Ambo
9bff3ae373 fix(tvl-slapd): load argon2 module with new name
This became an "official" module and dropped the `pw-` prefix.

Relates to b/184

Change-Id: I963f83b55b83015b022ab1b8330ea710d2258631
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5751
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-27 23:48:37 +00:00
William Carroll
11a8eea0e3 feat(wpcarro/tarasco): Support tarasco 🇲🇽
Named after the Mexican restaurant, El Tarasco, in El Porto, which I live 3m
walking distance from.

Change-Id: I2cd4b68eaa974ad6c8fec73e0566bc0b831c57a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5743
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
2022-05-27 23:21:25 +00:00
Vincent Ambo
e9e8e38db7 fix(ops/gerrit-tvl): Filter builds by commit hash
The patchsetSha is one of the things passed in to the `fetch()`
interface, and Buildkite's API (now?) supports filtering by the commit
hash in addition.

With this combination, we should not accidentally display builds for
the wrong patch set.

Change-Id: I6bb26dd7387f2dd00291990cadd38629ecda999b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5702
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-27 22:13:58 +00:00
Vincent Ambo
bdccd2c111 fix(ops/modules): Increase RestartSec= of oauth2_proxy service
When Keycloak and oauth2_proxy are restarted simultaneously, the
latter might try to come up (repeatedly!) before Keycloak can serve it
properly.

This leads to systemd considering the unit failed.

Since this all happens in the span of a second or so, slightly
increase the restart delay of the service to ensure it comes back
after Keycloak is ready.

A "proper" fix might be to add a script that runs before the actual
service and waits for Keycloak, but I don't want to prioritise that
right now.

Change-Id: I4dadba686de60ffc103fe889ce19f05ca1d7d4fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5695
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-27 21:10:35 +00:00
Vincent Ambo
772f8f1b90 feat(ops/pipelines): Evaluate depot pipeline in restricted-eval mode
Change-Id: Ic5b98a0777860b68dabb9a9b59e8c682236a71c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4884
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-05-26 16:57:16 +00:00
Vincent Ambo
7a0a4224a5 refactor(ops/nixos): Prepare for restricted eval
Change-Id: I7b5304dda3040830fe90fc188b35da3fd95451a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5682
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-05-26 16:24:34 +00:00
Vincent Ambo
48dfefe40d refactor(sanduny): Prepare for restricted-eval
Change-Id: I83a404dc7dbaf5ca53659d03df4e4de461a9d046
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5688
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-05-26 14:17:33 +00:00
Vincent Ambo
250300f167 refactor(whitby): Prepare for restricted-eval
Change-Id: I7604ca29310d759b0ffee2ffb0048b6365a2894c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5683
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-05-26 14:17:32 +00:00
Vincent Ambo
bc42c5a61b fix(ops/modules): adapt for changed ssh.knownHosts
Somehow this ended up generating an empty file, with this change it is
fine again. I was looking at the recent commits of the module in
nixpkgs but couldn't quite figure it out, there are also some vague
references to the attribute set key being used as a hostname, but this
doesn't seem to be true in practice.

To be clear, the previous code was wrong, but at some point it
generated a file that accidentally worked.

Change-Id: I42d55730c09daafe6d6fe0eb3647135e84737bca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5670
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-05-26 10:05:54 +00:00
Vincent Ambo
e3a31b702a feat(whitby): Deploy private SSH key for build agents
Change-Id: I5b1dfaaf28e835cac5b897e18b015d90ac3b2857
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5665
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2022-05-25 23:53:09 +00:00
Vincent Ambo
77f096771d feat(ops/secrets): Add private SSH key for Buildkite agent(s)
The public key is:

  ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIME13zAw3Fk6qsbWCe6mH2zkxOJ+NmG+FwMjLw00mcWt buildkite@tvl

Change-Id: Ia8591e5df42727e4068f26865d83d0af85424fde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5664
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-25 23:53:09 +00:00
Klemens Nanni
3a53587c2a feat(ops/modules/open_eid.nix): Access all key slots
`onepin-opensc-pkcs11.so` only enables PIN1, but PIN2 is also required.

Change-Id: Ic1c34ca58a46c2978c7e27e7a9b7e6a4d335ac0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5648
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: kn <klemens@posteo.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-25 20:38:11 +00:00
Klemens Nanni
45c46d4a73 feat(ops/modules/open_eid.nix): Add digidoc-tool(1) to PATH
libdigidocpp is a dependency of qdigidoc4(1) already.

This will need https://github.com/NixOS/nixpkgs/pull/174055
"libdigidocpp: Fix PKCS11 module library path" to work, though.

Change-Id: Ic8d671077977b1d1f099a8b4b23cc537b52aa954
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5647
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-25 20:37:53 +00:00
sterni
03d1986316 feat(3p/agenix): update to 2022-05-16 and add to niv
The new version brings the new secretsDir setting which means we no
longer have to hardcode /run/agenix everywhere.

Change-Id: I4b579d7233d315a780d7671869d5d06722d769fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5646
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: sterni <sternenseemann@systemli.org>
2022-05-25 15:00:37 +00:00
James Landrein
dc346bb8cf feat(ops/users): add j4m3s
I've only been a couple months lurking in the IRC ...

Change-Id: Idebf96d3bf1124f0a97e11e0f854e8c6d4be8d8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5662
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-05-24 23:39:44 +00:00
Vincent Ambo
f31edeec1b refactor(nixery): Modernise structure of binaries
Nixery is going to gain a new binary (used for building images without
a registry server); to prepare for this the server binary has moved to
cmd/server and the Nix build logic has been updated to wrap this
binary and set the required environment variables.

Change-Id: I9b4f49f47872ae76430463e2fcb8f68114070f72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5603
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-23 15:04:56 +00:00
Vincent Ambo
045cf244b5 chore(ops/secrets): add key for tazjin/zamalek
Change-Id: Ieb2fe49a67940d7cfbd261edbe10d0a8577a466d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5628
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-17 10:14:02 +00:00
Vincent Ambo
302b754d7a feat(tazjin/nixos): Add system configuration for zamalek
This is my new Huawei MateBook X.

Change-Id: I32a8b77dd8f53b3c89bf63f448cd2880f9a457b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5554
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-05-10 13:41:33 +00:00
Florian Klink
e8855f4bef feat(ops/modules/open_eid.nix): document firefox
Firefox users can add p11-kit-proxy (or other SecurityDevices)
system-wide, by making use of the extraPolicies functionality.

Change-Id: Id58b6cab425199fb0e09e846db2a86d302c0de0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5534
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2022-05-08 13:52:27 +00:00
Florian Klink
84c62eb68b feat(ops/modules/open_eid.nix): use p11-kit-proxy
… instead of onepin-opensc-pkcs11.

This acts as a glue to multiple PKCS#11 modules, and reads configuration
files from /etc/pkcs11/modules.

p11-kit is also used to propagate the system trust store to NSS:
https://p11-glue.github.io/p11-glue/sharing-trust-policy.html

See-Also: https://p11-glue.github.io/p11-glue/p11-kit.html
Change-Id: I135c3a80a4eea0bd06f6b00089dc197c82476746
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5533
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-05-07 21:29:56 +00:00
sterni
fbbee5584b subtree(3p/cgit): update for git 2.36.0 support
Merge commit '51596ba1c25ff0dbba894153015203b4f1d3947b' into canon

Change-Id: Iaaf7a849d111aebc6bf85cec118439ba1d49f1e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5521
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-04 16:18:51 +00:00
sterni
5fae8b0826 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: ccl dumped images have fixed themselves… again

* //3p/git: rebase patch on 2.36.0

* //3p/overlays/haskell: remove upstreamed workarounds

* Disable everything depending on cgit temporarily, since it doesn't
  compile with git 2.36 yet.

Change-Id: I9dc11c0846641341adbdcc7162cbf149a15fe0cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5519
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-04 16:18:50 +00:00
Vincent Ambo
96aea32084 fix(ops/besadii): Fix output formatting for non-CL builds
Change-Id: Ie9ffb2d287f6c8a1e3ae45a7ad6671b9b8fa9c8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5505
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
2022-04-26 09:25:03 +00:00
Vincent Ambo
c05c4995ab chore(3p/sources): Bump channels and overlays
Changes:

* updated keycloak configuration for new version
* migrate to emacs28 outside of //users, re-add emacs27 but with a
  warning attached urging people to migrate

Change-Id: I3e5765a63934541f72f6c4a8673d3b4671850c93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5501
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-04-21 16:54:07 +00:00
Vincent Ambo
6716bf018c chore(nixery): Housekeeping for depot compatibility
Cleans up a whole bunch of things I wanted to get out of the door
right away:

* depot internal references to //third_party/nixery have been replaced
  with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks

Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-04-20 15:31:16 +00:00
Vincent Ambo
34e95f514c chore(journaldriver): Bump dependencies
Change-Id: I8819639bf5ddcc52510f20a92ab4b93af873682d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5475
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-17 13:16:34 +00:00
William Carroll
d843f0bf4c feat(wpcarro/ava): Support new machine
ava is my new (NixOS!) work machine :)

Change-Id: I1f089f00c02519d5d1d93d011f29075d53500e74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5450
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
2022-04-15 19:07:51 +00:00
Vincent Ambo
e3cd8069ef feat(ops/open_eid): Add script for setting up browser integration
Change-Id: Ib339d62d862fd99dab2fda30376b8e47b337a26b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5441
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2022-04-14 16:18:43 +00:00
Vincent Ambo
1a2fe4b063 feat(whitby): Increase prometheus retention time to 90d
Change-Id: I67287d7b1d8ee2c3004d381b5bc684bf4fc7d42c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5429
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-04-11 16:18:38 +00:00
Vincent Ambo
186c2822b0 feat(ops/modules): Add module for using Estonian e-residency card
Someone already packaged the required software, so I didn't have to do
that.

Change-Id: Ifc6a68fd4cd89f4718368a05acb6c6f536e01aab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5431
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-09 08:49:06 +00:00
Vincent Ambo
017238a1be fix(ops/oauth_proxy): Depend on Keycloak service
If the Keycloak service is running on the same machine as the oauth2
proxy (spoiler alert: it is!), let the service depend on it.

Change-Id: I30e4222b4cd5589e08849ef6f37cf1fb4369f55a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5421
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-03-31 13:27:48 +00:00
Vincent Ambo
e70428e75b refactor(ops/pipelines): Configurable GraphQL token location
For external users of the pipeline construction, the token might be in
a different path than `/run/agenix/buildkite-graphql-token`.

It is made configurable through the BUILDKITE_TOKEN_PATH environment
variable. This should be configured on the pipeline level to apply to
all steps.

Change-Id: I23c52e2d705e4134b8b013f8603f92e5533a6e44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5424
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
2022-03-30 11:06:49 +00:00
sterni
508a62b603 chore(3p/sources): Bump channels & overlays
* Remove use of aliases that have been removed in nixpkgs commit
  a36f455905d55838a0d284656e096fbdb857cf3a:

  - ncat
  - x11
  - nologin
  - dbus_libs
  - emacsPackagesGen
  - man-pages
  - pulseaudioLight

Change-Id: Ide603bf48bc7f77e10e4aa715ba025aece3644fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5387
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-03-19 17:11:59 +00:00
Vincent Ambo
fb5f21067e fix(modules/quassel): Open firewall port automatically
Change-Id: Ie815495561f789590b5f49ecfd33441822f79047
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5382
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-03-11 21:38:40 +00:00
Vincent Ambo
2cff0712b3 refactor(journaldriver): Replace ureq with crimp
crimp is in TVL (//net/crimp), and it has fewer dependencies than
ureq (including - finally - no more old time or chrono).

Change-Id: I354f8f78b34a85abe3af671ffeffbe6a7fded5ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5318
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-20 15:46:57 +00:00
Vincent Ambo
2626d8348a chore(journaldriver): Bump medallion to 2.5
This version includes my patch for removing chrono from
medallion (https://gitea.cmdln.net/cmdln/medallion/commit/025b143d)

Change-Id: I2b745598538dd34e967e49c2b342be1b04ca9f27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5316
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-20 15:46:57 +00:00
Vincent Ambo
e8756239a8 chore(ops/modules): Remove fix-nginx timer unit
This doesn't seem to be needed anymore.

Change-Id: Id8d4192840e8ab10adb652abc9bd6540009a3dcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5319
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-20 14:26:57 +00:00
Vincent Ambo
71d6a02ca1 refactor(journaldriver): Use time crate directly instead of chrono
With this change, we still depend on chrono (through medallion), but
but I'm going to try and fix that upstream as well.

Change-Id: Iefd3d8578ea8870961107f3222dea7f936c2dd9a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5311
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-18 11:52:26 +00:00
Vincent Ambo
af512558e6 test(journaldriver): Add test for serialising timestamps
Change-Id: I5b769f5974546fd4f4f853111bd17c9d22d73a5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5310
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-18 11:52:26 +00:00
Vincent Ambo
19a13e08a8 chore(journaldriver): Migrate to Rust Edition 2021
Change-Id: I858738b6fc554060824bbb4e69d5ccd03789535d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5309
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-18 11:52:25 +00:00
Vincent Ambo
c075a2c541 chore(journaldriver): Update crates within bounds
Basically just `cargo update`.

Change-Id: I86e58d73ff67d69201124d65136773325b240cbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5308
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-18 11:52:24 +00:00
Vincent Ambo
ede837b687 refactor(journaldriver): Use anyhow instead of failure
Apparently failure is not hip anymore, and crate updates are forcing
the use of anyhow now. Whatever.

The functionality basically stays the same, maybe error messages will
look a little bit different now.

Change-Id: I173d644688785339c16161ddeec47a534123710f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5307
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-18 11:52:24 +00:00
Vincent Ambo
4ce2b49cd9 feat(ops/glesys): add DNS record for AAAA sanduny.tvl.su.
Change-Id: I4a74cd173b326941c12b7611841ced2038650137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5314
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-02-18 11:47:23 +00:00
Vincent Ambo
d68de096c7 feat(ops/glesys): add DNS record for A sanduny.tvl.su.
Change-Id: I12e678f161ca9bfb7e982ed067a0b8bd0325d737
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5296
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-02-18 11:41:13 +00:00
Vincent Ambo
ac6717fe3c fix(ops/modules/www): Make self-redirect to config a generic module
As suggested by sterni, this makes the self-redirect of a machine to
its configuration a generic module working by convention.

In the process of moving this two small fixes have been applied:

* redirect is only applied if the URI is `/`, this is required for
  ACME to work
* addSSL = true is added, otherwise we have a certificate but no TLS
  listener

Change-Id: Icaef041ff681253a61e36926417bdb2844e3f93d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5313
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-02-18 11:39:01 +00:00
Vincent Ambo
5b701ad713 feat(sanduny): Enable journaldriver module
Change-Id: I9026386664000448642ff635bd71a7af5ed546c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5303
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-18 11:38:34 +00:00
Vincent Ambo
536e01e967 refactor(ops/modules): Move journaldriver configuration into module
This makes the journaldriver configuration machine-independent.
The secret is loaded from agenix instead of being persisted on disk.

Change-Id: I592ae7f5726fcb7f37a406f69dcf5ac498eeb1b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5302
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-18 11:38:34 +00:00
Vincent Ambo
c72abe04f2 feat(sanduny): Configure Bitfolk nameservers
Change-Id: I81b252aedbf1ce3543a167b6c1942c404d4f1f1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5312
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-18 11:35:21 +00:00
Vincent Ambo
95780174e1 feat(ops/machines): Add a module for known SSH keys
Change-Id: I443e479f3edf9c6540de7b5a33bc6f7e2a9c5183
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5305
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-18 08:22:56 +00:00
Vincent Ambo
b936843bb0 feat(ops/modules): Redirect machine base names to their config
With this change, entering just "whitby.tvl.fyi" or "sanduny.tvl.su"
in a browser will redirect users to their machine configurations.

Change-Id: Ibf076a469bcce073e1b1970aa568d6fe16a5c75a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5304
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-18 08:15:56 +00:00
Vincent Ambo
f4f1d97052 refactor(ops/modules): Move ACME base configuration into base.nix
This needs to be present on all machines that run ACME stuff.

I've switched the address for a .su one because I have a catchall for
these.

Change-Id: I7af8e1f1cb2fcfbcba4b7d1930ed0edef0106d72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5306
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-18 08:15:56 +00:00
Vincent Ambo
6b3eed1fb5 feat(ops/secrets): Add journaldriver key
This changes the structure of secrets.nix a bit to split between
secrets for whitby, and secrets for all TVL machines.

Change-Id: I791f0ce42a16b33051e24a7a6c5b153761ed9eb3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5300
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-17 18:11:58 +00:00
Vincent Ambo
e1353ff2cf feat(ops/machines): Add tvl-users to sanduny.tvl.su
Change-Id: I20f54f4ab298cfee91062f7bf4cdc8b0b3ccb37c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5299
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-17 18:11:58 +00:00
Vincent Ambo
9e9284fc86 feat(ops/machines): add configuration for sanduny.tvl.su
This will be an additional web host / fallback git server for whitby
incidents.

Change-Id: Icd6f7ce574ffd520b5783a50ff317feed7b71fc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5297
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-17 18:11:58 +00:00
Vincent Ambo
dd5ce78dbd refactor(ops/modules): Move user configuration into module
Rather than defining all system users inline on whitby, move them into
a module that can be imported on multiple machines.

Configuration for terminfos that we've added follows along.

Note that while doing this I've disabled logins for riking and isomer
since they are currently inactive in TVL.

Change-Id: Id18031d355afc34079c5e6e49dc6943e61809a8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5298
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-17 18:08:19 +00:00
Vincent Ambo
c72c1efdeb refactor(ops/modules): Rename git-serving -> josh
cgit has its own module now

Change-Id: I9b4cc322374517b8bd3db43345831e2bf43c4bb1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5295
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-16 23:03:06 +00:00
Vincent Ambo
cb8f050b9c refactor(ops/modules): Move cgit configuration into a module
The ancient `//web/cgit-taz` path stems from the time I had
code.tazj.in serving my initial version of the depot.

I've been meaning to clean this up for forever, so here we go.

Note that this leaves the git-serving module in a strange state where
it only deals with josh. I'll rename it accordingly.

Change-Id: I47ed1e9d90958299b5440a18a1b9075274754e33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5294
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-16 23:03:06 +00:00
sterni
235eda6649 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: re-enable CCL, as the crash has been fixed upstream,
  although it is unclear what exactly caused / fixed it.

* //ops/whitby: the kitty build broke upstream, so we can't install the
  terminfo on whitby for a bit.

Change-Id: I5710acbe837fbc936e334b2e81f9cf00ed6ae280
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5274
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-13 09:15:09 +00:00
Vincent Ambo
3d8ee62087 style(rust): Format all Rust code with rustfmt
Change-Id: Iab7e00cc26a4f9727d3ab98691ef379921a33052
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5240
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-08 12:06:39 +00:00
Vincent Ambo
3318982f81 chore: move some meta.targets definitions to meta.ci.targets
Change-Id: Icdec1dec89158fb596c5185ac7105892081947f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5252
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-08 12:06:10 +00:00
Vincent Ambo
98b6af02f5 chore(mq_cli): Rename cargo project to mq_cli
I am trying to publish this to crates.io, and `mq` is already taken up
by what seems to be a dead project.

Change-Id: I14d1f5f31f167fde954d9c1e39fc9fec5f4c3d10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5234
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-02-05 13:46:11 +00:00
Vincent Ambo
0971705902 chore(mq_cli): prepare new release
Same trick as in the last commit, trying to hit the right revision.

Change-Id: I0af9e88b4d2fd8239a7819dbe0da13f26cca8d6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5233
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-02-05 13:37:31 +00:00
Vincent Ambo
77fbe57819 chore(posix_mq.rs): Prepare new release to crates.io
... and update some outdated stuff in the README while we're at it.

Change-Id: Ib3a12596bd1ba61e91ac6e1d37106b616da3030d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5232
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-02-05 13:29:04 +00:00
Vincent Ambo
91ef2b671e chore(ops/posix_mq.rs): upgrade to nix 0.23
The previous version had a CVE.

As part of this upgrade, the handling of errors inside of the Nix
crate changed, which we now accommodate.

Change-Id: Iad9a473c1782e0d79919cb5dc3f76316852d8a16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5226
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-04 15:58:19 +00:00
Vincent Ambo
132798cceb fix(ops/posix_mq.rs): use newer struct mem::MaybeUninit
The previous mem::unitialized method was deprecated in favour of this
struct which carries information about the initialisation state
forward to the compiler.

Change-Id: Ib5f5d1ad91f9957b18eebabc1048f8649bc74049
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5225
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-04 12:47:55 +00:00
Vincent Ambo
054fe8f94b refactor(ops/posix_mq.rs): Update code using deprecated Error trait
The previous Error::description method was deprecated.

Change-Id: I63efd272484c8715b9f324ab09e967fbf53cf55a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5224
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-04 12:47:55 +00:00
Vincent Ambo
aa122cbae7 style: format entire depot with nixpkgs-fmt
This CL can be used to compare the style of nixpkgs-fmt against other
formatters (nixpkgs, alejandra).

Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: cynthia <cynthia@tvl.fyi>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: grfn <grfn@gws.fyi>
2022-01-31 16:11:53 +00:00
Griffin Smith
5f42a160be fix(ops/whitby): enable services.nginx.statusPage
This is necessary for the nginx prometheus exporter to work

Change-Id: I2343d6f5d3d6d6772777d5e14426a537aa1c8ef7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5127
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2022-01-29 23:07:00 +00:00
Griffin Smith
84304b925a feat(whitby): Enable nginx prometheus exporter
Might be nice to look at rates of requests etc.

Change-Id: I4d12ab0c1a555793e803de4a9614e616951a94e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5125
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
2022-01-29 23:00:25 +00:00
Griffin Smith
63b09a6466 feat(ops/secrets): Add secret for telegram alerts bot token
This isn't actually used by anything that would use agenix, but this
seems like a vaguely sensible way of sharing the token with other people
regardless.

Anyone who finds this commit and wants to be added to the telegram
channel where the alerts go, lmk.

Change-Id: I06d6ed2d4bec099cbf68ede8fd00a5e6f4e7bc60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5124
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2022-01-29 22:59:58 +00:00
Vincent Ambo
dddeb04679 fix(www/tvl.fyi): Anchor /blog redirects at #blog
Since our blog index is on the index page, this makes slightly more
sense.

Change-Id: I7b8164490c133e23d892abef21275f8bfed50b66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5123
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2022-01-29 22:37:55 +00:00
Griffin Smith
be91bc97a9 fix(tvl.fyi): Redirect /blog/ (with trailing /) to /
This was already happening without the trailing slash, but needs to
happen separately with it.

Fixes: b/172
Change-Id: Ic3423fd7a2eaf76a073badd80965cee953df4ce9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5121
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-01-29 22:27:54 +00:00
Vincent Ambo
7b7dfa3bf2 feat(ops/www): Write JSON access log to journald
This means it will end up in journaldriver.

Change-Id: I66f781085b5dac9946b3b9a2bf30e447863e1213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5122
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-29 22:24:53 +00:00
Vincent Ambo
88595c23ce feat(ops/glesys): Move tvl.su email over to Yandex 360
This fixes b/158

Change-Id: Ie58737dfe144a342b6d4d30f72ec60d70405af31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5110
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-01-29 14:51:47 +00:00
Vincent Ambo
19d7ccf87f feat(ops/glesys): Set up Yandex 360 verification record on tvl.su
Relates to b/158

Change-Id: I6738505bcf0088d35bb7c8f9215dcf0b4e27f9ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5109
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-01-29 14:51:47 +00:00
Vincent Ambo
ec2b98d131 refactor(ops/pipelines): Move into postBuildSteps
There is no need for this step to be part of the static pipeline (it
should not run if the build fails anyways).

Change-Id: I71400a452d6f8f4708d146b346eaffda5da2f766
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5049
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-22 12:01:09 +00:00
Vincent Ambo
cb107c4fbe feat(ops/pipelines): Upload post-build steps in static pipeline
Change-Id: I5ce6d51837c734951fe10c4f21806cf0fc57ed23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5048
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-22 11:59:09 +00:00
Vincent Ambo
5a88e47b71 refactor(ops/pipelines): Split build/post steps into separate chunks
This will create `build-chunk-$n.json` files for steps that should run
_before_ duck, and `post-chunk-$n.json` files for steps that should
run after duck.

The post steps are not yet uploaded to Buildkite, but we also don't
have any right now.

Change-Id: I7e1b59cf55a8bf1d97266f6e988aa496959077bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5047
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-22 11:59:08 +00:00
Vincent Ambo
f12ceaa622 refactor(ops/pipelines): Use branches filter for canon-only steps
Using this instead of a conditional leads to nicer output in the UI,
but has no semantic difference.

Change-Id: I5b368d663f417d256e4792d2d46b84fc50d42d0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5045
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-22 11:59:08 +00:00
Vincent Ambo
a4aabaff68 refactor(ops/pipelines): Move :git: step up in the pipeline
This step is independent of the build result and can be scheduled at
the beginning while pipeline eval is still in progress.

Change-Id: I2ee268e4c333efa654dcb12c0b1562b43231d241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5044
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-22 11:29:04 +00:00
Vincent Ambo
4b9bd8afd7 feat(ops/pipelines): Always upload entire pipeline output
Previously we only stored the drvmap, but we will also need the build
chunks to refactor the generation of dynamic post-steps.

Change-Id: I256fffe13af8f8c4521835257f5d87dda323b248
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5043
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-22 11:29:04 +00:00
Vincent Ambo
3e9aa7722d refactor(depotfmt): Move formatting check into an extra step
Change-Id: I7e4cf6bb2351bd11a5396f1663c0d4cc97c0d94e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5009
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-21 11:49:03 +00:00
Vincent Ambo
ac7e989571 feat(ops/pipelines): Trigger pipeline for tvl-kit through canon
This CI pipeline in Buildkite verifies the external (josh-provided)
view of the depot at //views/kit.

See https://buildkite.com/tvl/tvl-kit

Note that this always triggers a build of HEAD. This is because we
don't know the transformed commit ID, and we currently have no way to
pass a ref through. The pipeline is configured to skip intermediate
builds.

I asked Buildkite for some ideas on how to improve this, lets see.

Change-Id: I6c60fb1ea7606c1c90219ef04fd7bada64661529
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5010
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-20 17:35:18 +00:00
Vincent Ambo
0652ac0ace refactor(nix/buildkite): Move fetch-parent-targets script here
This is no longer TVL-specific and should live here with the other
generalised stuff.

Change-Id: I95a1b4c0321f34812162d6fd40568269abf639dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5006
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-19 17:22:36 +00:00
Vincent Ambo
c2e6c0719c refactor(ops/pipelines): Generalise fetch-parent-targets script
Removes all TVL-specific values in favour of environment variables
supplied by Buildkite.

This makes it possible to reuse this script outside of TVL.

Change-Id: Ic543bc41e4c81e65ee349ad241c515231e97ab30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5005
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-19 17:22:36 +00:00
Åsmund Østvold
6e4b0f3cef feat(ops/besadii): make text 'cl' posted BuildKite configurable
Some companies do not know the 'cl' term. They do know of 'change' and
would maybe not like to introduce one more synonym.

This cl introduce an optional entry 'gerritChangeName' in
besadii.json. The string has to match `^[a-z0-9]+$` for readability.

Change-Id: Id70fcb1e45158869f88bf37669be49b8b8a3b295
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4825
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: asmundo <asmundo@gmail.com>
2022-01-19 10:31:52 +00:00
Vincent Ambo
1f3aa71cf2 fix(ops/oauth2_proxy): Fix cookie secret length
The cookie secret in the encrypted file was too long, because the
generation command in the oauth2_proxy docs is also wrong. Should
probably fix that upstream as well.

Also noticed that an extra '2' snuck into the service name and fixed
that.

Change-Id: I9a344a75993ab1f98299a8d45e7f5b2e146b7fc5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4957
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 13:51:47 +00:00
Vincent Ambo
9596c642d5 feat(ops/pipelines): Fetch parent target map for pipeline generation
Change-Id: I1c7d48fc0974549d67146a15f79ddb0b6ddfe805
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4947
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 11:49:01 +00:00
Vincent Ambo
0779f96687 feat(nix/buildkite): Check target map of parent to determine skips
This changes the logic for build pipeline generation to inspect
an (optional) parentTargetMap attribute which contains the derivation
map of a target commit.

Targets that existed in a parent commit with the same drv hash will be
skipped, as they are not considered to have changed.

This does not yet wire up any logic for retrieving the target map from
storage, meaning that at this commit all targets are always built.

The intention is that we will have logic to fetch the target
map (initially from Buildkite artefact storage), which we then pass to
the depot via externalArgs when actually generating the pipeline.

Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 11:49:01 +00:00
Vincent Ambo
0a21da2bb4 feat(ops/pipelines): Create drvmap structure for each commit
Always create a structure that maps all targets to derivations, and
persist it as a JSON file.

This relates to some of the ideas expressed in:

https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit

The file is always uploaded to Buildkite as an artifact. This allows
for retrieving it based on the commit ID in a Buildkite GraphQL query.

By default, Buildkite stores artefacts for 6 months. Storage location
can be overridden (with custom retention) through some environment
variables, but for now at TVL the Buildkite-managed storage is fine.
See also: https://buildkite.com/docs/pipelines/artifacts

In the subsequent filtering implementation, when diffing commits
across a time-range that exceeds artefact retention time, we should
simply default to building everything.

Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 10:26:08 +00:00
Vincent Ambo
005cd4037a fix(gerrit-tvl): Handle "broken" (skipped) jobs correctly
by simply skipping them

Change-Id: I9cbec3b79469ae01b1873d6a42e990b98cc4110a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4921
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2022-01-14 20:44:56 +00:00
Vincent Ambo
a5c9b11a6b fix(gerrit-tvl): Exclude non-command jobs from check results
Change-Id: I13727d30ac7a568f02614a4bbc778afed6a286ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4891
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-14 20:30:53 +00:00
Vincent Ambo
86a205220f fix(gerrit-tvl): Explicitly specify patchset on check runs
Since we now group patchsets inside of Buildkite, the results are no
longer guaranteed to be for the right patchset.

There might be some metadata passed in from Gerrit that would let us
do this with the commit ID instead, but I haven't checked.

Change-Id: I5b74a17697511160fcc89d3dbef23517d974dc6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4890
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-14 20:30:53 +00:00
Vincent Ambo
e8bf17317e fix(gerrit-tvl): Mark job as failed on all failure states
Change-Id: If0fa85d8178b9e457305d0244ddf67d12a4b3051
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4889
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-14 20:30:52 +00:00
Vincent Ambo
e1ffaee1dd fix(gerrit-tvl): Support all documented Buildkite job statuses
I'm not sure where the previous list originated, but it was missing
some officially documented statuses. However, the API definitely
returns statuses that are documented to only appear in other types, so
this commit simply maps ALL statuses that Buildkite has documented for
any type.

Also adds a log statement in case we encounter a brand new, unknown,
undocumented status.

Change-Id: Iff003a3bd2608702019ae0f4137958435ad0856f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4888
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2022-01-14 20:30:52 +00:00
Vincent Ambo
058bf61193 fix(gerrit-tvl): Fix construction of ref used in Buildkite
... and remove a spammy log statement.

This changed in besadii a while ago and lead to the behaviour of
failing silently, instead of failing with an error saying "undefined
undefined".

Note that with this change merged the plugin probably still won't work
again, but it gets us a step closer to the real error.

Change-Id: I3db25d246f4b1c634d316cd92574e27fb220d769
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4887
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2022-01-14 20:30:52 +00:00
Vincent Ambo
ee52fbc46c feat(besadii): Skip builds of patchsets with no code changes
Currently Gerrit is configured to copy forward the scores of the
'Verified' label if the tree of the commit does not change (e.g. only
author information or commit message is modified).

Besadii still triggers builds for these patchsets though. With this
change it will inspect the (previously ignored) "kind" of the patchset
and skip patchsets with the same tree as their predecessor.

See Gerrit docs for the semantics of "kind":

https://gerrit-review.googlesource.com/Documentation/json.html#patchSet

Note that an argument can be made that we should do the exact opposite
- stop carrying over 'Verified' at all and always build all patchsets.

I think this depends on whether we intend to use commit metadata in CI
runs at all. Adding a few people to the review for opinions.

Change-Id: I48a96a1ad1e07d92330d84e5cfdc820a39395297
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4867
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: asmundo <asmundo@gmail.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-14 17:35:45 +00:00
Vincent Ambo
bce5192566 feat(whitby): Install a handful of systemPackages
Adds more things I keep using via nix-shell, as well as the
deploy-whitby script (which is independent of a particular depot
checkout).

Change-Id: I36f87de7645768a05268c90ba9b3ab833bacca05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4881
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-13 08:39:03 +00:00
Vincent Ambo
9c025d62a0 refactor(deploy-whitby): use nvd instead of nix-diff
nvd only shows us changed versions of packages, as well as
added/removed packages, which means that for the majority of depot
packages nothing will be displayed

however, the current output of nix-diff is not usable anyways, so
having something that can be looked at is better than nothing

Change-Id: Iefbd8139c7ccf5c88ed1209897abdb2ae9302e91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4868
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-01-13 08:14:00 +00:00