Commit graph

561 commits

Author SHA1 Message Date
Vincent Ambo
13dc0793cf feat(ops/glesys): Create objectstorage key for litestream
Change-Id: I8b3e4f767440ae7763c1e6ce9fd97c557fe516ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4577
Tested-by: BuildkiteCI
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-24 18:54:43 +00:00
Vincent Ambo
e2596e930e feat(ops/glesys): Move Terraform state to GleSYS bucket
Change-Id: Ib14fba9a5f06ecdb065dd14580c8088f98e9cb3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4576
Tested-by: BuildkiteCI
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-24 18:38:40 +00:00
Vincent Ambo
ef92f8b231 feat(ops/glesys): Create bucket & key for storing terraform state
Change-Id: I73cfaa614d46afb65ba312e767d1e924669fbae1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4575
Tested-by: BuildkiteCI
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-24 18:38:40 +00:00
Vincent Ambo
afe648bbb2 feat(ops/glesys): Import existing object storage instance
Change-Id: I5a5269ef0d385d864dd8f62eb2332e6ae2cb2672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4574
Tested-by: BuildkiteCI
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-24 18:38:39 +00:00
sterni
7de43fe4ee style(ops/besadii): run depotfmt
Unclear if this reformat is caused by the channel update or if this file
was ignored previously.

Change-Id: I3498ab181c7fff1b132419783e33a96f7bebfe42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4520
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-21 14:19:28 +00:00
Vincent Ambo
4ad94b9cf8 feat(ops/pipelines): annotate patchset builds with Gerrit URLs
If available, provide a link back to Gerrit on the overview page of a
build.

Uses the default style (i.e. style unset), which makes it
non-intrusive visually.

Change-Id: I4271d589d548015b75762fd0584f3958bfcc53e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4442
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-19 21:23:51 +00:00
Vincent Ambo
562236085b style: format all Go code
The code in //users/wpcarro/tools/monzo_ynab/ynab/client.go was not
valid Go and has been commented out.

Change-Id: Icb4003607f30294dcbf60132eb7722702c7f0d84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4400
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-12-19 18:05:04 +00:00
Vincent Ambo
eea2dbadd0 fix(ops/besadii): fix Gerrit URL format ... again
got into some kind of race with different patchsets of this CL
somehow, idk

Change-Id: I3dcdb708f141829b866fbd786483710b43ea9824
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4481
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2021-12-19 17:08:08 +00:00
Vincent Ambo
1d4715781c fix(ops/besadii): Only set branch to CL when building patchsets
If we set this for canon, then stuff starts to fail in non-obvious ways.

Change-Id: I3bf38e29151c6066aaf4eba68ae387272d8a82c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4463
Tested-by: BuildkiteCI
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-19 16:53:47 +00:00
Vincent Ambo
f8b4029b17 fix(ops/besadii): Stop path.Join from eating our URL
apparently this chomps away at things inside of fragment strings

Change-Id: Ie60d52d101dc4281b3a62c228af076791e1c7928
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4462
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-19 16:46:00 +00:00
Vincent Ambo
3a2a5ffa9d feat(ops/besadii): Pass Gerrit link to builds as an envvar
This makes it possible to annotate builds with a link back to Gerrit.

Change-Id: If351785d3b631b96753d41f417ca94bc7a95ac54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4441
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2021-12-19 16:27:29 +00:00
Griffin Smith
b075b1df9d feat(ops/besadii): Make branch key cl/XXXX
The branch key for buildkite builds isn't actually used to fetch if a
commit is given - instead, it's just a visual grouping of multiple
builds. This means we can just make the branch key cl/<cl number>, which
is the convention we already use to refer to CLs and gets us a nice
visual grouping of builds of successive patchsets of the same CL number,
even though the ref we're providing isn't a real ref.

Change-Id: Iaa9111297a88f965fda94cd8266240106f58a100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4347
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: grfn <grfn@gws.fyi>
2021-12-19 16:16:56 +00:00
Griffin Smith
7f593781f3 feat(whitby): Add buildkite agents to docker group
I'd like to be able to run extra CI steps that include running docker
containers (to integration test things like webapps that connect to a
database). To do this the buildkite agents themselves need permission to
do docker things.

Change-Id: I3c9a488708f0e12a508754ac41f04148ca7aedac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4408
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-12-19 12:58:35 +00:00
William Carroll
06a20ba3df fix(ops/diogenes): Ensure diogenes builds
diogenes "passed" CI because the file was named configuration.nix
(vestage from the NixOS default /etc/nixos/configuration). This CL fixes
some issues I encountered after running depot/bin/rebuild-system.

TL;DR:
- rename configuration.nix -> default.nix to trigger CI
- add diogenes to my systems
- add public SSH key

Change-Id: I24197b8936c201267db6f71f00099dce590eac1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4388
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: wpcarro <wpcarro@gmail.com>
2021-12-17 22:12:40 +00:00
William Carroll
1ca4c6d2be feat(wpcarro/marcus): Add marcus to the depot
me: marcus, meet depot. depot, meet marcus.
Change-Id: Ic6a25ac85e4c7f6dfea2a42b46a4400f92df70a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4351
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-16 22:07:02 +00:00
zseri
3e18b034bb feat(ops/users): Add user zseri
Submitted via IRC, instead of the usual Google Groups
due to email issues.

Change-Id: I71a2bdfd10b02370df61bbba4dabc2f45b6c1009
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4384
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <mail@tazj.in>
2021-12-16 19:13:02 +00:00
Griffin Smith
d85c6a1b56 feat(ops/modules): Provide some modules to all nixoses
For modules that are gated behind a mkEnableOption, it's reasonable to
just provide them to all Depot-built nixos systems without requiring
people to explicitly import them. This defines a special module called
`default-imports.nix` which imports these modules (currently just
tvl-cache.nix and automatic-gc.nix, as I'm being rather conservative
adding things here to avoid breaking anyone's system), then provides
that module as one of the `modules` passed at the top-level
nixos/eval-config invocation.

Change-Id: I3be299ab10ae4c451ef11c514edb3c89318a2278
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4345
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-12-16 14:17:35 +00:00
Griffin Smith
0f5cc25697 feat(ops/modules): Add shared module for TVL cache
Add a shared nixos module for configuring whitby as a binary nix cache,
and refactor tverskoy to use this module.

This is enabled via an option to pave the way for including it as an
import in all depot-generated nixos configs at some point in the future.

Change-Id: I6dcc0e8eb48b1ac34457666dceebeedd5da6c526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4344
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: grfn <grfn@gws.fyi>
2021-12-15 23:36:22 +00:00
Griffin Smith
50b43cfb66 fix(ops/besadii): Don't send notifications for CI status
Don't notify reviewers ever on CI status changes, and only notify the
owner if the build fails.

Change-Id: If2cf63581b49e3de77181024ce8a4213031f4bd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4337
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: grfn <grfn@gws.fyi>
2021-12-15 19:01:55 +00:00
Vincent Ambo
743bee8686 fix(ops/pipelines): Allow steps to run immediately after upload
This fix was recommended by Buildkite and is explained in the comment.

Change-Id: I3f1c1c07cba0b417857d69c021c8af4750d645c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4334
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-12-15 16:55:03 +00:00
Vincent Ambo
38ec27e834 fix(ops/pipelines): Chunk build pipeline into multiple uploads
The number of jobs in the depot pipeline is reaching the limits of the
Buildkite backend's ability for a single pipeline upload. Based on a
conversation with their support my understanding is that this has to
do with internal locking mechanisms at Buildkite.

To work around this, we can instead chunk the pipeline into several
smaller chunks that are uploaded serially.

This commit introduces logic to chunk the pipeline accordingly. The
chunk size chosen is 256 for now (a multiple of our number of agents,
which is useful if we can get builds from the first chunk to start
before the next ones are uploaded).

Note that this chunk size is significantly below even the current
number of targets (~460 as of this commit), but choosing a lower chunk
size might alleviate problems we've been seeing with timeouts during
pipeline uploads.

Change-Id: I77030aaf8b874c330218b78c77d15216e13b9af7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4332
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: tazjin <mail@tazj.in>
2021-12-15 15:49:40 +00:00
Florian Klink
46a2c75a59 docs(ops/irccat): link to credentials RFE
https://cl.tvl.fyi/c/depot/+/4264 did move merging config with secrets
into ExecStart=, which is tracked in an RFE upstream:

https://github.com/systemd/systemd/issues/19604#issuecomment-989279884

We didn't link to this so far, neither in the commit message, nor in a
comment.

Let's add a comment, so people know when we can undo this.

Change-Id: I7bed370b671093bb876592b4dccd562f1c256cd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4326
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-14 17:30:33 +00:00
sterni
9f22b4f1c8 docs(ops/pipelines/depot): correct comment about fallback build cmd
We can gcroot the derivation files and drop this step, but have
elected not to do so for the moment, see cl/3436.

Change-Id: I993a1f3921e9f21e18fa260e76d3dd15ffa556bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4327
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <mail@tazj.in>
2021-12-14 17:02:29 +00:00
Åsmund Østvold
6842e25f14 feat(besadii): Make Gerrit label configurable
By default besadii will set the `Verified` label in Gerrit. This adds
a config option to set a different label instead if desired.

Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I254159e46994e01182987ed5e5e26e27c57f46ce
2021-12-14 09:28:09 +01:00
Vincent Ambo
b97d6b0f1b feat(ops/users): Add wpcarro
... this was overdue!

Change-Id: I435768007db4a0f3663e1aa9376e8cae4d1d0381
2021-12-13 23:54:12 +03:00
Vincent Ambo
79b4e0e1a4 chore(ops/users): Rotate password hash for asmundo
New hash received via an authenticated channel, of course.

Change-Id: Idca688d8a8bb2e943aef3937f54d292b48f79fad
2021-12-13 23:51:43 +03:00
sterni
fe0e19ead4 feat(ops/whitby): install alacritty terminfo
alacritty is used by grfn atm.

Change-Id: I10dacd301044f9c37790e22e955cb068fcbd2cfc
2021-12-13 19:40:28 +00:00
sterni
cdf7480662 feat(ops/whitby): add terminfos for other terminals used
* foot (me)
* kitty (lukegb)

Change-Id: I65303e39c4adb05e362792a544134fc2884175bf
2021-12-13 12:56:12 +00:00
Vincent Ambo
961443c23c feat(whitby): Add some more useful programs
I keep using these in nix-shell but really they should just be
installed.

Change-Id: Ic2c36bae8b582fef88029b288accdfd3c8bc0f1b
2021-12-13 15:48:41 +03:00
Vincent Ambo
3a410a78df feat(ops/secrets): Make (encrypted) secrets part of the tree
Currently in NixOS configuration using agenix secrets there is no
build time validation of secret paths - things fail at runtime (system
activation).

To prevent that, this CL makes the secrets part of the tree based on
the same configuration file used by agenix itself.

This guards against:

* agenix secrets.nix definition for a non-existent file
* age.secrets value in a NixOS config for a non-existent secret

Change-Id: I5b191dcbd5b2522566ff7c38f8a988bbf7679364
2021-12-12 11:19:24 +03:00
Vincent Ambo
8cbb42006a chore(ops/secrets): Reencrypt all secrets with sterni included
Change-Id: I14043c2bd9da43a6b7de65baf0ebb75eaf3c4e22
2021-12-11 18:51:36 +03:00
sterni
40096c2931 feat(ops/secrets): add keys for sterni
Change-Id: Idf13f7737dd51e74e87093e07cdf22ad24407944
2021-12-11 15:41:55 +00:00
Vincent Ambo
f9bd68e247 fix(ops/secrets): Fix missing file
... okay, this is like the 5th error related to something with this
and file paths. Need to write some validation logic.

Change-Id: I4314818aa1bc25b8cf7bd3593850d3836ccb867c
2021-12-10 23:53:50 +03:00
Vincent Ambo
aa5bf312e8 fix(tvl-buildkite): Use supported credential helper binary name
Git only allows binary names prefixed with `git-credential-` if the
path to the helper is not absolute.

Why? Who knows.

Change-Id: I216b2a621f62a73f05e21def7ec8016b29ede892
2021-12-10 23:37:57 +03:00
Vincent Ambo
2f1c654c14 refactor(ops): Move panettone secrets to agenix
Relates to b/161

Change-Id: I508e5a0eacab668f4bd39a2c888d894b96bed093
2021-12-10 23:19:56 +03:00
Vincent Ambo
2b9be81ea0 refactor(ops/pipelines): Use agenix-deployed besadii secrets
I *think* this is the final step for b/161

Change-Id: Ie7a2198a045f2f1866a245884ab0f5414e205327
2021-12-10 23:14:41 +03:00
Vincent Ambo
60f96d2b17 fix(whitby): Fix typo in buildkite-agents group name
... really would like some assertion helpers for this sort of stuff.

Change-Id: I32d1de18ebfbbdfa5128a8fbdad2efcc511f8514
2021-12-10 23:01:20 +03:00
Vincent Ambo
5baa9b6d87 refactor(tvl-buildkite): Prepare gerrit credentials helper
Currently this functionality is provided by a shell script stored in
/etc/secrets (which has the password value hardcoded).

This needs to happen in a separate commit from the one that changes
the pipeline to avoid breaking it (it needs to be deployed first).

Change-Id: I680754c828ccefbacfcf0d5c813a4bc19493ba4c
2021-12-10 19:52:39 +00:00
Vincent Ambo
2fe8d724d7 refactor(ops): Move Nix cache secret to agenix
... and also the public key, just to keep the distribution mechanism
the same.

Change-Id: Ief14daf9344c0fb99eeb5789c1ec9bfb1f12bee0
2021-12-10 19:48:26 +00:00
Vincent Ambo
82a885a750 refactor(ops): Use besadii configuration from agenix
We already checked this in, but this commit adds the configuration for
making use of it.

There are two copies of besadii's JSON configuration with different
permissions.

Note that the buildkite-graphql-token path needs to be updated in
static-pipeline.yml, but this needs to happen in a separate commit
after deploy because the pipeline will break otherwise.

Change-Id: I6fab4bf1a2e679df7cf76521e2b53bd9dadbac62
2021-12-10 19:31:36 +00:00
Vincent Ambo
b1108821a9 refactor(ops): Move grafana secret into agenix
Change-Id: Id141758135c796881e91d20b950dae74c40d9ab3
2021-12-10 19:31:36 +00:00
Vincent Ambo
b2d46aed2b fix(tvl-buildkite): Add more missing programs to agent path
... this option really is a pitfall! The list of programs is now the
same as in the upstream module, plus curl and jq.

Change-Id: I29edae4b2400a2724f62df9efa1dc184a8b0af5f
2021-12-10 17:13:22 +00:00
Vincent Ambo
b8267c261c fix(ops/irccat): Avoid permissions issue with LoadCredentials=
The DynamicUser + Group configuration does not work as planned, thus
the systemd LoadCredentials feature is used instead which makes the
file (which itself is only readable by root) available in a
memory-backed location only readable by the service.

The secret is only available to `ExecStart` commands, so units using
this feature can not be used with pre/post units and the like if those
commands need secrets.

To accommodate this, the merge of configuration files has been moved
into the service launch script, which is now the ExecStart= process.

For details take a look at https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LoadCredential=ID:PATH

Change-Id: I693fe5677cc0d63c7aa485c2c7472457c5262166
2021-12-10 15:09:09 +00:00
Vincent Ambo
67bde5ecc3 fix(tvl-buildkite): Explicitly set runtimePackages
It turns out the lib.mkAfter call doesn't behave as expected -
only *some* of the packages that are defaulted end up in the $PATH.

I suspect this is actually something else, e.g. these packages are
always added for some reason or another, and the option is completely
overridden every time.

Change-Id: I854c7198520d82b00e6338ed0fe653836226dc6d
2021-12-10 15:06:08 +00:00
Vincent Ambo
2ba481451c chore(ops/secrets): Reencrypt with grfn's key included
Change-Id: I66df150ab5070a81a92f0741334639df9df1f86f
2021-12-10 17:52:08 +03:00
Griffin Smith
a85ab68b12 chore(ops/users): Rotate password for grfn
Just a regular password rotation, plus I wasn't using argon2 unlike
everyone else.

Change-Id: Ic57fe79a2dbfdc15397d20f6b2b47c6aac911d29
2021-12-10 09:45:17 -05:00
Griffin Smith
66a1d3d5d4 feat(ops/secrets): Add key for grfn
Change-Id: I8063ae804932e3815e9a499e0206806818b9b021
2021-12-10 09:44:34 -05:00
Vincent Ambo
bc3d35f3d0 fix(tvl-buildkite): Add missing runtimePackages back
Turns out that the type of this option is not concatenative and it
replaces the packages needed to run Buildkite if set.

Change-Id: I9f52572bc165bccdd8c6518cfdf7b8967f7a50d0
2021-12-10 13:14:11 +00:00
Vincent Ambo
d4403638cf refactor(ops): Move irccat secret into agenix
The irccat module uses DynamicUser, so to grant permission to it a new
group has been added for irccat.

I have some vague memory of DynamicUser + Group not behaving as one
would expect, but we'll see what happens.

Change-Id: Iab9f6a3f1a53c4133b635458ce173250cc9a3fac
2021-12-10 16:13:31 +03:00
Vincent Ambo
002d183876 refactor(ops): Move clbot SSH key into agenix
Change-Id: Iae03ead7dda0509689a76f0d76f9cfeb8434e967
2021-12-10 16:13:31 +03:00