Commit graph

240 commits

Author SHA1 Message Date
Griffin Smith
7a83b8fe40 fix(ops/monorepo-gerrit): Escape escape characters with more escape characters
Escape!

Change-Id: Idec7d17feda13ccf1ec03c41106ec6ed581d9e47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1901
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-31 23:21:45 +00:00
Vincent Ambo
2afa22e512 feat(whitby): Double the number of Buildkite agents
Allow 16 things to happen in parallel, which is useful now that the
CI granularity is on a per-target level.

Change-Id: Ie65dd119ea0666618fbb249613e70a68276db834
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1902
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-31 23:21:16 +00:00
Vincent Ambo
9c482d6238 feat(ci): Add subtarget support for builds
We have naturally evolved a distinction between logical and physical
targets.

Physical targets are those which correspond directly to a tree
location on disk and can be built with `-A path.to.files`, while
logical targets are those that are exported from within an expression
but do not have a corresponding file on disk.

This change adds support for exporting logical targets from any tree
location by adding a `meta.targets` attribute containing keys into
itself, which will be consumed by the CI target gathering logic and
included in the generated pipeline.

Note that the labels for subtargets are syntactically different to
emphasise that they do not correspond to a file location. For example,
this change enables 'ops.nixos.whitbySystem' as a subtarget, which is
labeled in CI as `ops/nixos:whitbySystem`.

Change-Id: Ied09647a62c2ba98e3914548e3742ad422c63ecf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1893
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-31 23:14:11 +00:00
Vincent Ambo
61d2d2d503 feat(ops/pipelines): Dynamically generate CI pipeline from targets
Create the pipeline by outputting a file that contains nix-build
invocations for each target's *derivation path*.

Each invocation has a generated Nix expression passed to it with `-E`
which fetches the correct target from the tree while correctly
handling targets with strange characters (such as in Go-packages).

This makes it possible to run target-level granular pipelines. We're
getting somewhere!

Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-31 23:14:11 +00:00
Griffin Smith
9b4986c5ed feat(monorepo-gerrit): Auto-link panettone issues
Auto-link b/\d+ in gerrit comments + patch comments to panettone issues.
This is a departure from the previous syntax of #\d+ that we were using
historically, but has been requested by multiple people

Fixes: b/40
Change-Id: I8e556eb6b2f55ae80d945b86b51041751ecb053b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1888
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-30 22:16:50 +00:00
Vincent Ambo
5ad61ee408 chore: Add __readTree attributes on systems that need to be built
This is a temporary state (TODO added) to be picked up by the new CI
logic.

Change-Id: Id4702740ffd18325088e2a8a0c6157a8cee7ccf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-26 23:49:32 +00:00
Vincent Ambo
d4824b64f6 feat(whitby): Enable log forwarding via journaldriver
Change-Id: I474159acfe514f6f2eb7867e4eba854016590ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1836
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-08-23 20:45:18 +00:00
tazjin
674d921202 revert(whitby): Use Tvix as the system Nix on whitby
This reverts commit 8fa3087067.

Reason for revert:

This almost worked. We discovered two important issues:

- The daemon startup does not correctly handle the socket passed in by systemd.

- There is some issue with chunking of large calls, running a build for ci-builds resulted in:

tazjin@whitby /depot (canon)> nix-build -A ciBuilds.__allTargets
E20200821 01:42:22.846053 12601 shared.cc:306] error: Rpc call addTextToStore to unix:///nix/var/nix/daemon-socket/socket failed (RESOURCE_EXHAUSTED): Received message larger than max (10889961 vs. 4194304)

Change-Id: Ic5ba4ef06a4953cf71a36b139fe25ea673cb6fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1802
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 02:01:33 +00:00
Vincent Ambo
8fa3087067 feat(whitby): Use Tvix as the system Nix on whitby
... this is going to break so much stuff. Lets have some fun.

Change-Id: If0185e0323391c7055d47b797083bb5afde57cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1829
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 01:37:25 +00:00
V
d6f17f48de chore(tvl-slapd): decapitalise V's username, use displayName instead
Change-Id: I59cf5e1c850960ae639c6a3ebeb273a4441c48bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1788
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-19 01:23:39 +00:00
V
b8fe5469c6 feat(whitby): add V
Change-Id: I887760edd67135df4e2f58a874314b317838d2e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1787
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-19 01:08:33 +00:00
V
8179fb6285 feat(tvl-slapd): add V
Change-Id: Id9253635b73b0eac7871a6baa4f0c7417d135cfe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1786
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-19 00:12:04 +00:00
eta
737d192aec feat(whitby): add eta
Change-Id: I7aa2bd2cb2c001b48ebd25b20f28cdfb0883ba3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1782
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-17 22:27:25 +00:00
Vincent Ambo
85f53a45db chore(nixos/sourcegraph): Bump version to 3.18.0
Change-Id: I5cb8b2da7e40075c99fab6bd57295c8c1d770e86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1781
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-17 21:52:48 +00:00
Vincent Ambo
a7868e4e17 feat(nixos/clbot): Add ability to post in multiple channels
Adds the ability to post to multiple channels by simply running
multiple instances of clbot.

We should probably implement support for this in clbot itself, but
right now I can't be bothered to write Go.

Change-Id: I5cffd0dc10a7f6cc19c37c5834c5610166b4ae23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1771
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-17 21:50:45 +00:00
multi
9108256ce4 feat(whitby): enable programs.mosh.
Change-Id: Ibc8df6f6382b5b64e272bedece6b65762f9693c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1750
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-14 21:08:27 +00:00
multi
8af65f6858 fix(whitby): disable sshd(8) password authentication.
Change-Id: I44068c253840a34e3c21be2bd03b7569df1c3b98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1718
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-13 16:38:58 +00:00
multi
5e58c8bc28 feat(whitby): add multi
Change-Id: Ibfc2a5fcf73099b8414b8c46958007374d14fd0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1701
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-08-09 19:42:07 +00:00
multi
c0baddeed0 feat(tvl-slapd): add multi
Change-Id: I2fec9b5dd92da0343426c4a129d882fa87d92e6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1699
Reviewed-by: cynthia <cynthia@tvl.fyi>
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-08-09 15:44:04 +00:00
Vincent Ambo
81f09b2dcd fix(whitby): Increase nrBuildUsers to 128
Change-Id: I3a444e163745d17d10f923c0be7565840937c53a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1662
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 00:21:23 +00:00
Vincent Ambo
bc1293b944 fix(whitby): I'm a trusted user, owo
Change-Id: I2666b3cf8bdefcb5d4caeddf191dc65f6a8cb05f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1661
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 00:21:23 +00:00
edef
6a128fc162 chore(whitby): add edef
Change-Id: I7265259bc87594bd481c7bd455187c09b1effd1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1650
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-05 22:32:54 +00:00
edef
c9a645b69d chore(tvl-slapd): refresh edef's password
Change-Id: I5a7a913656bfb9dd6c9fb4e2b4a1212607c50dd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1592
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-03 11:15:23 +00:00
Vincent Ambo
cc3c45f739 fix(whitby): Move Restic's cache into /var/backup/restic
It tries to write this to ~/.cache otherwise, which worked for the git
user but does not work for root (??)

Change-Id: I02d04da7d8e2b8782ce70bc72bce0b90c3961aa0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1546
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 22:28:50 +00:00
Vincent Ambo
cc8033f592 fix(whitby): Make timer unit match the unit it should start
Oversight in the previous CL.

Change-Id: I8767322d7d860fc410796f8d63b7a6c38a8ab447
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1545
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 22:25:05 +00:00
Vincent Ambo
09b3d20031 feat(whitby): Include PostgreSQL dumps in Restic backups
Changes the restic backup service to run as root, rather than git, and
include the PostgreSQL dumps in its scope.

The on-machine credentials have already been placed in the right
location in /var/backup/restic

Fixes: 27
Change-Id: Iae76357442f07596a2297ce7b6d51aae392d2074
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1541
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 21:50:15 +00:00
Vincent Ambo
717d12f2f3 feat(whitby): Enable daily PostgreSQL backups
... daily is just the default cron pattern for this, but we might also
want this to happen more frequently. Not sure yet.

Change-Id: I4e433fefebd93488891e765b5842fdb6537e3c6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1518
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-01 16:52:00 +00:00
Vincent Ambo
1fe4a47aa2 fix(ops/paroxysm): Ensure paroxysm is started on boot
Change-Id: Iba6557cbf4e0001277bd996df59318b4308fc92e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1510
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-01 02:46:20 +00:00
Vincent Ambo
f41324db8c feat(ops/nixos): Add module for running paroxysm on whitby
Change-Id: I415e3b046d4e0fcd7e800ddab0c7f1aeb639c5e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1502
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
2020-07-31 21:58:34 +00:00
Griffin Smith
e8f893ee10 refactor(web/panettone): Remove prevalence
Now that we've migrated over all the data to postgresql, we can get rid
of cl-prevalence as a dependency from Panettone along with all code that
mentions it.

Change-Id: I945f50a88fea5770aac5b4a058342b8269c0bea2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1495
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-29 01:57:49 +00:00
Griffin Smith
d9262bd6c6 feat(ops/nixos): Use database password for Panettone
It appears this didn't even *work* without a password, so we've been
forced into being more secure.

Change-Id: I4ff9d04961a703a85299dafb79e8447b0a933fc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1491
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 01:35:25 +00:00
Griffin Smith
9ae4ac8f50 fix(ops/nixos): allow connections on hostnossl
This is how panettone is currently connecting, so this needs to be here
in order for it to work. Shortly I'll update all of this to use
passwords, but for now this gets things up and running again

Change-Id: If87f4dbce0800dcbc4f7bf10e88f3e591410b416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1488
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 00:46:26 +00:00
Griffin Smith
69f402563a feat(whitby): Create a Postgres database for Panettone
Create a running Postgres database server along with a user and database
for Panettone, and pass configuration for it to the panettone module

Change-Id: I333994288131be328e62069382d6d40f8034c400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1466
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-27 21:04:50 +00:00
Luke Granger-Brown
c7cd12eb25 chore(monorepo-gerrit): add Checks plugin to Gerrit module
This adds the Gerrit checks plugin. Hooray.

Change-Id: I784e9728256d1665b85b666d58bc0308bd6614ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1463
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-27 00:00:48 +00:00
Luke Granger-Brown
3542e1f62b chore(whitby): add rxvt-unicode's terminfo
Otherwise I have to set TERM to something else so that I can actually use the
machine when I'm booted into Linux and it's incredibly tedious and I hate it.

Change-Id: Icfb5aacfea8cd6227743d29d9b07dc1b745d22c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1435
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-25 20:27:47 +00:00
Luke Granger-Brown
b8fa8ff4a4 chore(monorepo-gerrit): enable attention set, disable assignee, disable polygerrit CDN
Change-Id: I66c09afc0813e032a1b5a04cbdbe4b95db2e97d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1438
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-25 20:27:13 +00:00
Bartosz Stebel
32c3f7731b feat(tvl-slapd): add implr
Change-Id: I7d22bf61ac72e86a17035d6125055da8aa53d762
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1387
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-23 22:19:23 +00:00
Griffin Smith
7101cc5375 feat(ops/nixos): Deploy Panettone to Whitby
Deploy Panettone to whitby as a systemd service, proxied to from an
nginx virtual host listening at b.tvl.fyi

Change-Id: I69755566151a45120e6b3453751af0e9291fa241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1339
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-23 22:09:15 +00:00
Vincent Ambo
37cc98d078 fix(whitby): Use fish shell as my default shell
I don't have time for bash's history.

Change-Id: I741107d33f09999ef43a7609079ad926e8127e69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1362
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-23 19:39:07 +00:00
Vincent Ambo
b422688da1 feat(whitby): Add SSH key for qyliss
... also bootstraps her user directory to store the key in.

Change-Id: Iecd341c655adc7d81be5ce9eb765c531b7512e80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1361
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2020-07-23 19:37:09 +00:00
Vincent Ambo
cbad0991de chore(whitby): Move isomer's SSH key to user directory
This is inline with how other user keys are managed.

Change-Id: Ica0b3b30336aee02a78e019b13e1cf576e4e1943
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1360
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-23 19:32:15 +00:00
Vincent Ambo
406e37fde1 feat(whitby): Deploy todo.tvl.fyi page with //web/todolist
Note that this is not yet updated automatically, so the page will be
stale until somebody rebuilds whitby.

Change-Id: I91f4b03c9309aed289df055fac292a214dca7668
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1297
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
2020-07-19 23:40:42 +00:00
Kane York
3c9ee24929 chore(whitby): add riking
Change-Id: I33cc1324eac9a13be56d296d09cfdbe066d90e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1256
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 21:21:10 +00:00
Alyssa Ross
effbb277c3 chore(tvl-slapd): add display name for qyliss
Not having this set led to gerrit setting the committer to
"qyliss <hi@alyssa.is>", which is wrong.

Change-Id: I3fe02264e22dd6d739575b34ceb1221d1d6a9d98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1267
Tested-by: BuildkiteCI
Reviewed-by: qyliss <hi@alyssa.is>
2020-07-18 16:50:07 +00:00
Kane York
501d6bdaab chore(tvl-slapd): change display name to a username-like
Change-Id: I289400de6638844586a32a729333cb65a0dca4a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1254
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 01:23:13 +00:00
Alyssa Ross
0f7bdd6711 feat(tvl-slapd): add qyliss
Change-Id: Ia95c77be8a9c123f2e52174f76c4b01d44272191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1260
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 22:53:17 +00:00
Vincent Ambo
67f0fbfcea feat(whitby): Hardcode Google DNS servers
The Hetzner DNS servers were unhappy after today's Cloudflare outage,
and that broke some of our builds - this wouldn't have happened with
Google DNS!

Change-Id: Ib74c6de9526e739f55d4a9830d945ece35b72138
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1259
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 22:27:49 +00:00
isomer
cbff4fab9e chore(whitby): += Isomer
Change-Id: I446ab16d009dc24340606ab2f411197af24d79c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1142
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-17 19:38:51 +00:00
Vincent Ambo
356dde149f feat(whitby): Configure Gerrit backups on whitby
Change-Id: I84245fb809725853a301f217cdb11eacc1984cae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1103
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-12 14:11:24 +00:00
Vincent Ambo
e035c46c6e chore(whitby): Give the git user a home directory
Change-Id: I5e6e13fa8a1656434ca897c83fe7ac48eb869369
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1102
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-12 13:51:20 +00:00