Commit graph

54 commits

Author SHA1 Message Date
Vincent Ambo
afc07b1b74 chore(whitby): Double the number of build agents again
The main bottleneck of our builds right now is Nix evaluation, which
means that most of the time is spent idling during builds.

Since we're evaluating in parallel, lets give it a few more builders.

I don't want to go all the way to 64 immediately to first see if we
get any adverse effects from highly concurrent builds running
concurrently (if we do we could group them into different "concurrency
groups" in Buildkite).

Change-Id: Ibc3f89fb59cb4ee471b152ff36887ffe2b39f8f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1932
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-06 14:42:40 +00:00
Vincent Ambo
2afa22e512 feat(whitby): Double the number of Buildkite agents
Allow 16 things to happen in parallel, which is useful now that the
CI granularity is on a per-target level.

Change-Id: Ie65dd119ea0666618fbb249613e70a68276db834
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1902
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-31 23:21:16 +00:00
Vincent Ambo
d4824b64f6 feat(whitby): Enable log forwarding via journaldriver
Change-Id: I474159acfe514f6f2eb7867e4eba854016590ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1836
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-08-23 20:45:18 +00:00
tazjin
674d921202 revert(whitby): Use Tvix as the system Nix on whitby
This reverts commit 8fa3087067.

Reason for revert:

This almost worked. We discovered two important issues:

- The daemon startup does not correctly handle the socket passed in by systemd.

- There is some issue with chunking of large calls, running a build for ci-builds resulted in:

tazjin@whitby /depot (canon)> nix-build -A ciBuilds.__allTargets
E20200821 01:42:22.846053 12601 shared.cc:306] error: Rpc call addTextToStore to unix:///nix/var/nix/daemon-socket/socket failed (RESOURCE_EXHAUSTED): Received message larger than max (10889961 vs. 4194304)

Change-Id: Ic5ba4ef06a4953cf71a36b139fe25ea673cb6fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1802
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 02:01:33 +00:00
Vincent Ambo
8fa3087067 feat(whitby): Use Tvix as the system Nix on whitby
... this is going to break so much stuff. Lets have some fun.

Change-Id: If0185e0323391c7055d47b797083bb5afde57cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1829
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 01:37:25 +00:00
V
b8fe5469c6 feat(whitby): add V
Change-Id: I887760edd67135df4e2f58a874314b317838d2e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1787
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-19 01:08:33 +00:00
eta
737d192aec feat(whitby): add eta
Change-Id: I7aa2bd2cb2c001b48ebd25b20f28cdfb0883ba3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1782
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-17 22:27:25 +00:00
Vincent Ambo
a7868e4e17 feat(nixos/clbot): Add ability to post in multiple channels
Adds the ability to post to multiple channels by simply running
multiple instances of clbot.

We should probably implement support for this in clbot itself, but
right now I can't be bothered to write Go.

Change-Id: I5cffd0dc10a7f6cc19c37c5834c5610166b4ae23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1771
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-17 21:50:45 +00:00
multi
9108256ce4 feat(whitby): enable programs.mosh.
Change-Id: Ibc8df6f6382b5b64e272bedece6b65762f9693c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1750
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-14 21:08:27 +00:00
multi
8af65f6858 fix(whitby): disable sshd(8) password authentication.
Change-Id: I44068c253840a34e3c21be2bd03b7569df1c3b98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1718
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-13 16:38:58 +00:00
multi
5e58c8bc28 feat(whitby): add multi
Change-Id: Ibfc2a5fcf73099b8414b8c46958007374d14fd0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1701
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-08-09 19:42:07 +00:00
Vincent Ambo
81f09b2dcd fix(whitby): Increase nrBuildUsers to 128
Change-Id: I3a444e163745d17d10f923c0be7565840937c53a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1662
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 00:21:23 +00:00
Vincent Ambo
bc1293b944 fix(whitby): I'm a trusted user, owo
Change-Id: I2666b3cf8bdefcb5d4caeddf191dc65f6a8cb05f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1661
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 00:21:23 +00:00
edef
6a128fc162 chore(whitby): add edef
Change-Id: I7265259bc87594bd481c7bd455187c09b1effd1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1650
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-05 22:32:54 +00:00
Vincent Ambo
cc3c45f739 fix(whitby): Move Restic's cache into /var/backup/restic
It tries to write this to ~/.cache otherwise, which worked for the git
user but does not work for root (??)

Change-Id: I02d04da7d8e2b8782ce70bc72bce0b90c3961aa0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1546
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 22:28:50 +00:00
Vincent Ambo
cc8033f592 fix(whitby): Make timer unit match the unit it should start
Oversight in the previous CL.

Change-Id: I8767322d7d860fc410796f8d63b7a6c38a8ab447
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1545
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 22:25:05 +00:00
Vincent Ambo
09b3d20031 feat(whitby): Include PostgreSQL dumps in Restic backups
Changes the restic backup service to run as root, rather than git, and
include the PostgreSQL dumps in its scope.

The on-machine credentials have already been placed in the right
location in /var/backup/restic

Fixes: 27
Change-Id: Iae76357442f07596a2297ce7b6d51aae392d2074
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1541
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 21:50:15 +00:00
Vincent Ambo
717d12f2f3 feat(whitby): Enable daily PostgreSQL backups
... daily is just the default cron pattern for this, but we might also
want this to happen more frequently. Not sure yet.

Change-Id: I4e433fefebd93488891e765b5842fdb6537e3c6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1518
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-01 16:52:00 +00:00
Vincent Ambo
f41324db8c feat(ops/nixos): Add module for running paroxysm on whitby
Change-Id: I415e3b046d4e0fcd7e800ddab0c7f1aeb639c5e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1502
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
2020-07-31 21:58:34 +00:00
Griffin Smith
d9262bd6c6 feat(ops/nixos): Use database password for Panettone
It appears this didn't even *work* without a password, so we've been
forced into being more secure.

Change-Id: I4ff9d04961a703a85299dafb79e8447b0a933fc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1491
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 01:35:25 +00:00
Griffin Smith
9ae4ac8f50 fix(ops/nixos): allow connections on hostnossl
This is how panettone is currently connecting, so this needs to be here
in order for it to work. Shortly I'll update all of this to use
passwords, but for now this gets things up and running again

Change-Id: If87f4dbce0800dcbc4f7bf10e88f3e591410b416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1488
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 00:46:26 +00:00
Griffin Smith
69f402563a feat(whitby): Create a Postgres database for Panettone
Create a running Postgres database server along with a user and database
for Panettone, and pass configuration for it to the panettone module

Change-Id: I333994288131be328e62069382d6d40f8034c400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1466
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-27 21:04:50 +00:00
Luke Granger-Brown
3542e1f62b chore(whitby): add rxvt-unicode's terminfo
Otherwise I have to set TERM to something else so that I can actually use the
machine when I'm booted into Linux and it's incredibly tedious and I hate it.

Change-Id: Icfb5aacfea8cd6227743d29d9b07dc1b745d22c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1435
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-25 20:27:47 +00:00
Griffin Smith
7101cc5375 feat(ops/nixos): Deploy Panettone to Whitby
Deploy Panettone to whitby as a systemd service, proxied to from an
nginx virtual host listening at b.tvl.fyi

Change-Id: I69755566151a45120e6b3453751af0e9291fa241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1339
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-23 22:09:15 +00:00
Vincent Ambo
37cc98d078 fix(whitby): Use fish shell as my default shell
I don't have time for bash's history.

Change-Id: I741107d33f09999ef43a7609079ad926e8127e69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1362
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-23 19:39:07 +00:00
Vincent Ambo
b422688da1 feat(whitby): Add SSH key for qyliss
... also bootstraps her user directory to store the key in.

Change-Id: Iecd341c655adc7d81be5ce9eb765c531b7512e80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1361
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2020-07-23 19:37:09 +00:00
Vincent Ambo
cbad0991de chore(whitby): Move isomer's SSH key to user directory
This is inline with how other user keys are managed.

Change-Id: Ica0b3b30336aee02a78e019b13e1cf576e4e1943
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1360
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-23 19:32:15 +00:00
Vincent Ambo
406e37fde1 feat(whitby): Deploy todo.tvl.fyi page with //web/todolist
Note that this is not yet updated automatically, so the page will be
stale until somebody rebuilds whitby.

Change-Id: I91f4b03c9309aed289df055fac292a214dca7668
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1297
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
2020-07-19 23:40:42 +00:00
Kane York
3c9ee24929 chore(whitby): add riking
Change-Id: I33cc1324eac9a13be56d296d09cfdbe066d90e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1256
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 21:21:10 +00:00
Vincent Ambo
67f0fbfcea feat(whitby): Hardcode Google DNS servers
The Hetzner DNS servers were unhappy after today's Cloudflare outage,
and that broke some of our builds - this wouldn't have happened with
Google DNS!

Change-Id: Ib74c6de9526e739f55d4a9830d945ece35b72138
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1259
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 22:27:49 +00:00
isomer
cbff4fab9e chore(whitby): += Isomer
Change-Id: I446ab16d009dc24340606ab2f411197af24d79c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1142
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-17 19:38:51 +00:00
Vincent Ambo
356dde149f feat(whitby): Configure Gerrit backups on whitby
Change-Id: I84245fb809725853a301f217cdb11eacc1984cae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1103
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-12 14:11:24 +00:00
Vincent Ambo
e035c46c6e chore(whitby): Give the git user a home directory
Change-Id: I5e6e13fa8a1656434ca897c83fe7ac48eb869369
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1102
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-12 13:51:20 +00:00
Vincent Ambo
405b7ec95b feat(whitby): Enable Gerrit & cgit deployments
Change-Id: Ic701552e130252cfff005938d9c4e98423a7a96a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1069
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-12 13:36:45 +00:00
Vincent Ambo
93575158c6 feat(whitby): Enable SourceGraph server
Change-Id: Ia8a20d54a4ac77d64f5e3fd2255ffad78dce0fb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1067
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-12 13:36:45 +00:00
Vincent Ambo
90b8433828 feat(nixos/www): Add configuration for tvl.fyi homepage
... and enable it on whitby

Change-Id: Ife45f15227f9d95823ebd3b97d2a17175b84eaff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1064
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-12 13:36:45 +00:00
Vincent Ambo
31f65f5d2b feat(whitby): Move over clbot deployment from camden
There is only one minor configuration change: CLBot now connects to
cl.tvl.fyi, instead of localhost, because Gerrit is still on camden.

Change-Id: Ibd8d46ec2c18312a270471a2f0be3e58eaf0cbab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1062
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-11 12:20:17 +00:00
Vincent Ambo
ea428faf99 feat(whitby): Enable smtprelay module
This is required for the Gerrit setup.

Change-Id: I02e03dafe36e6c47ffabf4d590e0c6f1dea027e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1061
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-11 11:58:43 +00:00
Luke Granger-Brown
57ade16b9d feat(whitby): add apereo-cas/tvl-sso
Change-Id: I29f5e762852593f05b9936d5635aadcc7eba283e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/935
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-06 22:59:56 +00:00
Luke Granger-Brown
aae3d25234 feat(ops/nixos/www): create login.tvl.fyi host
Change-Id: Ifad80915a61a1a5ac14e598a9d788aec3482693c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/936
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-06 22:48:40 +00:00
Griffin Smith
a73714a93c feat(ops/nixos): Add generic rebuild-system script
This adds a first crack at one idea for a generic, non-user-specific
rebuild-system script to ops.nixos.rebuild-system. The idea here is that
we enumerate all the nixos systems stored in the monorepo (similarly to
what we do for ci-builds right now) then search through them by hostname
to find the one matching the hostname of the current system, which is an
attempt at a more generic version of tazjin's rebuilder script which
does the same thing but with an explicit case block.

As a caveat, it feels like there's a slight possibility that this way of
finding systems is going to get slow to evaluate - on my system it feels
fine but if it grows out of hand it's probably feasible to just bake
this into the built script as a dynamically generated case statement.

Change-Id: I2e4c5401913b6f4d936ab48ba2f95f96e0e78eb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/894
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-06 15:16:36 +00:00
Luke Granger-Brown
25cebc3a62 feat(whitby): enable tvl-slapd on whitby
Change-Id: I3fac108802671abfb9a508359390b063bce16202
Reviewed-on: https://cl.tvl.fyi/c/depot/+/923
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-05 16:54:48 +00:00
Luke Granger-Brown
f54a48f831 chore(whitby): add lukegb to trusted-users for remote builds
Change-Id: Id1e67bb30bb7f4d329006688f1783b900d16d164
Reviewed-on: https://cl.tvl.fyi/c/depot/+/914
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-04 21:27:27 +00:00
Vincent Ambo
3ce41f4fa4 feat(whitby): Enable nix.sshServe
This exposes a binary cache over SSH.

Change-Id: Ib934a118cd7315ef76f3dfe795c76a570fbbc47a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/895
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: BuildkiteCI
Tested-by: BuildkiteCI
2020-07-03 14:25:35 +00:00
Griffin Smith
0d4f709757 feat(whitby): Allow wheel users to sudo without a password
This *should* translate to the required invocation to make sudo allow
nopasswd for users in the wheel group.

Change-Id: I3713862b8df9087cfbaa72d7e824bc43469f7c1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/857
Reviewed-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-02 22:00:41 +00:00
Griffin Smith
1ecae26afa feat(whitby): Add grfn as a trusted user
So I can remote builder

Change-Id: I8106244d3d197c010b618e4337a9ccfc13a116f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/856
Reviewed-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-02 21:36:11 +00:00
Vincent Ambo
4d9c6dbbe2 feat(whitby): Run a handful of Buildkite agents
This is the point of the machine, afterall.

Change-Id: I15c11600c1c18fa8962d57f75f99a72e1553f9c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/853
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: BuildkiteCI
Tested-by: BuildkiteCI
2020-07-02 20:54:11 +00:00
Vincent Ambo
7114e72d81 feat(whitby): Enable Nix signing for the binary cache
Change-Id: I9047667cc1a40668c0c7da72c070044b91b53014
Reviewed-on: https://cl.tvl.fyi/c/depot/+/852
Reviewed-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-02 20:54:11 +00:00
Vincent Ambo
cabdd4aa51 fix(whitby): Explicitly set an interface for the v6 default gw
systemd gets sad otherwise and it is very difficult to console it

Change-Id: Ic6405489532c407273e5634474185f2947420b37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/851
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: BuildkiteCI
Tested-by: BuildkiteCI
2020-07-02 20:54:11 +00:00
Griffin Smith
298060dba9 feat(whitby): Add grfn
it's not glittershark because grfn is the username I have on my laptop
and I want to be able to ssh without an `@`.

Change-Id: Ie1fb6f5e12f3ac52a44680704179bd27a00a7768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/850
Reviewed-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-02 20:28:48 +00:00