Commit graph

574 commits

Author SHA1 Message Date
Vincent Ambo
5baa9b6d87 refactor(tvl-buildkite): Prepare gerrit credentials helper
Currently this functionality is provided by a shell script stored in
/etc/secrets (which has the password value hardcoded).

This needs to happen in a separate commit from the one that changes
the pipeline to avoid breaking it (it needs to be deployed first).

Change-Id: I680754c828ccefbacfcf0d5c813a4bc19493ba4c
2021-12-10 19:52:39 +00:00
Vincent Ambo
2fe8d724d7 refactor(ops): Move Nix cache secret to agenix
... and also the public key, just to keep the distribution mechanism
the same.

Change-Id: Ief14daf9344c0fb99eeb5789c1ec9bfb1f12bee0
2021-12-10 19:48:26 +00:00
Vincent Ambo
82a885a750 refactor(ops): Use besadii configuration from agenix
We already checked this in, but this commit adds the configuration for
making use of it.

There are two copies of besadii's JSON configuration with different
permissions.

Note that the buildkite-graphql-token path needs to be updated in
static-pipeline.yml, but this needs to happen in a separate commit
after deploy because the pipeline will break otherwise.

Change-Id: I6fab4bf1a2e679df7cf76521e2b53bd9dadbac62
2021-12-10 19:31:36 +00:00
Vincent Ambo
b1108821a9 refactor(ops): Move grafana secret into agenix
Change-Id: Id141758135c796881e91d20b950dae74c40d9ab3
2021-12-10 19:31:36 +00:00
Vincent Ambo
b2d46aed2b fix(tvl-buildkite): Add more missing programs to agent path
... this option really is a pitfall! The list of programs is now the
same as in the upstream module, plus curl and jq.

Change-Id: I29edae4b2400a2724f62df9efa1dc184a8b0af5f
2021-12-10 17:13:22 +00:00
Vincent Ambo
b8267c261c fix(ops/irccat): Avoid permissions issue with LoadCredentials=
The DynamicUser + Group configuration does not work as planned, thus
the systemd LoadCredentials feature is used instead which makes the
file (which itself is only readable by root) available in a
memory-backed location only readable by the service.

The secret is only available to `ExecStart` commands, so units using
this feature can not be used with pre/post units and the like if those
commands need secrets.

To accommodate this, the merge of configuration files has been moved
into the service launch script, which is now the ExecStart= process.

For details take a look at https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LoadCredential=ID:PATH

Change-Id: I693fe5677cc0d63c7aa485c2c7472457c5262166
2021-12-10 15:09:09 +00:00
Vincent Ambo
67bde5ecc3 fix(tvl-buildkite): Explicitly set runtimePackages
It turns out the lib.mkAfter call doesn't behave as expected -
only *some* of the packages that are defaulted end up in the $PATH.

I suspect this is actually something else, e.g. these packages are
always added for some reason or another, and the option is completely
overridden every time.

Change-Id: I854c7198520d82b00e6338ed0fe653836226dc6d
2021-12-10 15:06:08 +00:00
Vincent Ambo
2ba481451c chore(ops/secrets): Reencrypt with grfn's key included
Change-Id: I66df150ab5070a81a92f0741334639df9df1f86f
2021-12-10 17:52:08 +03:00
Griffin Smith
a85ab68b12 chore(ops/users): Rotate password for grfn
Just a regular password rotation, plus I wasn't using argon2 unlike
everyone else.

Change-Id: Ic57fe79a2dbfdc15397d20f6b2b47c6aac911d29
2021-12-10 09:45:17 -05:00
Griffin Smith
66a1d3d5d4 feat(ops/secrets): Add key for grfn
Change-Id: I8063ae804932e3815e9a499e0206806818b9b021
2021-12-10 09:44:34 -05:00
Vincent Ambo
bc3d35f3d0 fix(tvl-buildkite): Add missing runtimePackages back
Turns out that the type of this option is not concatenative and it
replaces the packages needed to run Buildkite if set.

Change-Id: I9f52572bc165bccdd8c6518cfdf7b8967f7a50d0
2021-12-10 13:14:11 +00:00
Vincent Ambo
d4403638cf refactor(ops): Move irccat secret into agenix
The irccat module uses DynamicUser, so to grant permission to it a new
group has been added for irccat.

I have some vague memory of DynamicUser + Group not behaving as one
would expect, but we'll see what happens.

Change-Id: Iab9f6a3f1a53c4133b635458ce173250cc9a3fac
2021-12-10 16:13:31 +03:00
Vincent Ambo
002d183876 refactor(ops): Move clbot SSH key into agenix
Change-Id: Iae03ead7dda0509689a76f0d76f9cfeb8434e967
2021-12-10 16:13:31 +03:00
Vincent Ambo
811e6d7d9f chore(whitby): Remove shadowsocks service
No longer required on whitby.

Change-Id: I93951c6b708eae81ddb03df920a4068c1ccde9e7
2021-12-10 13:07:09 +00:00
Vincent Ambo
fc14c21bb9 fix(ops/pipelines): Move to static pipeline
This step would get inserted at the wrong point in the build pipeline
otherwise, causing a dependency cycle and causing the pipeline to fail.

Change-Id: I534568eec77f74ae6c47276820f8a9e99493a3ea
2021-12-10 11:01:21 +03:00
Vincent Ambo
e4231c9816 refactor(ops/pipelines): Move 🦆 logic into static pipeline
This simplifies the fallback logic used in case of Nix evaluation
failure and makes it so that the evaluation step itself is the one
that is marked as failed in Buildkite.

This is possible because the pipeline upload command will insert new
steps at the point where it runs in the pipeline, and not later.

Change-Id: I870534c004ebc457a1602623c4e5f9c0c68e28fc
2021-12-10 07:55:34 +00:00
Vincent Ambo
9ea4d55d81 refactor(ops): Move buildkite-agent-token into agenix
Relates to b/161

Change-Id: I5d3a698d437928966d8b78ce9e0ba226c1437655
2021-12-10 10:32:44 +03:00
Vincent Ambo
a123b9e0a2 refactor(ops): Move owothia secret into agenix
Relates to b/161

Change-Id: I25445281b0dd3c3f3660f8bb0d8337506a1e427b
2021-12-10 10:32:14 +03:00
Vincent Ambo
78744c00f5 refactor(ops): Move clbot secret into agenix
Relates to b/161

Change-Id: I7badf22ff93bb4e8b06e4dd4a8bf880b0bd48f09
2021-12-10 10:32:14 +03:00
Vincent Ambo
496d899428 feat(ops/secrets): Configure secrets for gerrit-queue
Adds a systemd EnvironmentFile secret that contains the Gerrit
username & password for gerrit-queue.

Change-Id: I25acf87764c26774045138402b8a417b6813ee8f
2021-12-10 10:32:14 +03:00
Vincent Ambo
4870b1a2ff feat(ops/modules): Add module for running gerrit-queue
This is not yet including the secret configuration for gerrit-queue,
and just expects the secret (gerrit username & password) to be
available in /etc/secrets.

Change-Id: Ia465ef7f3f521c70d606d7fdeba9aa83c7e1b98b
2021-12-10 10:32:14 +03:00
Vincent Ambo
a9dd719e7c chore(tvl-buildkite): Add jq and curl to agent paths
This is required for a simplification of the build pipeline (following
CL) and needs to be in a separate commit as it can not be done
atomically (merging the other commit to deploy it would immediately
break pipelines otherwise).

Change-Id: I5d8ec8f3238f79b5518d799486bf98d1d9516c43
2021-12-10 10:21:34 +03:00
Vincent Ambo
f1e1f71883 feat(ops/secrets): Bootstrap agenix secrets folder
Sets up the key set and adds an initial secret (besadii config with
tokens) to be deployed to whitby.

Change-Id: Ic07fd5e66b9e7a533013e04c35e052c2aa11f77d
2021-12-08 18:22:00 +00:00
Vincent Ambo
c1479a6221 chore(besadii): Improve error messages on parse failure
Change-Id: I3cc4637aca8a940a0fdeca2d8bd6ac620ea384c0
2021-12-07 18:27:44 +00:00
Vincent Ambo
8a944484f0 fix(ops/besadii): Unquote Gerrit's extra-quotes around emails
Gerrit wraps RFC5322 emails in another layer of quotes when passing
them as flags, and this needs to be unquoted.

Otherwise hook invocations fail with cryptic errors.

Change-Id: Ieeb74c662873d99a4154f8cbc92da77b039cb88e
2021-12-07 18:27:44 +00:00
Vincent Ambo
6faf0edaff fix(ops): Correctly pass command name to besadii invocations
Ensure that besadii sees $0 as the correct command name, since that is
the sole mechanism by which its functionality is switched around.

There was a lingering commit that introduced this bug and hadn't been
deployed in a couple of days. Maybe time to tighten deploy cycles soon
...

Change-Id: Ie4284c0f6e5e06d71a71a3702ec7e092260e0ce5
2021-12-07 18:27:44 +00:00
Vincent Ambo
7f2f5d07f2 fix(ops/besadii): Pass Build.Author to Buildkite
Extracts author information from the flags passed by Gerrit and moves
them along to Buildkite. This should display the owners of builds
correctly in the UI, rather than marking everything as coming from me.

Change-Id: If9efe5553a13f0dbdb8bf3936c1d341ae5922318
2021-12-06 17:42:57 +03:00
Vincent Ambo
b679bb4034 refactor(ops/besadii): Get config from home directory by default
Slightly more ergonomic in some setups.

Change-Id: I565f2d242852ffd299ef5d5740a47520187dd4b4
2021-12-02 17:13:49 +03:00
Vincent Ambo
dcb2410982 refactor(ops/besadii): Generalise for use with non-TVL URLs
This makes it possible to use besadii for any TVL-ish setup using
Gerrit and Buildkite, with the same hook functionality as for TVL.

Change-Id: I1144b68d7ec01c4c8e34f7bee4da590f2ff8c53c
2021-12-02 13:10:21 +03:00
Vincent Ambo
e48ae26e8e feat(ops/besadii): Add other missing configuration keys
Adds configuration keys and rudimentary validation for all other
besadii settings that are currently hardcoded.

This adds the config options:

* repository: Name of the repository in Gerrit.
* branch: Name of the HEAD branch in the repository.
* gerritUrl: Base URL of the Gerrit instance
* gerritUser: Username of the Gerrit user
* gerritPassword: Password of the Gerrit user
* buildkiteOrg: Name of the Buildkite organisation
* buildkiteProject: Name of the pipeline inside the Buildkite
  organisation
* buildkiteToken: Auth token for Buildkite access

All of these configuration options are required.

Change-Id: Ie6b109de9cd8484a3773c6351d7fd140f39a49ed
2021-12-02 13:10:21 +03:00
Vincent Ambo
ee635d4645 chore(ops/modules): Configure besadii call sites to load config
On whitby, the besadii config will live in
/etc/secrets/besadii.json. This CL updates the call sites to pass this
config path to besadii so that it can load Sourcegraph configuration.

Change-Id: Ia139b9fa3b827e7a5f2386214390acc6fe19a75a
2021-12-02 13:10:20 +03:00
Vincent Ambo
168114df52 refactor(ops/besadii): Move Sourcegraph config to a file
Initial step towards moving besadii away from hardcoded values and
onto config files. This is required because I want to reuse besadii
outside of the TVL context.

Change-Id: Id4fa7a49c5d4f876a02b202f04a421ab5ba0dcc4
2021-12-02 13:10:20 +03:00
Vincent Ambo
c53d6d3453 fix(ops/nixery): Temporarily stop serving depot packages in Nixery
Change the Nixery configuration to use the plain nixpkgs package path
instead of the depot path. AFAIK, nobody uses this to fetches depot
packages at the moment - but plenty of people fetch non-depot
packages.

This means that Nixery is cache-busted less often (previously on every
commit => every deploy).

We'll figure out another way to have a depot Nixery later.

Change-Id: Iba632333346181c3d2ce992fbab396ed0d9f86aa
2021-12-02 09:16:52 +03:00
Vincent Ambo
433f0ae5cd fix(ops/www): Redirect tvl.fyi/blog -> tvl.fyi
The blog index page is at the root and people may manually edit the
URL.

Change-Id: I6cdaaaee6223524a9e950584379cfac34f8be160
2021-12-01 23:41:23 +03:00
Vincent Ambo
c1aab56a02 feat(besadii): Support invocation as different Gerrit hooks
Removes besadii support for the previously used 'ref-updated' hook and
instead introduces support for the 'change-merged' and
'patchset-created' hooks.

These hooks more accurately capture the semantics of when besadii
should trigger CI builds and using them will avoid problems such as
skipping 'canon' builds if chains of CLs are submitted together.

Change-Id: Ib90356c069780bf0c0250e56b927e46a5b31ce7f
2021-12-01 12:49:31 +03:00
Vincent Ambo
68d1f834a3 fix(ops/www): Strip .html from TVL blog post URLs
Change-Id: I4d1f9284ec004931c07c04d614b01f28eedea508
2021-11-30 13:56:29 +03:00
Vincent Ambo
6edfdd0773 refactor(ops/pipelines): Query build status from Buildkite API
Instead of manually tracking the build status through Buildkite
metadata, use the Buildkite GraphQL API in the `🦆` build
step (i.e. the one that determines the status of the entire pipeline
to be reported back to Gerrit) to fetch the number of failed jobs.

This way we have less manual state accounting in the pipeline.

The downside is that the GraphQL query embedded here is a little hard
to read.

Notes:

  * This needs an access token for Buildkite. We already have one for
    besadii which is also run by the agents, so I've given it GraphQL
    permissions and reused it.

  * I almost introduced a very rare bug here: My initial intuition was
    to simply `exit $FAILED_JOBS` - in the extremely rare case where
    `$FAILED_JOBS % 256 = 0` this would mean we would ... fail to fail
    the build :)

Change-Id: I61976b11b591d722494d3010a362b544efe2cb25
2021-11-29 23:38:24 +03:00
Vincent Ambo
4d57898af4 chore(ops/users): Update password hash for asmundo
... some issue snuck in on the first one, as is tradition.

Change-Id: I06ce4df82cde26231cd1ab3df500de02e981d9bc
2021-11-29 18:12:19 +03:00
asmundo
78f51edf8c feat(ops/users): Add user asmundo
Change-Id: Ie666b6556d91513babd884b2ed1140cd6c0ed2a9
2021-11-29 14:14:04 +00:00
Vincent Ambo
eca2bc572e refactor(besadii): Rename refUpdated -> buildTrigger
We are changing the Gerrit hooks which invoke besadii, but this
structure will be used for both kinds.

Change-Id: Idb1cb0c640d2c42db8e7af39f3ab372a97bfef91
2021-11-29 13:54:47 +00:00
Vincent Ambo
104f002a07 fix(ops/besadii): Trim whitespace of auth tokens
This is causing failures when trying to update Sourcegraph at least,
for good measure I've trimmed both.

Change-Id: I40266ee83b4e266ffe50f16bb365eb2e51952513
2021-11-28 10:34:36 +00:00
Vincent Ambo
4f1249e46f refactor(readTree): Move 'drvTargets' into readTree
This function is also generally useful for readTree consumers that
have the concept of subtargets.

Change-Id: Ic7fc03380dec6953fb288763a28e50ab3624d233
2021-11-23 14:42:08 +00:00
Vincent Ambo
15cb37f877 fix(ops/restic): Move whitby's backup to GleSYS object storage
Since GCP nuked us, the backups are now moving to GleSYS'
S3-compatible object storage.

This refactors the restic module to support S3-compatible storage
instead of GCP, and switches to the appropriate new secret paths.

The secrets were placed on whitby manually and I verified that the
backups work.

This fixes b/157

Change-Id: I6a9d2b0581967605ce736605a3befb44cdeae7e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3883
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-11-21 12:01:26 +00:00
Vincent Ambo
099f36e5ee fix(ops/pipelines): Fix tagging of commit revisions
It seems that shell variables don't work as expected inside the
Buildkite pipeline, so usage of variables has been removed.

We also don't echo the revision anymore because of that, but it does
still appear in the log of `git push`.

Change-Id: I124e3b09af896da898f2a78715ed371651a1c5f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3780
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-11-06 00:33:23 +00:00
Vincent Ambo
4b33401a36 refactor(ops/pipelines): Move revision tagging into static pipeline
This makes the revision number available much earlier (before the rest
of the pipeline runs, while Nix eval is happening) which should only
be a few seconds after a commit to canon.

It is also more readable in this shape.

Change-Id: Iccbb17dfef6afe68f54fda41e8d10c4dc52b08c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3775
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-11-05 14:24:53 +00:00
Vincent Ambo
00ae396eeb feat(ops/pipelines): Create revision numbers in CI
This automatically pushes a new ref at refs/r/$revision to Gerrit
whenever a CI run completes on canon.

Revision numbers can be fetched from Gerrit with this command:

    git fetch gerrit "refs/r/*:refs/r/*"

Note that this build step requires credentials to be provisioned on
the CI runner machine.

Change-Id: I37bb14346832f891240aa47bb55affaace3d5f21
2021-11-04 15:16:08 +01:00
Smitty' via Issues & Patches
8696726244 fix(ops/users): correct password hash for smitop
The previous hash had a weird salt length and a trailing newline.
This fixes it.

Change-Id: I1f03238181d0caad38e1f1dbc477356bc20fc32d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3689
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-10-05 23:52:14 +00:00
sterni
3f0de23d61 feat(ops/users): add smitop to users
Change-Id: I1fc67c0e33e1e1add8a4ea53c8c94e90e53d8bd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3687
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-10-05 22:20:51 +00:00
Vincent Ambo
0eef0e343f feat(whitby): serve static.tvl.{fyi|su} with max cache settings
The setup is explained in the comment, but TL;DR: Use the derivation
hash of static files to create permanent URLs.

Relates to b/151.

Change-Id: Ib1ca3a1a00c90a47f4bf39c29a8b4bbf5b215e7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3664
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-10-01 20:45:50 +00:00
Vincent Ambo
9f177062c7 feat(ops/dns): add static.tvl.{fyi|su}
This hostname can be used for hosting static assets with aggressive
caching for everything, or potentially CDNing stuff if we ever have
large things here.

Change-Id: I10afdad5eb08125d8d09108e9e099f5573362fe5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3663
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-10-01 15:59:53 +00:00
Vincent Ambo
ce575bf65b feat(whitby): Serve //corp/website on tvl.su
Change-Id: I21e1ddf9a32568cac8ad2595869ac8670867efa9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3658
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-10-01 15:24:35 +00:00
Vincent Ambo
94cebe41f3 chore(ops/git-serving): Remove josh state from whitby backups
As cschilling explained on cl/3563, there isn't actually anything in
this state that we *need* to persist. We're still keeping it in a
persistent directory on disk as this serves as an optimisation after
restarts of josh.

Change-Id: Ia88886792a5acac34508b5b8a669bd519ca033de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3631
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-24 16:14:30 +00:00
Vincent Ambo
0e3858b5e5 refactor(whitby): Move restic path configuration into modules
This lets each service declare their backup paths together with the
configuration for the service, which is a lot more sensible than what
we had before.

Fixes b/147

Change-Id: If76fe62639f4cc0e6fbb63a2959d584479d8f0fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3583
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-18 15:10:34 +00:00
Vincent Ambo
86a114ac45 fix(ops/restic): types.string -> types.str
I can never remember which is which.

Change-Id: I69b8235862b8c5b49030a74bfca25aaa113273b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3582
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-18 14:38:19 +00:00
Vincent Ambo
98e4d4b18f feat(besadii): Link to started builds in CL comments
This makes it easier to click through to a build from Gerrit after
submitting a CL.

Change-Id: Ic5c6eeb81c87bc4ea23c5c5ca25704434b081fd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3572
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-18 14:28:26 +00:00
Vincent Ambo
ae93094ebf refactor(besadii): Extract logic for posting review comments
Currently besadii only posts comments when builds succeed, but it
might be very useful to also have a link to a build when the build is
started.

This just shuffles code around. The only functional change is that the
`labels` field in the review input is marked as `omitempty`, as this
will not be needed when posting the build start comment.

Change-Id: Id4a43fad8817c9a15da02f01ab2b781d48b46978
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3571
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-18 14:28:26 +00:00
Vincent Ambo
b6957923ff docs(kontemplate): Remove mention of kontemplate website
This doesn't exist anymore.

Change-Id: I4535e056acba3bbc7bbd1e764a0b3043639b0877
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3570
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-09-18 12:35:47 +00:00
Vincent Ambo
80fb67a75d docs(kontemplate): Update cloning docs in README
Change-Id: I42bf2524650bf09104e48c1c1a54c97f3470b628
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3566
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2021-09-16 20:34:05 +00:00
Vincent Ambo
387c55582b refactor(ops/restic): Move restic configuration into a new module
Relates to b/147.

First step towards giving depot modules the ability to declare their
own backup directories by moving all restic configuration into a new
module and adding a NixOS option for inclusion/exclusion paths for
backups.

This still keeps all backup paths within the whitby config.

Change-Id: Ia96833668f1a3d02da892261153d8b02156b8ac0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3565
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-09-16 20:34:05 +00:00
Vincent Ambo
ec38839c33 feat(git-serving): Configure josh to serve the depot over HTTP
Previously we served the dumb git HTTP protocol from code.tvl.fyi via
cgit. This CL disables this feature and instead runs josh in the same
location (by redirecting appropriately), but while also enabling
partial cloning of all subtrees of the depot.

For example, after this CL the following would result in an
independent clone of //nix/readTree:

    git clone https://code.tvl.fyi/depot.git:/nix/readTree.git

Note that there are no josh workspaces configured at all for now,
these references are only for static depot subpaths.

Please refer to the documentation for josh for more information on
available kinds of josh filters.

Josh state is kept in a systemd state directory in /var/lib/josh and
backed up to Restic. Backing this up is necessary, as josh uses
stateful information to do things like tracking merges and rewriting
history per subtree appropriately to avoid cloned repositories ending
up in peculiar states.

Change-Id: I156f0298c2aa42e3bdbf5a0e86109070d640c56e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3563
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-09-16 20:34:05 +00:00
Vincent Ambo
48091a3416 fix(deploy-whitby): Add jq to script $PATH
Change-Id: Ide669bce545394335b8643fa2896a242cac3df65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3528
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 14:33:42 +00:00
Vincent Ambo
933edf7764 fix(deploys.*): Folder for diffs is in /diff/
... this was missing before.

Change-Id: I5b79cb78665f24fdb7cc6496e3782d3940dc77b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3527
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 14:33:42 +00:00
Vincent Ambo
f5f0b80843 feat(sourcegraph): Upgrade 3.30.4 -> 3.31.2
This one seems a little more involved:
https://docs.sourcegraph.com/admin/migration/3_31

I believe we skip that corruption issue in the previous CL though, by
simply never deploying a version with that weird broken image.

See b/144

Change-Id: I3bbf1b719d00905e08a92011ace5485467f504ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3525
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 14:11:52 +00:00
Vincent Ambo
4d0eb5037a feat(sourcegraph): Upgrade 3.29.1 -> 3.30.4
See b/144

Change-Id: Ied9490f3ce6fb3fda8cbb9983416b02ea451fb44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3524
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 14:08:36 +00:00
Vincent Ambo
90971e07a1 feat(sourcegraph): Upgrade 3.28.0 -> 3.29.1
See b/144

Change-Id: Ia62d4cbf581caaefa0dba455376eec60b8c817d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3523
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 14:04:50 +00:00
Vincent Ambo
c148f89251 fix(sourcegraph): Temporarily comment out our syntax highlighter
We changed away from the default sourcegraph one because it didn't
support Nix, but it seems that there's been a change in the
interaction protocol.

Change-Id: I3a2691df6a87672cf83b819143f25d93d9cd6d13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3531
Tested-by: BuildkiteCI
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 14:00:38 +00:00
Vincent Ambo
5e61c5d246 feat(sourcegraph): Upgrade 3.27.5 -> 3.28.0
See b/144

Change-Id: Ia09ad2af6043dcac6681c549103d1e6f52b4e0a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3522
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 13:47:10 +00:00
Vincent Ambo
52c040eed5 feat(sourcegraph): Upgrade 3.26.0 -> 3.27.5
See b/144

Change-Id: I50d417c51b05bafcd3fe7e285f30079db8be499a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3521
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-11 11:30:37 +00:00
Vincent Ambo
164fc97b4f fix(deploy-whitby): Make diffs world-readable
Change-Id: I1610a8d189f95908bab4cd00057cc080ae47a21a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3530
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 01:41:36 +00:00
Vincent Ambo
3a80ab2ba5 fix(deploy-whitby): Add .html suffix to diff filenames
This makes nginx' content-type recognition work correctly.

Change-Id: I990b00f1e0f4ef311f53a8885718fa33d249c886
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3529
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-11 01:41:36 +00:00
Griffin Smith
9c038cbff0 feat(ops/deploy-whitby): Add the start of a script to deploy whitby
Add the beginnings of an auto-deploy script for whitby, intended to
be (eventually) suitable for running automatically in a systemd timer.
The current iteration of the script doesn't actually do any deploying,
but instead takes as an argument a revision, creates a new git worktree
in /tmp with that revision checked out, runs a nix-diff of whitby's
system derivation in the running system and at that closure, puts an
html-rendered version of that diff in the public directory used by
deploy.tvl.fyi, and finally sends a message to IRC via irccat with a
link to that HTML page.

Refs: b/110
Change-Id: Id40525567f8845590c909568befd8d00c07a481c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3145
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kn <klemens@posteo.de>
2021-09-10 16:13:20 +00:00
Griffin Smith
79b39bb66e feat(whitby): Serve static HTML dir for deploys.tvl.fyi
Add a new domain and nginx virtual host at deploys.tvl.fyi, serving out
of a static directory on whitby which is created by systemd-tmpfiles.

This will be used to serve diffs rendered by nix-diff for
pending deploys for whitby

Since this contains stateful data, it is added to the restic backups
on whitby.

Refs: b/110
Change-Id: I5869d40800bbf5fb8fb39878a857f66ff5787830
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3144
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-10 14:37:36 +00:00
Mike Johnson
9fc206f072 fix(ops/users): Another try at a working password hash for mdjnsn
Change-Id: I8b4aea53abb2004585241ad17c5fdfd9186c58f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3481
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-08-31 20:23:00 +00:00
Mike Johnson
fa8dd0f3ab feat(ops/users): Add mdjnsn to users
Change-Id: I94975d848287c32e11b1d3986986f2dbc6c220b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3466
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-08-31 18:43:49 +00:00
Vincent Ambo
43269730e6 refactor(ops/pipelines): Move failure status zeroing to setup
We changed the configured pipeline in Buildkite to upload
`static-pipeline.yaml` instead of containing the steps of that
pipeline itself.

This makes it easier to test changes to builds and such, but adds
another build step with scheduling overhead etc.

However - we can work around this by killing one of the existing build
steps. There's no reason the failure status zeroing (required for
status reporting) shouldn't be part of the pipeline setup, so I've
moved it there instead and nuked that step.

This should mean that the pipeline is configurable from within the
repo, but without slowing anything down.

Change-Id: I206ecc02647de42a461e33c02879ab84daf5ed2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3461
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-29 12:37:04 +00:00
Griffin Smith
d857d5ad68 refactor(gs/system): Remove chupacabra
This machine no longer exists

Change-Id: I8e549b8397777a01404bd84c10c195e80f281744
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3431
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-08-26 19:41:42 +00:00
Vincent Ambo
60b25b49de fix(ops/pipelines/depot): Buildkite branches use full ref names
... otherwise the filtering also applies to canon.

Change-Id: Ia1c67b99282fb8fd0e4d22e997535170f0326e33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3432
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2021-08-26 16:35:44 +00:00
Vincent Ambo
d5ddfb7b96 feat(pipelines/depot): Skip build steps if their out paths exist
Skip build steps if they have already been built, reducing pipelines
to the things that actually changed between builds. On canon all
targets are always built (we require this for anchoring).

Note that this is not perfect, garbage collection and competing
pipelines may affect each other.

Also note that we have some impure targets that change on every
commit.

Change-Id: Ic6bae3b6c8e1e7fd2116ec252f5089f471854ab6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3427
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-08-26 16:29:32 +00:00
sterni
17d78867bb feat(ops/pipelines/depot): only evaluate once if possible
We currently evaluate every target twice -- once when the depot pipeline
is built and once when actually running the build step in question. Nix
evaluation is quite slow especially given heavy use of import from
derivation in depot, so avoiding the second evaluation is desireable.

Evaluating a derivation yields a `drv` file in the nix store which can
be passed to `nix-store --realise` in order to build it eliminating the
need to wait for evaluation. We can obtain the path to the `drv` file
while building the pipeline via `target.drvPath` and remember it for the
build later.

However we need to work around a flaw (or oversight) in Nix's dependency
tracking via string context: This is based on derivations, not output
path (because this is what evaluation deals with, likely). This is no
problem per se, but an issue is that Nix can't express a dependency on
a `drv` file without any of its output paths. This means for us that we
either have to build all output paths at evaluation time (which we don't
want, obviously) or to deal with the fact that the `drv` file we need
may be garbage collected at any moment after discarding the string
context -- then nix is unable to track the reference from the pipeline
to the `drv` file in the store.

So to prevent a race condition between the pipeline and the garbage
collector we fall back to the normal nix-build invocation as we did
before.

Change-Id: I9ef8bd233085dc6e30eba54f403ea03ac2d35748
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3426
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-26 15:24:33 +00:00
Luke Granger-Brown
febf340303 fix(tvl-sso): set memory limit to 512M
This is because I'm bored of CAS gradually consuming all the RAM on Whitby.

Change-Id: Idcc14c19d99a6d3553739c5765be3faf2bdf9d84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3233
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 16:28:14 +00:00
Griffin Smith
330b7067a0 feat(besadii): Tag gerrit comments as autogenerated
This is a bit of an under-documented feature, but if the "tag" field for
a gerrit review starts with the string
"autogenerated:<something>~<something-else>", only the last comment per
instance of <something> will be shown by default on the CL page (with
the rest viewable by toggling the "Show all entries" switch). The idea
behind the "<something-else>" tag is to be used for the "type" of
comment within a particular system - gerrit's documentation gives the
example of one tag for "the build is running" and another for "the build
has finished, here's the result".

Change-Id: I9199a6ed97beca1b3a51ec5d6230c6c8358ba2b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3374
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 13:00:59 +00:00
Vincent Ambo
eb6c7fd3bf feat(ops/dns): Point nixery.dev to whitby
The dropping of `www.` is intentional, that was unused.

Change-Id: I300f82bb6e5626e2658be8fc5b5e3cf872ab7099
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3384
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-24 11:53:10 +00:00
Vincent Ambo
f218f2fd56 feat(ops): Serve nixery.dev from whitby
Adds a new module for the nixery.dev domain and serves it from whitby.
Note that the DNS records do *not* point to whitby yet, so deploying
this will lead to a failed TLS provisioning unit - but this is
intentional.

Change-Id: I911f67a0aa24f8df3cb52d2cfc49a8b6132cf718
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3383
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-24 11:30:16 +00:00
Vincent Ambo
b033871638 chore(ops/dns): Reduce Nixery TTLs to 1 minute temporarily
We'll need to do a DNS switchover, likely with a short amount of
downtime due to TLS provisioning.

It would be possible to avoid this by provisioning a cert manually
pre-hoc through the DNS challenge and then configuring whitby to use
that, however I simply don't have time for that right now and the
Google Cloud Project for Nixery is going away in O(days) for $reasons.

Change-Id: I88dface5aaacec5acfa525ae117462f8ad296d92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3382
Tested-by: BuildkiteCI
Reviewed-by: kn <klemens@posteo.de>
2021-08-24 11:30:16 +00:00
Vincent Ambo
03c3d49b87 fix(monorepo-gerrit): Enable adding new email addresses to accounts
This is required when people change their email addresses (e.g.
cl/3349) as nothing in Gerrit will update that information from the
OAuth provider.

Change-Id: I1eafdf22efd37898dcd0d06bb9a5d1471ffb5e31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3356
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-08-15 13:59:18 +00:00
eta
80ecce37b4 chore(ops/users): change my email address to tvl@eta.st
I got a new domain, etc.

Change-Id: Ic8ffc01f4e5e89dc2458d80a9c38757438cfa764
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3349
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-08-13 11:21:19 +00:00
Vincent Ambo
7c16a71156 feat(ops/www): Point images.tvl.* at Nixery
Change-Id: I39f979c68e7b74f6da6a7da0f07aaa470886d451
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3346
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-13 10:57:53 +00:00
Vincent Ambo
6cc065ea09 chore(ops/dns): Move nixery.dev to tvl-fyi GCP project
Change-Id: Ifbe7939a98a12d52ffbed3fb198558e6a7743e93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3344
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-12 18:04:30 +00:00
Vincent Ambo
aad636c8ae feat(ops/dns): Add images.tvl.* record
This record is intended to serve Nixery.

Change-Id: I575dedac18c98f9f4bd5e459babe79e850361651
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3343
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-12 15:20:27 +00:00
Vincent Ambo
47409b9610 feat(ops/modules): Add module for running Nixery
This sets up a very simple Nixery instance with some things lacking:

* no support for garbage-collecting image fragments (yet)
* no popularity setup

The plan is to use this to get the ball rolling on a separate
domain (e.g. images.tvl.fyi), iron things out and then look into
flipping over nixery.dev

Change-Id: Ic594809f9d487fec7a0f632d608752a3f9c61315
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3280
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-12 14:55:59 +00:00
Vincent Ambo
79c9506eea fix(monorepo-gerrit): Pin JVM version used for Gerrit
Change-Id: Ib22cdc415cbd5a8345b9589b2c34b3908996dd57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3322
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-12 13:07:55 +00:00
Vincent Ambo
c72f3a73ee feat(ops/dns): Import current nixery.dev zone
Change-Id: I3c5684fedb516740c7048c117cdfda01a2a23260
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3278
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-06 13:24:48 +00:00
Griffin Smith
702594ca64 refactor(ops): Break out prometheus-fail2ban-exporter module
Break out the configuration for the prometheus fail2ban exporter, which
is a simple python script that exports stats from fail2ban as a
prometheus-scrapable textfile, from Mugwump into a reusable nixos module
in //ops/nixos/modules.

Change-Id: I5451c9c5de6c7bc4431150ae596a9c758bf1b693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-06-12 15:51:49 +00:00
Vincent Ambo
b36a75a223 fix(wigglydonke.rs): Don't rebuild nginx config unnecessarily
This fix is essentially the same as the one in cl/1263.

Change-Id: I27be280a610914fcfbb6d7fee7aebaa56b993812
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3158
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2021-05-25 17:10:50 +00:00
Vincent Ambo
65be8f20e0 chore(nixpkgs): Bump channels to 2021-05-25
* users/grfn/system/home/yeren: remove obsolete awscli2 overrides

* ops: make new isSystemUser || isNormalUser assertion happy

* users/grfn/system/system/mugwump: make buildkite agents system users

* users/tazjin/nixos/camden: set isSystemUser = true for git

* users/tazjin/emacs: Remove missing & broken packages

* third_party/openldap: remove, as the argon2 module is now enabled upstream

* third_party/gerrit_plugins: Pinned new unstable hashes

* third_party/nix, third_party/grpc: Disabled CI as these are broken

* third_party/overlays/emacs: Bumped version to stay in sync with channel

* third_party/buzz: Update LIBCLANG_PATH to reference libclang.lib,
  since libclang's default output no longer contains libclang.so

* users/grfn/system/home: Install julia-stable instead of julia (which
  aliases to julia-lts), as the latter depends on an insecure version of
  libgit

Change-Id: Iff33b0ecb0ef07a82d1de35e23c40d2f4bf0f8ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3001
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-05-25 17:09:28 +00:00
Vincent Ambo
878957d2f6 chore(whitby): Add ZNC state to Restic backups
Until we have declarative ZNC config (which requires a solution for
secrets handling in it), make sure we back this up as well.

Change-Id: Idb186327da171eb6d3dbbd83801639f1f9321a40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3159
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-05-25 08:15:19 +00:00
Vincent Ambo
46b136c22e fix(tvl-slapd): Replace deprecated OpenLDAP module options
Use the new module settings which apply configuration in cn=config
instead of slapd.conf.

The module performed this update via lib.mkChangedModuleOption, I've
applied the transformations contained therein manually. Note that some
of the settings were already in place, which means that the `suffix`
and `database` options seemingly disappear into the void.

Fixes b/105.

Change-Id: I8a968c1eb8cb7827618cb732cdb46006a5d011f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3157
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-05-24 22:52:59 +00:00
Vincent Ambo
4a89bcd6a5 refactor(ops/nixos): Pass depot as a special argument
This changes the evaluation order for the `depot` argument and ensures
it is partially evaluated before the module system starts resolving
imports.

This way we can import modules from `depot.path` without `depot`
having to come from readTree.

Fixes b/129.

Change-Id: Icf4dd2be15011055dac8b27e991a4ff6a12bf827
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3156
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-05-24 21:48:37 +00:00
Cynthia Revström
ab7c409530 chore(ops/users): Update email address for cynthia
Change-Id: Ieb59d9215c5c1159113375dea0dd96d3d29e1303
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3154
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-05-24 20:22:53 +00:00
Klemens Nanni
a26c4a1c77 fix(ops/users): Rehash password for kn
This time using `tools.hash-password` because login did not work with the
initially created hash.

Change-Id: I1eb62a496d2d8497d27573af47bf8bf70dac9bbb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3153
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-05-24 18:31:10 +00:00