Trialing this as an alternative to CAS that is a little easier to
configure and can help us delegate authentication to other OIDC
services.
Change-Id: Iad63724d349334910af8fed0b148e4ba428f796b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4608
Tested-by: BuildkiteCI
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
I'd like to be able to run extra CI steps that include running docker
containers (to integration test things like webapps that connect to a
database). To do this the buildkite agents themselves need permission to
do docker things.
Change-Id: I3c9a488708f0e12a508754ac41f04148ca7aedac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4408
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
For modules that are gated behind a mkEnableOption, it's reasonable to
just provide them to all Depot-built nixos systems without requiring
people to explicitly import them. This defines a special module called
`default-imports.nix` which imports these modules (currently just
tvl-cache.nix and automatic-gc.nix, as I'm being rather conservative
adding things here to avoid breaking anyone's system), then provides
that module as one of the `modules` passed at the top-level
nixos/eval-config invocation.
Change-Id: I3be299ab10ae4c451ef11c514edb3c89318a2278
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4345
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Add a shared nixos module for configuring whitby as a binary nix cache,
and refactor tverskoy to use this module.
This is enabled via an option to pave the way for including it as an
import in all depot-generated nixos configs at some point in the future.
Change-Id: I6dcc0e8eb48b1ac34457666dceebeedd5da6c526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4344
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: grfn <grfn@gws.fyi>
https://cl.tvl.fyi/c/depot/+/4264 did move merging config with secrets
into ExecStart=, which is tracked in an RFE upstream:
https://github.com/systemd/systemd/issues/19604#issuecomment-989279884
We didn't link to this so far, neither in the commit message, nor in a
comment.
Let's add a comment, so people know when we can undo this.
Change-Id: I7bed370b671093bb876592b4dccd562f1c256cd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4326
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
Git only allows binary names prefixed with `git-credential-` if the
path to the helper is not absolute.
Why? Who knows.
Change-Id: I216b2a621f62a73f05e21def7ec8016b29ede892
Currently this functionality is provided by a shell script stored in
/etc/secrets (which has the password value hardcoded).
This needs to happen in a separate commit from the one that changes
the pipeline to avoid breaking it (it needs to be deployed first).
Change-Id: I680754c828ccefbacfcf0d5c813a4bc19493ba4c
We already checked this in, but this commit adds the configuration for
making use of it.
There are two copies of besadii's JSON configuration with different
permissions.
Note that the buildkite-graphql-token path needs to be updated in
static-pipeline.yml, but this needs to happen in a separate commit
after deploy because the pipeline will break otherwise.
Change-Id: I6fab4bf1a2e679df7cf76521e2b53bd9dadbac62
... this option really is a pitfall! The list of programs is now the
same as in the upstream module, plus curl and jq.
Change-Id: I29edae4b2400a2724f62df9efa1dc184a8b0af5f
The DynamicUser + Group configuration does not work as planned, thus
the systemd LoadCredentials feature is used instead which makes the
file (which itself is only readable by root) available in a
memory-backed location only readable by the service.
The secret is only available to `ExecStart` commands, so units using
this feature can not be used with pre/post units and the like if those
commands need secrets.
To accommodate this, the merge of configuration files has been moved
into the service launch script, which is now the ExecStart= process.
For details take a look at https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LoadCredential=ID:PATH
Change-Id: I693fe5677cc0d63c7aa485c2c7472457c5262166
It turns out the lib.mkAfter call doesn't behave as expected -
only *some* of the packages that are defaulted end up in the $PATH.
I suspect this is actually something else, e.g. these packages are
always added for some reason or another, and the option is completely
overridden every time.
Change-Id: I854c7198520d82b00e6338ed0fe653836226dc6d
Turns out that the type of this option is not concatenative and it
replaces the packages needed to run Buildkite if set.
Change-Id: I9f52572bc165bccdd8c6518cfdf7b8967f7a50d0
The irccat module uses DynamicUser, so to grant permission to it a new
group has been added for irccat.
I have some vague memory of DynamicUser + Group not behaving as one
would expect, but we'll see what happens.
Change-Id: Iab9f6a3f1a53c4133b635458ce173250cc9a3fac
This is not yet including the secret configuration for gerrit-queue,
and just expects the secret (gerrit username & password) to be
available in /etc/secrets.
Change-Id: Ia465ef7f3f521c70d606d7fdeba9aa83c7e1b98b
This is required for a simplification of the build pipeline (following
CL) and needs to be in a separate commit as it can not be done
atomically (merging the other commit to deploy it would immediately
break pipelines otherwise).
Change-Id: I5d8ec8f3238f79b5518d799486bf98d1d9516c43
Ensure that besadii sees $0 as the correct command name, since that is
the sole mechanism by which its functionality is switched around.
There was a lingering commit that introduced this bug and hadn't been
deployed in a couple of days. Maybe time to tighten deploy cycles soon
...
Change-Id: Ie4284c0f6e5e06d71a71a3702ec7e092260e0ce5
On whitby, the besadii config will live in
/etc/secrets/besadii.json. This CL updates the call sites to pass this
config path to besadii so that it can load Sourcegraph configuration.
Change-Id: Ia139b9fa3b827e7a5f2386214390acc6fe19a75a
Change the Nixery configuration to use the plain nixpkgs package path
instead of the depot path. AFAIK, nobody uses this to fetches depot
packages at the moment - but plenty of people fetch non-depot
packages.
This means that Nixery is cache-busted less often (previously on every
commit => every deploy).
We'll figure out another way to have a depot Nixery later.
Change-Id: Iba632333346181c3d2ce992fbab396ed0d9f86aa
Removes besadii support for the previously used 'ref-updated' hook and
instead introduces support for the 'change-merged' and
'patchset-created' hooks.
These hooks more accurately capture the semantics of when besadii
should trigger CI builds and using them will avoid problems such as
skipping 'canon' builds if chains of CLs are submitted together.
Change-Id: Ib90356c069780bf0c0250e56b927e46a5b31ce7f
Since GCP nuked us, the backups are now moving to GleSYS'
S3-compatible object storage.
This refactors the restic module to support S3-compatible storage
instead of GCP, and switches to the appropriate new secret paths.
The secrets were placed on whitby manually and I verified that the
backups work.
This fixes b/157
Change-Id: I6a9d2b0581967605ce736605a3befb44cdeae7e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3883
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
The setup is explained in the comment, but TL;DR: Use the derivation
hash of static files to create permanent URLs.
Relates to b/151.
Change-Id: Ib1ca3a1a00c90a47f4bf39c29a8b4bbf5b215e7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3664
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
As cschilling explained on cl/3563, there isn't actually anything in
this state that we *need* to persist. We're still keeping it in a
persistent directory on disk as this serves as an optimisation after
restarts of josh.
Change-Id: Ia88886792a5acac34508b5b8a669bd519ca033de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3631
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This lets each service declare their backup paths together with the
configuration for the service, which is a lot more sensible than what
we had before.
Fixes b/147
Change-Id: If76fe62639f4cc0e6fbb63a2959d584479d8f0fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3583
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
I can never remember which is which.
Change-Id: I69b8235862b8c5b49030a74bfca25aaa113273b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3582
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Relates to b/147.
First step towards giving depot modules the ability to declare their
own backup directories by moving all restic configuration into a new
module and adding a NixOS option for inclusion/exclusion paths for
backups.
This still keeps all backup paths within the whitby config.
Change-Id: Ia96833668f1a3d02da892261153d8b02156b8ac0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3565
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Previously we served the dumb git HTTP protocol from code.tvl.fyi via
cgit. This CL disables this feature and instead runs josh in the same
location (by redirecting appropriately), but while also enabling
partial cloning of all subtrees of the depot.
For example, after this CL the following would result in an
independent clone of //nix/readTree:
git clone https://code.tvl.fyi/depot.git:/nix/readTree.git
Note that there are no josh workspaces configured at all for now,
these references are only for static depot subpaths.
Please refer to the documentation for josh for more information on
available kinds of josh filters.
Josh state is kept in a systemd state directory in /var/lib/josh and
backed up to Restic. Backing this up is necessary, as josh uses
stateful information to do things like tracking merges and rewriting
history per subtree appropriately to avoid cloned repositories ending
up in peculiar states.
Change-Id: I156f0298c2aa42e3bdbf5a0e86109070d640c56e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3563
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This one seems a little more involved:
https://docs.sourcegraph.com/admin/migration/3_31
I believe we skip that corruption issue in the previous CL though, by
simply never deploying a version with that weird broken image.
See b/144
Change-Id: I3bbf1b719d00905e08a92011ace5485467f504ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3525
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
We changed away from the default sourcegraph one because it didn't
support Nix, but it seems that there's been a change in the
interaction protocol.
Change-Id: I3a2691df6a87672cf83b819143f25d93d9cd6d13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3531
Tested-by: BuildkiteCI
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: sterni <sternenseemann@systemli.org>
Add a new domain and nginx virtual host at deploys.tvl.fyi, serving out
of a static directory on whitby which is created by systemd-tmpfiles.
This will be used to serve diffs rendered by nix-diff for
pending deploys for whitby
Since this contains stateful data, it is added to the restic backups
on whitby.
Refs: b/110
Change-Id: I5869d40800bbf5fb8fb39878a857f66ff5787830
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3144
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is because I'm bored of CAS gradually consuming all the RAM on Whitby.
Change-Id: Idcc14c19d99a6d3553739c5765be3faf2bdf9d84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3233
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Adds a new module for the nixery.dev domain and serves it from whitby.
Note that the DNS records do *not* point to whitby yet, so deploying
this will lead to a failed TLS provisioning unit - but this is
intentional.
Change-Id: I911f67a0aa24f8df3cb52d2cfc49a8b6132cf718
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3383
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is required when people change their email addresses (e.g.
cl/3349) as nothing in Gerrit will update that information from the
OAuth provider.
Change-Id: I1eafdf22efd37898dcd0d06bb9a5d1471ffb5e31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3356
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
This sets up a very simple Nixery instance with some things lacking:
* no support for garbage-collecting image fragments (yet)
* no popularity setup
The plan is to use this to get the ball rolling on a separate
domain (e.g. images.tvl.fyi), iron things out and then look into
flipping over nixery.dev
Change-Id: Ic594809f9d487fec7a0f632d608752a3f9c61315
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3280
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Break out the configuration for the prometheus fail2ban exporter, which
is a simple python script that exports stats from fail2ban as a
prometheus-scrapable textfile, from Mugwump into a reusable nixos module
in //ops/nixos/modules.
Change-Id: I5451c9c5de6c7bc4431150ae596a9c758bf1b693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This fix is essentially the same as the one in cl/1263.
Change-Id: I27be280a610914fcfbb6d7fee7aebaa56b993812
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3158
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
* users/grfn/system/home/yeren: remove obsolete awscli2 overrides
* ops: make new isSystemUser || isNormalUser assertion happy
* users/grfn/system/system/mugwump: make buildkite agents system users
* users/tazjin/nixos/camden: set isSystemUser = true for git
* users/tazjin/emacs: Remove missing & broken packages
* third_party/openldap: remove, as the argon2 module is now enabled upstream
* third_party/gerrit_plugins: Pinned new unstable hashes
* third_party/nix, third_party/grpc: Disabled CI as these are broken
* third_party/overlays/emacs: Bumped version to stay in sync with channel
* third_party/buzz: Update LIBCLANG_PATH to reference libclang.lib,
since libclang's default output no longer contains libclang.so
* users/grfn/system/home: Install julia-stable instead of julia (which
aliases to julia-lts), as the latter depends on an insecure version of
libgit
Change-Id: Iff33b0ecb0ef07a82d1de35e23c40d2f4bf0f8ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3001
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Use the new module settings which apply configuration in cn=config
instead of slapd.conf.
The module performed this update via lib.mkChangedModuleOption, I've
applied the transformations contained therein manually. Note that some
of the settings were already in place, which means that the `suffix`
and `database` options seemingly disappear into the void.
Fixes b/105.
Change-Id: I8a968c1eb8cb7827618cb732cdb46006a5d011f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3157
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This doesn't replace all of them in the repo, but at least the ones
that are relevant to our move.
Change-Id: I842e7594b4c16af30d880272417874f6b29afd22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3134
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: grfn <grfn@gws.fyi>
This configures owothia to use her new bouncer to HackInt.
Change-Id: I80eb8191c2b0f2a6f8a31d19b60250ade27c1913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3129
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
The following commit itends to bind on port 8443 on all interfaces,
so let's move this to something else.
Change-Id: Ibb94a0f4e6892b6e543b542b89bcdaaefb617f23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3126
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Sourcegraph logs warnings about this on startup otherwise. Unclear
to what degree it really affects operation though.
Change-Id: I6ee7c5358631aafd9a7f8155150361bf7499314d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3098
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
.. this is actually likely not disabling it for some pages, that will
need this to be copy & pasted, but it's hard to tell just from the
nginx docs. We'll make sure after deploying.
Change-Id: I2fa6e31ca10835a206673b858594fa071e729d82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3020
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
It needs to refer to this by full path of course.
Change-Id: I911c876ba18877681accb722426314d92b9f2318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3042
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This will require the daemon to be running when launching GC, but
won't start it if it happens to not be running for some reason.
Change-Id: If48fe336030173f028428fc00a81d339ef4b8bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3015
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Adds a module that automatically collects garbage based on disk space
thresholds, and configures it to run hourly on whitby.
This is implemented as an alternative to cl/2937, which I've been told
uses a Nix feature that doesn't actually work.
Under-the-hood this is simply a systemd timer running a shell script
which checks available disk space and runs GC when necessary.
Change-Id: I3c6b5de85b74ea52e7e16c53f2f900e0911c9805
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3014
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Enables a Grafana service pointing to whitby's local Prometheus
instance, accessible at status.tvl.su.
I've no idea how to configure Grafana and if it's possible to link it
to CAS, but we'll see about that later.
Notes:
* the explicit fixpoint for whitby config has been removed as we
have the `config` parameter available now
* backups are enabled for the Grafana storage location
Change-Id: If5ffe0c1a3378d1c88529129487c643642705fd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2948
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Instead of having two ways of accessing the path to the depot (one of
which was stuttering, depot.depotPath) we settle on only one:
depot.path.
This was mostly used for NixOS module imports.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I2c0db23383fc34f6ca76baaad4cc4af2d9dfae15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2962
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Rename my //users directory and all places that refer to glittershark to
grfn, including nix references and documentation.
This may require some extra attention inside of gerrit's database after
it lands to allow me to actually push things.
Change-Id: I4728b7ec2c60024392c1c1fa6e0d4a59b3e266fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2933
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
Splits //ops/nixos into:
* //ops/nixos.nix - utility functions for building systems
* //ops/machines - shared machine definitions (read by readTree)
* //ops/modules - shared NixOS modules (skipped by readTree)
This simplifies working with the configuration fixpoint in whitby, and
is overall a bit more in line with how NixOS systems in user folders
currently work.
Change-Id: I1322ec5cc76c0207c099c05d44828a3df0b3ffc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2931
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: glittershark <grfn@gws.fyi>
