I like running fail2ban on any machine that has stuff like ssh
world-open, to limit the potential for password brute-force attacks etc.
Change-Id: I0c60811ae5a2fddb44f04679fb455e646b8e39c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3138
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This doesn't replace all of them in the repo, but at least the ones
that are relevant to our move.
Change-Id: I842e7594b4c16af30d880272417874f6b29afd22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3134
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: grfn <grfn@gws.fyi>
This configures owothia to use her new bouncer to HackInt.
Change-Id: I80eb8191c2b0f2a6f8a31d19b60250ade27c1913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3129
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Points clbot at the new local ZNC instead. This will make it part of
the things happening through the `tvlbot` account.
Relates to b/101
Change-Id: I1c15ffa5720d3af34475c15bee3fdaa537ac659b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3127
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
This adds a shadowsocks service, running on port 8443, tcp and udp.
The password is read from /etc/secrets/shadowsocks-secret.sec, and needs
to be populated externally.
Change-Id: I6797150db108ba14459502dee43d8e4ed6cfa910
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3125
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
The following commit itends to bind on port 8443 on all interfaces,
so let's move this to something else.
Change-Id: Ibb94a0f4e6892b6e543b542b89bcdaaefb617f23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3126
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Bouncer to be used for TVL's IRC bots, see b/101
Change-Id: Ic9f71ecd94365d3baa31e0552b1ce16362f94557
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3124
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Sourcegraph logs warnings about this on startup otherwise. Unclear
to what degree it really affects operation though.
Change-Id: I6ee7c5358631aafd9a7f8155150361bf7499314d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3098
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
The shorter one is going to be more convenient when we get
go-link (or, well, at-link) support.
Change-Id: Ic24adcdad679b893c40c87731add818660259dac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3091
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
.. this is actually likely not disabling it for some pages, that will
need this to be copy & pasted, but it's hard to tell just from the
nginx docs. We'll make sure after deploying.
Change-Id: I2fa6e31ca10835a206673b858594fa071e729d82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3020
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
This is currently done ad-hoc in a bunch of our systems, but we should
just do it centrally.
The commit message is a bit of a lie, as this doesn't yet update
grfn's systems.
Change-Id: Ic771c1a1da78ec5de9cffbf94c296dce5e11fd84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3047
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
It needs to refer to this by full path of course.
Change-Id: I911c876ba18877681accb722426314d92b9f2318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3042
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This will require the daemon to be running when launching GC, but
won't start it if it happens to not be running for some reason.
Change-Id: If48fe336030173f028428fc00a81d339ef4b8bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3015
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Adds a module that automatically collects garbage based on disk space
thresholds, and configures it to run hourly on whitby.
This is implemented as an alternative to cl/2937, which I've been told
uses a Nix feature that doesn't actually work.
Under-the-hood this is simply a systemd timer running a shell script
which checks available disk space and runs GC when necessary.
Change-Id: I3c6b5de85b74ea52e7e16c53f2f900e0911c9805
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3014
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
There's a hard-coded list of Admin usernames for the moment. We should
revisit this and get an actual groups setup in LDAP that's propagated
through...
Change-Id: Ic3601f1a9753573076769f4912038e9f1b60e139
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2982
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
Enables a Grafana service pointing to whitby's local Prometheus
instance, accessible at status.tvl.su.
I've no idea how to configure Grafana and if it's possible to link it
to CAS, but we'll see about that later.
Notes:
* the explicit fixpoint for whitby config has been removed as we
have the `config` parameter available now
* backups are enabled for the Grafana storage location
Change-Id: If5ffe0c1a3378d1c88529129487c643642705fd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2948
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Instead of having two ways of accessing the path to the depot (one of
which was stuttering, depot.depotPath) we settle on only one:
depot.path.
This was mostly used for NixOS module imports.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I2c0db23383fc34f6ca76baaad4cc4af2d9dfae15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2962
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Dropping the message field will make Buildkite use the commit messages
instead, which makes for much more readable build logs.
Change-Id: I1849f811632526893b700f117c9f6cf64888c329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2949
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Enables Prometheus with a local node exporter, and nothing else for
now.
Some additional collectors have been enabled for things that might be
relevant on whitby:
* systemd: all our services run in systemd
* processes: might be interesting for build-related stats
* logind: might be interesting for interactive usage stats
Change-Id: I48dacdd9c68b4be9edff7b3cb6256dad562498c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2930
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
--show-trace should make it easier to debug tricky evaluation errors
without running nix-build -A ops.pipelines.depot locally again.
Change-Id: Ice540562c3b389fc2a49ec1fc0adacb17db2a528
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2947
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Rename my //users directory and all places that refer to glittershark to
grfn, including nix references and documentation.
This may require some extra attention inside of gerrit's database after
it lands to allow me to actually push things.
Change-Id: I4728b7ec2c60024392c1c1fa6e0d4a59b3e266fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2933
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
Splits //ops/nixos into:
* //ops/nixos.nix - utility functions for building systems
* //ops/machines - shared machine definitions (read by readTree)
* //ops/modules - shared NixOS modules (skipped by readTree)
This simplifies working with the configuration fixpoint in whitby, and
is overall a bit more in line with how NixOS systems in user folders
currently work.
Change-Id: I1322ec5cc76c0207c099c05d44828a3df0b3ffc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2931
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: glittershark <grfn@gws.fyi>
This change is required to run the ⚓ step on canon builds.
Change-Id: Ib3cebac67c9f5337b27a948f120b0a9ba834ef2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2932
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: glittershark <grfn@gws.fyi>
Adds a conditional build step that only runs on the canon branch, and
only if 🦆 (the status reporting step) succeeds, which creates a
new Nix GC root for all depot targets named `depot-canon`.
In practice this might be a bit racey, as canon builds are not
guaranteed to succeed in order (though it is likely). This shouldn't
matter much in practice: We only want to prevent rebuilds of the whole
world.
This fixes b/102
Change-Id: Id3d0bf4158bffcb1ed6929888a29d31609b6ece1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2904
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This ensures files created by the Buildkite agents are always owned by
the same group, without having to manually chgrp afterwards.
Change-Id: Idbaedec43c16b2ee137d1a95719a05d46db8f900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2929
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Please read b/108 to make sense of this.
This gets rid of the explicit list of exposed packages from nixpkgs,
and instead makes the entire package set available at
`third_party.nixpkgs`.
To accommodate this, a LOT of things have to be very slightly shuffled
around. Some of this was done in already submitted CLs, but this
change is unfortunately still quite noisy.
Pay extra attention to:
* overlay-like functionality that was partially moved to actual
overlays (partially as in, the minimum required to get a green
build)
* modified uses of the package set path, esp. in NixOS systems
Special notes:
* xanthous has been disabled in CI because of issues with the Haskell
overlay
* //third_party/nix has been disabled because of other unclear
dependency issues
Both of these will be tackled in a followup CL.
Change-Id: I2f9c60a4d275fdb5209264be0addfd7e06c53118
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2910
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This configures accepting requests for b/ and cl/ on plain HTTP ports,
and redirecting to b.tvl.fyi & cl.tvl.fyi appropriately.
Additionally, Panettone request URIs that only contain decimals are
redirected to `/issues/$request_uri` to enable issue short-links.
This fixes b/32.
Change-Id: I56954d8d69a3624267778b467520c509f4daa6c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2908
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
In preparation for the solution of b/108, we need to consistently use
`depot.third_party` for packages that are only packed in the TVL depot
and `pkgs` for things that come from nixpkgs.
This commit cleans up a huge chunk of these uses in //ops
Change-Id: I00faeb969eaa70760a26256274925b07998c2351
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2915
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This lets us grant permissions to them, e.g. on local folders.
Change-Id: I823ac414be1cb7d6baa4f17d95003709e5911b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2905
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
There will be more Buildkite-agent specific configuration, and it's
already more than just the module setup, so extracting this makes
sense.
Change-Id: I56ce205c0cb4365317ed7ed5f2d525a0b425b861
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2906
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
This small(*) pile of JavaScript queries the Buildkite API for the
latest builds for the depot and displays the results in the rebooted
Check UI.
Change-Id: I7025a1c6d0d0afa000a9df4682133e03824ea10d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2881
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is just going to be a grab bag of things which do TVL-specific
things to Gerrit, whether that be exposing new Prolog predicates or, as
I intend to do as the first thing, expose Buildkite builds as checks.
Change-Id: Iaeab987a1fdbd078b85e274691c986489903bf3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2872
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Having a slow cache is better than having no cache.
Change-Id: Ie3cfcd4a2937d90b0e2ad899816bc31ae806631f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2847
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
This drops the old LDAP configuration and uses CAS instead. All hail the
hypnotoad.
Change-Id: I515a213f09073bb52bfb75afe2988b935a076087
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2783
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Until I come up with a better idea.
Change-Id: Ie44cae4c2df264cbe1a70f5ebcca814262dd2800
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2771
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
All user configs and modules have been migrated to using the depot
module parameter. All hail the hypnotoad.
Change-Id: Ic05c61fccba3ac505a339283b6ef3105a2d0711c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2765
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Previously the depot argument was provided as config.depot, but the "new
way" of doing things (which is more like the args list provided in the
rest of the depot) is to provide this as the "depot" NixOS module
argument instead.
Change-Id: Ib48b1c7c1bdff9c1eb0618c6cbacc22b651f5f98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2763
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
For the moment I've opted to not import all of the other things we'd
usually provide to things imports via readTree, because I think it's a
bit dangerous to accidentally overwrite things like NixOS' notion of
"lib" with our own version.
So for the moment, baseModule provides only "depot".
Change-Id: I3db9132a3d9227055d4c1b00f02effcb84edcc53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2760
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is not for the domain root though, as that's going to be
something else eventually.
The canonical URLs are the .fyi ones (at least for now), and some of
these tools will eventually generate links that make user sessions
started from *.tvl.su converge on *.tvl.fyi.
Relates to b/98
Change-Id: I1c3bcf72a3063059002e4b0bdd57c269a410a8bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2758
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
I am somewhat trustworthy… maybe? Also I tend to gc depot stuff so ssh
serve would be neat.
Change-Id: I4672f20a32a756692dd156b5e40e5a7f37ba5ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2660
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
This is a quick hack to make it possible to view the rendered SVG on
https://code.tvl.fyi/about/tvix/docs/components.md
We want to be able to do this sort of thing dynamically in the future,
but we can't yet, so ... well. Deal with it.
Change-Id: Id2b819679d748b6f517018a9c6e72d5c1d806c4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2743
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This reverts commit 3b05be2fd0.
Reason for revert: Sourcegraph still does not support fetching arbitrary refs, so we'll have to wait until its Gerrit integration lands before this will work correctly.
Change-Id: Icee82c50f92c34ba1741b608449aed16538ccbaa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2721
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Some quick testing shows that this improves my data transfer speed to
whitby by roughly 200%.
Change-Id: Id94de975b1ae0930f8d0fe038582dbac0037676c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2659
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: ben <tvl@benjojo.co.uk>
I'm looking at removing some of these because they can cause
unnecessary build steps during CI pipeline generation.
Change-Id: I84742968918090c050d2eedab8a1b42692632a42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2655
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reading through the changelogs, this includes the following two
changes that may require us to do something:
* For users of single-image Sourcegraph instance, please delete the
secret key file /var/lib/sourcegraph/token inside the container before
attempting to upgrade to 3.21.x.
* A campaigns.restrictToAdmins site configuration option has been
added to prevent non site-admin users from using campaigns.
Change-Id: Ieacf85a9059ad5222800f8d7d4a43435f489a39f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2638
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
I want to host something like Vigil[0] on this to show the status of
Gerrit, SourceGraph and maybe other components.
(Yes, the status page will be on the same infrastructure ... but this
is mostly for service failure cases).
[0]: https://github.com/valeriansaliou/vigil
Change-Id: If71496300b94035976a685d9bf166d525d89fc5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2637
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
This was a leftover from the time we were installing.
Change-Id: Id875b907d7f76081a45e7f8f2666b7fba6aefc86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2632
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This is my new X13 AMD Thinkpad, on which many fun things will be done.
Change-Id: I4de114a8c5ebb37d2f4844f407d2dc0e7cc9557e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2620
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Since we have a dedicated util for this, we may as well use it
to reduce code duplication.
Change-Id: Ie52647be8c786d0b6a4dceb2fa6778b94625fafc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2604
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Imports the current state of the tvl.fyi zone and configures simple CI
checks on the file format.
No deployment automation exists for this (yet?).
Change-Id: Ia7d72e02b9f6d3adef994c5dc1898cc0df9dfcfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2600
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
The ciBuilds attribute seems to no longer exist and it breaks the
evaluation of the config attribute. It's only appearance was in
besadii which doesn't actually use the attribute.
Removing the ciBuilds inherit fixes these issues.
Change-Id: Ibbf3413ba6efe10ad868cf57cf0711d574860f97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2487
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Nobody has actually done any experimentation with typed Go, so we're
getting rid of it for now - it's causing annoying IFD during build
graph generation.
Change-Id: Ibac3dea98ebed1b3ee08acda184d24c500cf695d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2458
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
This commit removes my user directory in the depot, my user account on whitby,
my entry in the LDAP database, and my entry in the website graph. I've had my
fun with TVL, but I want to move on to spending time on some other things.
This additionally removes aranea from the website graph, which they have
requested in private.
Change-Id: I2d098c8fe239f20d9f6c6cbf66a3dfb4a955a4cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2436
Tested-by: BuildkiteCI
Reviewed-by: multi <depot@in-addr.xyz>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Since the slapd data is static and generated using nix, we can simply
move the user list into ops/users, so it's recognized by readTree and we
can use it as ops.users both in ops/nixos/tvl-slapd and web/todolist as
a general purpose user registry for depot.
Update docs/REVIEWS.md as well.
Change-Id: I35caaaab70a5578c47cedc7f33077dd513766290
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2419
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
camden.tazj.in (the host in my flat) is going down as my belongings
are being moved into storage.
Change-Id: Id66512fd2ec6dbdcb6dfc3862af49cfadb15cfa1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2405
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
My main workstation is a Thinkpad without a great deal of compute
power available, so enabling the use of whitby as both a substituter
(services.sshServe) and a remote builder (openssh.authorizedKeys) will save me
some time when working on nix things and depot things.
Change-Id: I17bfcbb9860f42fb667603ad819e38e82e6052da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2399
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
Changes:
* ops/nixos/tvl-slapd: The NixOS module for OpenLDAP has removed the
ability to configure OpenLDAP directly and now forces users to use
some kind of weird Nix->OLC mapping that is mostly undocumented.
This moves the config we need to the new format in a way that may or
may not work and does the other arbitrary dance steps that someone
decided to impose on us. Note that this now throws lots of warnings,
but I can't be bothered to fix them.
* 3p: Random package removals accomodated
* users/glittershark: Pin grfn's kernel to 5.9, because the CK patch
is not yet updated for 5.10
* users/glittershark: Update vendor hash for pg-dump-upsert, I suspect
this changed because of something in the Go build machinery in
nixpkgs. The deleteVendor flag also has no effect anymore and has been
removed.
* users/glittershark: agda build is broken, commenting out development
home-manager environment until it can be fixed
* third_party/haskell_overlay: updating random needs upper boundarles
of a few dependencies relaxed (curse them)
* third_party/gerrit_plugins: for some cursed reason the fixed-output
hash of the gerrit owners plugin fetchgit changed, updated.
Same for the checks plugin.
Change-Id: Ica37995fe8039d3ba80eab643867f98795c56734
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2295
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
This feature can cause object removal to happen while the git folder
is in use in Buildkite, causing CI to fail semi-reegularly.
Change-Id: Ide1a9b2f1761be029e97a058c1983b4cff5e27bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2285
Tested-by: BuildkiteCI
Reviewed-by: multi <depot@in-addr.xyz>
My new work laptop, a dell XPS 13.
Change-Id: Ieab06622c9b280182025edfa63adf649e5fc70d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2205
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Mugwump is too unstable for such an important internet service
Change-Id: Ic714200ce5ce51f366777f538b4a6f443f010960
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2124
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Add the depot.nix module and a depot config option to all nixos system
derivations that're build through the `bin/rebuild-system` machinery.
I can't imagine a scenario where we wouldn't want this level of
integration.
Change-Id: Ieeb98db2eee23919256adb4654bc45d540e055ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2128
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
This file represents the static pipeline which is configured in the
Buildkite web UI. Updates to this file should be applied in the admin
interface.
These steps are responsible for launching the dynamic pipeline
evaluation, or falling back to the fallback pipeline if evaluation fails.
Change-Id: I6d7dd623cde65e8c69faea729f737c9bba00c2fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2103
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This adds a simple fallback Buildkite pipeline configuration which
always fails the pipeline, but correctly reports back the failure
status.
Note that this also requires changes in the Buildkite configuration
that is not in version-control.
Relates to b/66.
Change-Id: I6802a6f76448c3893798a06d514e6ccba0f50dd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2102
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Adds configuration options for the (inconsistently named) environment
variables that configure irccat integration with Panettone.
The defaults match the irccat setup on whitby.
Change-Id: I6857512a2e3f29f16777493eb981cc69ce3c045f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2080
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This module configures irccat by creating a JSON configuration file
from a user-supplied Nix struct (this is not checked for correctness),
and merging it recursively with secrets from
`/etc/secrets/irccat.json` at service launch time.
This way we get the ability to configure (most) options declaratively
via Nix, while providing the secrets outside of Nix.
Side note: We need to figure out a secrets distribution mechanism.
Tested: Wrote a dummy config in whitby/default.nix locally and checked
that this builds, but I have not actually run the service yet. I
expect that some minor tweaks will end up being necessary.
Change-Id: I02a2e8dc40a7f8417fd77afcf8a12ac3df117988
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2074
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
... I found this location in the logs, because the certs are now valid
for this, but I'm not actually sure if it's right.
Change-Id: I5ac88073e3bf6a95fead4c1d34515622c4416c6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2070
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Sometimes (like today) paroxysm crashes. We'd like it to restart if that
happens.
Change-Id: I98841096bcd6605c4279744ae5c65a9c92092a21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2069
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
The main bottleneck of our builds right now is Nix evaluation, which
means that most of the time is spent idling during builds.
Since we're evaluating in parallel, lets give it a few more builders.
I don't want to go all the way to 64 immediately to first see if we
get any adverse effects from highly concurrent builds running
concurrently (if we do we could group them into different "concurrency
groups" in Buildkite).
Change-Id: Ibc3f89fb59cb4ee471b152ff36887ffe2b39f8f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1932
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Allow 16 things to happen in parallel, which is useful now that the
CI granularity is on a per-target level.
Change-Id: Ie65dd119ea0666618fbb249613e70a68276db834
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1902
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
We have naturally evolved a distinction between logical and physical
targets.
Physical targets are those which correspond directly to a tree
location on disk and can be built with `-A path.to.files`, while
logical targets are those that are exported from within an expression
but do not have a corresponding file on disk.
This change adds support for exporting logical targets from any tree
location by adding a `meta.targets` attribute containing keys into
itself, which will be consumed by the CI target gathering logic and
included in the generated pipeline.
Note that the labels for subtargets are syntactically different to
emphasise that they do not correspond to a file location. For example,
this change enables 'ops.nixos.whitbySystem' as a subtarget, which is
labeled in CI as `ops/nixos:whitbySystem`.
Change-Id: Ied09647a62c2ba98e3914548e3742ad422c63ecf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1893
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Create the pipeline by outputting a file that contains nix-build
invocations for each target's *derivation path*.
Each invocation has a generated Nix expression passed to it with `-E`
which fetches the correct target from the tree while correctly
handling targets with strange characters (such as in Go-packages).
This makes it possible to run target-level granular pipelines. We're
getting somewhere!
Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Auto-link b/\d+ in gerrit comments + patch comments to panettone issues.
This is a departure from the previous syntax of #\d+ that we were using
historically, but has been requested by multiple people
Fixes: b/40
Change-Id: I8e556eb6b2f55ae80d945b86b51041751ecb053b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1888
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Automatically walk the entire depot tree and pick out things that are
"buildable", then include them in the attribute `ci.targets` (which is
now also the target for CI builds).
A long time ago, in a land far away, we (well, I, at the time) had a
prototype of this which ran into constant issues with infinite
recursions while trying to walk the tree. In fact, this is why
readTree originally gained the `__readTree`-attribute which marks
things that were imported automatically.
Based on some code edef whipped up earlier (with the breakthrough
being that we also add the attribute to top-level folders, which
suddenly resolves a whole bunch of problems), I've now implemented
this actually working version.
At the moment all builds still happen as one big bag of builds, but at
some point we will granularise this.
Change-Id: I86f12ce7f63dae98e7e5c6646a4e9d220de783f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1854
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
This was used previously when build granularity was besadii's task,
which it no longer is.
Change-Id: I6df2db1ed4730a7953199b7b48aa9ad916418b22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1853
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This is a temporary state (TODO added) to be picked up by the new CI
logic.
Change-Id: Id4702740ffd18325088e2a8a0c6157a8cee7ccf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This reverts commit 8fa3087067.
Reason for revert:
This almost worked. We discovered two important issues:
- The daemon startup does not correctly handle the socket passed in by systemd.
- There is some issue with chunking of large calls, running a build for ci-builds resulted in:
tazjin@whitby /depot (canon)> nix-build -A ciBuilds.__allTargets
E20200821 01:42:22.846053 12601 shared.cc:306] error: Rpc call addTextToStore to unix:///nix/var/nix/daemon-socket/socket failed (RESOURCE_EXHAUSTED): Received message larger than max (10889961 vs. 4194304)
Change-Id: Ic5ba4ef06a4953cf71a36b139fe25ea673cb6fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1802
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
... this is going to break so much stuff. Lets have some fun.
Change-Id: If0185e0323391c7055d47b797083bb5afde57cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1829
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Adds a lot more information about what is actually going on here, as
well as links to relevant things such as our Monorepo document.
I'm aware that I didn't make all `//...` links clickable, but I
realised at some point that it might be easier to just update cheddar
to support highlighting those natively :-)
Change-Id: Icbb212a6c07a5cf38ab8e65d83a64bec783eb8d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1768
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Adds the ability to post to multiple channels by simply running
multiple instances of clbot.
We should probably implement support for this in clbot itself, but
right now I can't be bothered to write Go.
Change-Id: I5cffd0dc10a7f6cc19c37c5834c5610166b4ae23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1771
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
This uses the not very documented ignore_default_attention_set_rules review API.
Change-Id: I650777bbbd24a1922f26967fbbd7da06d14b6782
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1513
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
It tries to write this to ~/.cache otherwise, which worked for the git
user but does not work for root (??)
Change-Id: I02d04da7d8e2b8782ce70bc72bce0b90c3961aa0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1546
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Changes the restic backup service to run as root, rather than git, and
include the PostgreSQL dumps in its scope.
The on-machine credentials have already been placed in the right
location in /var/backup/restic
Fixes: 27
Change-Id: Iae76357442f07596a2297ce7b6d51aae392d2074
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1541
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
... daily is just the default cron pattern for this, but we might also
want this to happen more frequently. Not sure yet.
Change-Id: I4e433fefebd93488891e765b5842fdb6537e3c6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1518
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Now that we've migrated over all the data to postgresql, we can get rid
of cl-prevalence as a dependency from Panettone along with all code that
mentions it.
Change-Id: I945f50a88fea5770aac5b4a058342b8269c0bea2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1495
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
It appears this didn't even *work* without a password, so we've been
forced into being more secure.
Change-Id: I4ff9d04961a703a85299dafb79e8447b0a933fc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1491
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is how panettone is currently connecting, so this needs to be here
in order for it to work. Shortly I'll update all of this to use
passwords, but for now this gets things up and running again
Change-Id: If87f4dbce0800dcbc4f7bf10e88f3e591410b416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1488
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Create a running Postgres database server along with a user and database
for Panettone, and pass configuration for it to the panettone module
Change-Id: I333994288131be328e62069382d6d40f8034c400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1466
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Otherwise I have to set TERM to something else so that I can actually use the
machine when I'm booted into Linux and it's incredibly tedious and I hate it.
Change-Id: Icfb5aacfea8cd6227743d29d9b07dc1b745d22c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1435
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Deploy Panettone to whitby as a systemd service, proxied to from an
nginx virtual host listening at b.tvl.fyi
Change-Id: I69755566151a45120e6b3453751af0e9291fa241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1339
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
I don't have time for bash's history.
Change-Id: I741107d33f09999ef43a7609079ad926e8127e69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1362
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
