Commit graph

452 commits

Author SHA1 Message Date
Vincent Ambo
54f59a5cc5 feat(ops/modules/www): Disable FLoC tracking for all TVL pages
.. this is actually likely not disabling it for some pages, that will
need this to be copy & pasted, but it's hard to tell just from the
nginx docs. We'll make sure after deploying.

Change-Id: I2fa6e31ca10835a206673b858594fa071e729d82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3020
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-04-20 10:44:43 +00:00
Vincent Ambo
5f19e8e6a7 refactor(ops/nixos): Ensure that pkgs == depot.third_party.nixpkgs
This is currently done ad-hoc in a bunch of our systems, but we should
just do it centrally.

The commit message is a bit of a lie, as this doesn't yet update
grfn's systems.

Change-Id: Ic771c1a1da78ec5de9cffbf94c296dce5e11fd84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3047
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-20 10:43:19 +00:00
Vincent Ambo
7e74e17931 fix(automatic-gc): Fix garbage collection script
It needs to refer to this by full path of course.

Change-Id: I911c876ba18877681accb722426314d92b9f2318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3042
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-04-18 09:44:04 +00:00
Vincent Ambo
75d507223b fix(modules/automatic-gc): Add nix-daemon to requisites
This will require the daemon to be running when launching GC, but
won't start it if it happens to not be running for some reason.

Change-Id: If48fe336030173f028428fc00a81d339ef4b8bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3015
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-14 19:53:14 +00:00
Vincent Ambo
cc903bc3b0 feat(ops/modules): Add module for automatically collecting garbage
Adds a module that automatically collects garbage based on disk space
thresholds, and configures it to run hourly on whitby.

This is implemented as an alternative to cl/2937, which I've been told
uses a Nix feature that doesn't actually work.

Under-the-hood this is simply a systemd timer running a shell script
which checks available disk space and runs GC when necessary.

Change-Id: I3c6b5de85b74ea52e7e16c53f2f900e0911c9805
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3014
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-04-14 19:37:21 +00:00
Luke Granger-Brown
a0cfa097e0 feat(whitby/grafana): use CAS SSO
There's a hard-coded list of Admin usernames for the moment. We should
revisit this and get an actual groups setup in LDAP that's propagated
through...

Change-Id: Ic3601f1a9753573076769f4912038e9f1b60e139
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2982
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-04-13 00:08:36 +00:00
Vincent Ambo
da5512f2e9 feat(whitby): Enable Grafana at status.tvl.su
Enables a Grafana service pointing to whitby's local Prometheus
instance, accessible at status.tvl.su.

I've no idea how to configure Grafana and if it's possible to link it
to CAS, but we'll see about that later.

Notes:
* the explicit fixpoint for whitby config has been removed as we
  have the `config` parameter available now
* backups are enabled for the Grafana storage location

Change-Id: If5ffe0c1a3378d1c88529129487c643642705fd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2948
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-04-12 22:01:05 +00:00
Vincent Ambo
f520bd40ca refactor: Replace 'depotPath' with 'depot.path'
Instead of having two ways of accessing the path to the depot (one of
which was stuttering, depot.depotPath) we settle on only one:
depot.path.

This was mostly used for NixOS module imports.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I2c0db23383fc34f6ca76baaad4cc4af2d9dfae15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2962
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-12 21:55:07 +00:00
Vincent Ambo
e09b44bdec chore(besadii): Stop passing explicit messages to Buildkite
Dropping the message field will make Buildkite use the commit messages
instead, which makes for much more readable build logs.

Change-Id: I1849f811632526893b700f117c9f6cf64888c329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2949
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-12 17:42:36 +00:00
Vincent Ambo
adc9d026e0 feat(whitby): Enable Prometheus instance on whitby
Enables Prometheus with a local node exporter, and nothing else for
now.

Some additional collectors have been enabled for things that might be
relevant on whitby:

* systemd: all our services run in systemd
* processes: might be interesting for build-related stats
* logind: might be interesting for interactive usage stats

Change-Id: I48dacdd9c68b4be9edff7b3cb6256dad562498c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2930
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-04-12 17:32:48 +00:00
sterni
0e6ac814c6 feat(ops/pipelines): pass --show-trace to nix-build
--show-trace should make it easier to debug tricky evaluation errors
without running nix-build -A ops.pipelines.depot locally again.

Change-Id: Ice540562c3b389fc2a49ec1fc0adacb17db2a528
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2947
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-12 15:50:43 +00:00
Griffin Smith
6266c5d32f refactor(users/glittershark): Rename to grfn
Rename my //users directory and all places that refer to glittershark to
grfn, including nix references and documentation.

This may require some extra attention inside of gerrit's database after
it lands to allow me to actually push things.

Change-Id: I4728b7ec2c60024392c1c1fa6e0d4a59b3e266fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2933
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-12 14:45:51 +00:00
Vincent Ambo
90281c4eac refactor(ops): Split //ops/nixos into different locations
Splits //ops/nixos into:

* //ops/nixos.nix - utility functions for building systems
* //ops/machines - shared machine definitions (read by readTree)
* //ops/modules - shared NixOS modules (skipped by readTree)

This simplifies working with the configuration fixpoint in whitby, and
is overall a bit more in line with how NixOS systems in user folders
currently work.

Change-Id: I1322ec5cc76c0207c099c05d44828a3df0b3ffc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2931
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-11 22:18:22 +00:00
Vincent Ambo
9073ac18c4 fix(pipelines/depot): Buildkite refers to branches by full ref
This change is required to run the  step on canon builds.

Change-Id: Ib3cebac67c9f5337b27a948f120b0a9ba834ef2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2932
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-11 21:18:59 +00:00
Vincent Ambo
d7b89df748 feat(ops/pipelines): Add gcroots for depot builds on canon
Adds a conditional build step that only runs on the canon branch, and
only if 🦆 (the status reporting step) succeeds, which creates a
new Nix GC root for all depot targets named `depot-canon`.

In practice this might be a bit racey, as canon builds are not
guaranteed to succeed in order (though it is likely). This shouldn't
matter much in practice: We only want to prevent rebuilds of the whole
world.

This fixes b/102

Change-Id: Id3d0bf4158bffcb1ed6929888a29d31609b6ece1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2904
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-11 20:09:53 +00:00
Vincent Ambo
6c3585f764 fix(tvl-buildkite): Set agents' primary group to buildkite-agents
This ensures files created by the Buildkite agents are always owned by
the same group, without having to manually chgrp afterwards.

Change-Id: Idbaedec43c16b2ee137d1a95719a05d46db8f900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2929
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2021-04-11 16:03:17 +00:00
Vincent Ambo
473604f567 refactor: Move nixpkgs attribute to third_party.nixpkgs
Please read b/108 to make sense of this.

This gets rid of the explicit list of exposed packages from nixpkgs,
and instead makes the entire package set available at
`third_party.nixpkgs`.

To accommodate this, a LOT of things have to be very slightly shuffled
around. Some of this was done in already submitted CLs, but this
change is unfortunately still quite noisy.

Pay extra attention to:

* overlay-like functionality that was partially moved to actual
  overlays (partially as in, the minimum required to get a green
  build)

* modified uses of the package set path, esp. in NixOS systems

Special notes:

* xanthous has been disabled in CI because of issues with the Haskell
  overlay
* //third_party/nix has been disabled because of other unclear
  dependency issues

Both of these will be tackled in a followup CL.

Change-Id: I2f9c60a4d275fdb5209264be0addfd7e06c53118
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2910
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-04-10 21:18:55 +00:00
Vincent Ambo
d8ec573bc7 feat(ops/nixos/www): Enable short links for b/ and cl/
This configures accepting requests for b/ and cl/ on plain HTTP ports,
and redirecting to b.tvl.fyi & cl.tvl.fyi appropriately.

Additionally, Panettone request URIs that only contain decimals are
redirected to `/issues/$request_uri` to enable issue short-links.

This fixes b/32.

Change-Id: I56954d8d69a3624267778b467520c509f4daa6c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2908
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 15:33:55 +00:00
Vincent Ambo
f205d8ee73 feat(gerrit): Auto link 'cl/123'-style shortlinks
Same as linking to bugs (e.g. b/108).

Change-Id: I447020bc07059c98c53322d745f961d8d471d9a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2919
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-04-10 15:28:59 +00:00
Vincent Ambo
9c5b5600ea refactor(ops): Consistent use of depot.third_party vs. pkgs
In preparation for the solution of b/108, we need to consistently use
`depot.third_party` for packages that are only packed in the TVL depot
and `pkgs` for things that come from nixpkgs.

This commit cleans up a huge chunk of these uses in //ops

Change-Id: I00faeb969eaa70760a26256274925b07998c2351
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2915
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 12:09:20 +00:00
Vincent Ambo
4b78875726 feat(tvl-buildkite): Add all buildkite agent users to a local group
This lets us grant permissions to them, e.g. on local folders.

Change-Id: I823ac414be1cb7d6baa4f17d95003709e5911b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2905
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-09 19:58:28 +00:00
Vincent Ambo
c021370213 refactor(whitby): Extract Buildkite agents into a module
There will be more Buildkite-agent specific configuration, and it's
already more than just the module setup, so extracting this makes
sense.

Change-Id: I56ce205c0cb4365317ed7ed5f2d525a0b425b861
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2906
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-09 19:08:24 +00:00
Luke Granger-Brown
6d008f6412 feat(gerrit-tvl): add Buildkite-backed Checks plugin implementation
This small(*) pile of JavaScript queries the Buildkite API for the
latest builds for the depot and displays the results in the rebooted
Check UI.

Change-Id: I7025a1c6d0d0afa000a9df4682133e03824ea10d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2881
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-07 11:19:04 +00:00
Luke Granger-Brown
21765a1407 feat(ops/gerrit-tvl): init TVL Gerrit plugin
This is just going to be a grab bag of things which do TVL-specific
things to Gerrit, whether that be exposing new Prolog predicates or, as
I intend to do as the first thing, expose Buildkite builds as checks.

Change-Id: Iaeab987a1fdbd078b85e274691c986489903bf3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2872
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-06 18:43:04 +00:00
Vincent Ambo
0f1d3de26f feat(whitby): Configure nix-serve on cache.tvl.su
Having a slow cache is better than having no cache.

Change-Id: Ie3cfcd4a2937d90b0e2ad899816bc31ae806631f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2847
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 18:54:19 +00:00
Luke Granger-Brown
8ae128af77 feat(monorepo-gerrit): use CAS for authentication
This drops the old LDAP configuration and uses CAS instead. All hail the
hypnotoad.

Change-Id: I515a213f09073bb52bfb75afe2988b935a076087
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2783
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-03 19:11:18 +00:00
Vincent Ambo
5111d9d493 feat(www/tazj.in): Add a temporary route for serving static blobs
Until I come up with a better idea.

Change-Id: Ie44cae4c2df264cbe1a70f5ebcca814262dd2800
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2771
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-04-03 11:44:51 +00:00
Luke Granger-Brown
0456de3733 chore(ops/nixos): finish removal of depot.nix
All user configs and modules have been migrated to using the depot
module parameter. All hail the hypnotoad.

Change-Id: Ic05c61fccba3ac505a339283b6ef3105a2d0711c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2765
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-02 18:00:14 +00:00
Luke Granger-Brown
b35e358eb5 refactor(ops/nixos): migrate to depot module arg
Previously the depot argument was provided as config.depot, but the "new
way" of doing things (which is more like the args list provided in the
rest of the depot) is to provide this as the "depot" NixOS module
argument instead.

Change-Id: Ib48b1c7c1bdff9c1eb0618c6cbacc22b651f5f98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2763
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-02 18:00:14 +00:00
Luke Granger-Brown
4bfcd09682 refactor(ops/nixos): add "depot" argument to NixOS modules
For the moment I've opted to not import all of the other things we'd
usually provide to things imports via readTree, because I think it's a
bit dangerous to accidentally overwrite things like NixOS' notion of
"lib" with our own version.

So for the moment, baseModule provides only "depot".

Change-Id: I3db9132a3d9227055d4c1b00f02effcb84edcc53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2760
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-02 18:00:14 +00:00
Vincent Ambo
817cd6f166 feat(ops/nixos/www): Enable tvl.su aliases for dev tools
This is not for the domain root though, as that's going to be
something else eventually.

The canonical URLs are the .fyi ones (at least for now), and some of
these tools will eventually generate links that make user sessions
started from *.tvl.su converge on *.tvl.fyi.

Relates to b/98

Change-Id: I1c3bcf72a3063059002e4b0bdd57c269a410a8bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2758
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-04-02 11:10:51 +00:00
sterni
8cc1dcf588 feat(ops/users): add milan to users
Change-Id: I77e5e9a0ae1bf2ee59ac4967c5481b9044f97934
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2757
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-02 10:21:37 +00:00
sterni
27b300fe34 feat(ops/whitby): add sterni to trusted users
I am somewhat trustworthy… maybe? Also I tend to gc depot stuff so ssh
serve would be neat.

Change-Id: I4672f20a32a756692dd156b5e40e5a7f37ba5ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2660
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-02 10:21:32 +00:00
Vincent Ambo
53d8dd6a1e feat(ops/nixos/www): Serve rendered Tvix component SVG (hack!)
This is a quick hack to make it possible to view the rendered SVG on
https://code.tvl.fyi/about/tvix/docs/components.md

We want to be able to do this sort of thing dynamically in the future,
but we can't yet, so ... well. Deal with it.

Change-Id: Id2b819679d748b6f517018a9c6e72d5c1d806c4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2743
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2021-03-31 23:10:54 +00:00
tazjin
99148c9b8f revert(monorepo-gerrit): Revert to using cgit for Gerrit
This reverts commit 3b05be2fd0.

Reason for revert: Sourcegraph still does not support fetching arbitrary refs, so we'll have to wait until its Gerrit integration lands before this will work correctly.

Change-Id: Icee82c50f92c34ba1741b608449aed16538ccbaa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2721
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-03-31 21:48:06 +00:00
Vincent Ambo
03a2616ccb feat(ops/dns): Add GSuite site aliases for tvl.su
Change-Id: Ib18b5bef23a198e7ca031f48290cf0ff0655d8dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2687
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-03-27 11:06:56 +00:00
Florian Klink
372b35dffa feat(ops/nixos/whitby): add flokli user
Change-Id: Ibdb5b498f8bbc837fffdb38cdf95499b279773aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2683
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2021-03-26 20:31:48 +00:00
Vincent Ambo
93af4644ab chore(ops/users): Purge some inactive users from LDAP
Change-Id: Iab2d2d6b7096ef302ea2fd8b051041426c6c8ca6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2670
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-03-26 20:00:22 +00:00
Florian Klink
3026891302 feat(ops/users): Add flokli to users
Change-Id: I87ca0f20663ff858237f641ef2245778601e0416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2671
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-03-26 19:31:37 +00:00
Vincent Ambo
743993d7b1 feat(ops/dns): Configure email settings for tvl.su.
Change-Id: Ida5beca7b4efd39fad15ba220dc1fc887ed79b4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2669
Tested-by: BuildkiteCI
Reviewed-by: adisbladis <adisbladis@gmail.com>
2021-03-26 19:31:10 +00:00
Vincent Ambo
5a90c8276b feat(ops/dns): Add Google Workspace verification for tvl.su.
Change-Id: I44db2bca7aa5814bbefd8943d727cc66ab800fd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2668
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-03-26 18:26:34 +00:00
Griffin Smith
f2963cffcd fix(ops/whitby): Set tcp congestion control to bbr
Some quick testing shows that this improves my data transfer speed to
whitby by roughly 200%.

Change-Id: Id94de975b1ae0930f8d0fe038582dbac0037676c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2659
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: ben <tvl@benjojo.co.uk>
2021-03-26 02:42:37 +00:00
Vincent Ambo
d34c527372 refactor: Replace some uses of builtins.toFile with pkgs.writeText
I'm looking at removing some of these because they can cause
unnecessary build steps during CI pipeline generation.

Change-Id: I84742968918090c050d2eedab8a1b42692632a42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2655
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-03-25 19:14:36 +00:00
Vincent Ambo
a747b46f19 chore(ops/nixos): Update Sourcegraph to 3.26.0
Reading through the changelogs, this includes the following two
changes that may require us to do something:

* For users of single-image Sourcegraph instance, please delete the
  secret key file /var/lib/sourcegraph/token inside the container before
  attempting to upgrade to 3.21.x.

* A campaigns.restrictToAdmins site configuration option has been
  added to prevent non site-admin users from using campaigns.

Change-Id: Ieacf85a9059ad5222800f8d7d4a43435f489a39f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2638
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-03-22 20:46:31 +00:00
Vincent Ambo
1c719cdc14 feat(ops/dns): Add status subdomain
I want to host something like Vigil[0] on this to show the status of
Gerrit, SourceGraph and maybe other components.

(Yes, the status page will be on the same infrastructure ... but this
is mostly for service failure cases).

[0]: https://github.com/valeriansaliou/vigil

Change-Id: If71496300b94035976a685d9bf166d525d89fc5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2637
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-22 16:15:07 +00:00
Vincent Ambo
2f7cb2b6c4 chore(whitby): Remove SSH key from root
This was a leftover from the time we were installing.

Change-Id: Id875b907d7f76081a45e7f8f2666b7fba6aefc86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2632
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-03-21 18:04:22 +00:00
Vincent Ambo
6e94b3ca2f feat(tazjin/nixos): Initial check in of new host (tverskoy)
This is my new X13 AMD Thinkpad, on which many fun things will be done.

Change-Id: I4de114a8c5ebb37d2f4844f407d2dc0e7cc9557e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2620
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-03-21 00:55:58 +00:00
sterni
c32e8424be refactor(ops/dns): use drvTargets for meta.targets population
Since we have a dedicated util for this, we may as well use it
to reduce code duplication.

Change-Id: Ie52647be8c786d0b6a4dceb2fa6778b94625fafc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2604
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-03-15 22:16:19 +00:00
Vincent Ambo
5b9229186f feat(ops/dns): Configure tvl.su zone
Change-Id: I6016d92e9c231a257e06644dfcf44a4aaa12ac4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2601
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-03-15 21:23:35 +00:00
Vincent Ambo
b4e87f8254 feat(ops/dns): Import tvl.fyi DNS zone into depot
Imports the current state of the tvl.fyi zone and configures simple CI
checks on the file format.

No deployment automation exists for this (yet?).

Change-Id: Ia7d72e02b9f6d3adef994c5dc1898cc0df9dfcfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2600
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-15 21:23:35 +00:00
Adam H
4d193f2395 feat(users/adisbladis): Add to users
Change-Id: I2a3532605c602dd6ba44a6c723333db219a55907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2599
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-03-13 19:55:24 +00:00
Vincent Ambo
c9726d8a00 chore(ops/journaldriver): Expand wildcard imports
... to appease Profpatsch.

Change-Id: Id8576645a6920312c2304ea7880524d9cda8e21b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2544
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-02-24 14:07:22 +00:00
Vincent Ambo
3a45e029af fix(ops/www/tazj.in): Force SSL for git.tazj.in redirect
Change-Id: If5b8096cb693d96936f9b954e2ebe3dc9b63af66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2521
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-02-10 17:56:29 +00:00
Vincent Ambo
8d7c24d3db fix(ops/www/tazj.in): Redirect git.tazj.in to our cgit
Change-Id: Ia0be95e2618aeb4f8d394a8e3602c73faec0d72f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2508
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-02-10 17:38:05 +00:00
sterni
e91d5e4e61 fix(config): remove ciBuilds inherit
The ciBuilds attribute seems to no longer exist and it breaks the
evaluation of the config attribute. It's only appearance was in
besadii which doesn't actually use the attribute.

Removing the ciBuilds inherit fixes these issues.

Change-Id: Ibbf3413ba6efe10ad868cf57cf0711d574860f97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2487
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-02-06 17:54:18 +00:00
Profpatsch
e7ad8143e4 fix(ops/piplines/static-pipeline): add --show-trace to nix-build
Change-Id: Ib0473f916b1436934844e620ce981f52d11e8512
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2467
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-01-30 09:02:09 +00:00
Vincent Ambo
8f57ca92bd chore(3p|nix): Remove typed Go
Nobody has actually done any experimentation with typed Go, so we're
getting rid of it for now - it's causing annoying IFD during build
graph generation.

Change-Id: Ibac3dea98ebed1b3ee08acda184d24c500cf695d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2458
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-01-30 08:20:45 +00:00
multi
4ac51ea504 chore(users/multi): remove user from the depot.
This commit removes my user directory in the depot, my user account on whitby,
my entry in the LDAP database, and my entry in the website graph. I've had my
fun with TVL, but I want to move on to spending time on some other things.

This additionally removes aranea from the website graph, which they have
requested in private.

Change-Id: I2d098c8fe239f20d9f6c6cbf66a3dfb4a955a4cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2436
Tested-by: BuildkiteCI
Reviewed-by: multi <depot@in-addr.xyz>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-01-23 21:13:39 +00:00
V
29db630a39 chore: Remove banned user
Change-Id: Icd61f7c567a327c74a4f381168e94737b2b30702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2422
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: tazjin <mail@tazj.in>
2021-01-19 10:30:19 +00:00
sterni
2d136e0327 feat(todolist): use static slapd user data for knownUsers
Since the slapd data is static and generated using nix, we can simply
move the user list into ops/users, so it's recognized by readTree and we
can use it as ops.users both in ops/nixos/tvl-slapd and web/todolist as
a general purpose user registry for depot.

Update docs/REVIEWS.md as well.

Change-Id: I35caaaab70a5578c47cedc7f33077dd513766290
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2419
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-01-18 23:18:55 +00:00
Vincent Ambo
c033229a61 chore(ops/whitby): Move ACME registrations to an @tvl.fyi address
Change-Id: I371550aa456c0fb64da4789feed494cc50497522
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2410
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-01-18 00:54:33 +00:00
Vincent Ambo
e4976c49dc feat(ops/nixos): Serve tazj.in from whitby temporarily
camden.tazj.in (the host in my flat) is going down as my belongings
are being moved into storage.

Change-Id: Id66512fd2ec6dbdcb6dfc3862af49cfadb15cfa1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2405
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-01-17 05:42:55 +00:00
multi
2fe90c34c0 feat(ops/nixos/whitby): Enable remote use of whitby for my Thinkpad.
My main workstation is a Thinkpad without a great deal of compute
power available, so enabling the use of whitby as both a substituter
(services.sshServe) and a remote builder (openssh.authorizedKeys) will save me
some time when working on nix things and depot things.

Change-Id: I17bfcbb9860f42fb667603ad819e38e82e6052da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2399
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2021-01-15 13:06:33 +00:00
sterni
e93a2fc48f feat(ops/nixos/whitby): add sterni user
Change-Id: Ia6790913ea2777a9d4ca89830436623766991c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2368
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-01-13 22:05:33 +00:00
sternenseemann
dd323b5c0d feat(tvl-slapd): add sterni to slapd
Change-Id: I4b832f60c69e1bdd1a6bf0595d523c052aa8f794
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2348
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-01-11 10:58:52 +00:00
Vincent Ambo
88bf43878f chore(3p): Bump NixOS channels to 2020-12-28
Changes:

* ops/nixos/tvl-slapd: The NixOS module for OpenLDAP has removed the
  ability to configure OpenLDAP directly and now forces users to use
  some kind of weird Nix->OLC mapping that is mostly undocumented.

  This moves the config we need to the new format in a way that may or
  may not work and does the other arbitrary dance steps that someone
  decided to impose on us. Note that this now throws lots of warnings,
  but I can't be bothered to fix them.

* 3p: Random package removals accomodated

* users/glittershark: Pin grfn's kernel to 5.9, because the CK patch
  is not yet updated for 5.10

* users/glittershark: Update vendor hash for pg-dump-upsert, I suspect
  this changed because of something in the Go build machinery in
  nixpkgs. The deleteVendor flag also has no effect anymore and has been
  removed.

* users/glittershark: agda build is broken, commenting out development
  home-manager environment until it can be fixed

* third_party/haskell_overlay: updating random needs upper boundarles
  of a few dependencies relaxed (curse them)

* third_party/gerrit_plugins: for some cursed reason the fixed-output
  hash of the gerrit owners plugin fetchgit changed, updated.
  Same for the checks plugin.

Change-Id: Ica37995fe8039d3ba80eab643867f98795c56734
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2295
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-01-09 13:21:00 +00:00
Vincent Ambo
4a86a06466 chore(whitby): Double number of build users
more = betterer

Change-Id: I6d5414d6ebb087e7f9fb912d5a514c31ebcd8b7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2296
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-12-26 14:56:20 +00:00
Vincent Ambo
c3bbb861b8 fix(whitby): Include lukegb's & grfn's SSH keys in initrd
Change-Id: I8921d645b1a81510e04314e519195c1c01d3fd14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2286
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-12-20 22:03:39 +00:00
Vincent Ambo
86ec8c1b94 fix(whitby): Disable git's gc.autoDetach feature
This feature can cause object removal to happen while the git folder
is in use in Buildkite, causing CI to fail semi-reegularly.

Change-Id: Ide1a9b2f1761be029e97a058c1983b4cff5e27bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2285
Tested-by: BuildkiteCI
Reviewed-by: multi <depot@in-addr.xyz>
2020-12-20 17:54:19 +00:00
Griffin Smith
d4fb573cf7 feat(gs/system): Init yeren
My new work laptop, a dell XPS 13.

Change-Id: Ieab06622c9b280182025edfa63adf649e5fc70d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2205
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-11-30 00:03:50 +00:00
Luke Granger-Brown
b344ae8783 fix(cl.tvl.fyi): Correct Gerrit shortlink redirects.
Before: http://cl.tvl.fyi/123 -> https://cl.tvl.fyi:80/c/depot/+/123/
After: http://cl.tvl.fyi/123 -> https://cl.tvl.fyi/c/depot/+/123/

I think Jetty changed it's behaviour, and Gerrit is now configuring it
incorrectly.

Fixes #88.

Change-Id: I9238c0922b9f627e06eb81fa99dc748dada8909a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2202
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-29 11:50:53 +00:00
Jamie McClymont
3cbef06629 feat(tvl-slapd): add jamie to slapd
o/

- Jamie

Change-Id: I9c21e9a58c4514160f08133465a9cca720055cbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2148
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-11-26 00:36:06 +00:00
Griffin Smith
64ef09c475 feat(whitby): Move wigglydonke.rs to whitby
Mugwump is too unstable for such an important internet service

Change-Id: Ic714200ce5ce51f366777f538b4a6f443f010960
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2124
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 22:55:49 +00:00
Griffin Smith
9f4d37e5df feat(ops/nixos): Give all nixoses a config.depot
Add the depot.nix module and a depot config option to all nixos system
derivations that're build through the `bin/rebuild-system` machinery.
I can't imagine a scenario where we wouldn't want this level of
integration.

Change-Id: Ieeb98db2eee23919256adb4654bc45d540e055ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2128
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-11-22 21:59:58 +00:00
Vincent Ambo
73c862279a feat(ops/pipelines): Check in the static pipeline
This file represents the static pipeline which is configured in the
Buildkite web UI. Updates to this file should be applied in the admin
interface.

These steps are responsible for launching the dynamic pipeline
evaluation, or falling back to the fallback pipeline if evaluation fails.

Change-Id: I6d7dd623cde65e8c69faea729f737c9bba00c2fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2103
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-17 22:33:11 +00:00
Vincent Ambo
1857334d37 feat(ops/pipelines): Add a fallback Buildkite configuration
This adds a simple fallback Buildkite pipeline configuration which
always fails the pipeline, but correctly reports back the failure
status.

Note that this also requires changes in the Buildkite configuration
that is not in version-control.

Relates to b/66.

Change-Id: I6802a6f76448c3893798a06d514e6ccba0f50dd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2102
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-17 22:33:11 +00:00
Vincent Ambo
77097f8056 feat(ops/panettone): Add configuration for irccat
Adds configuration options for the (inconsistently named) environment
variables that configure irccat integration with Panettone.

The defaults match the irccat setup on whitby.

Change-Id: I6857512a2e3f29f16777493eb981cc69ce3c045f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2080
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-17 22:00:52 +00:00
Vincent Ambo
1442c5c8ac feat(whitby): Enable irccat module
Enables irccat, running as 'tvlbot' on ##tvl and ##tvl-dev and listening on TCP 4722.

Change-Id: Ia1eb533d0aacb0c15d6b3fa1cfd854ffbce27d23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2075
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-11-08 18:38:08 +00:00
Vincent Ambo
cbfcf14301 feat(ops/irccat): Add a NixOS module for launching irccat
This module configures irccat by creating a JSON configuration file
from a user-supplied Nix struct (this is not checked for correctness),
and merging it recursively with secrets from
`/etc/secrets/irccat.json` at service launch time.

This way we get the ability to configure (most) options declaratively
via Nix, while providing the secrets outside of Nix.

Side note: We need to figure out a secrets distribution mechanism.

Tested: Wrote a dummy config in whitby/default.nix locally and checked
that this builds, but I have not actually run the service yet. I
expect that some minor tweaks will end up being necessary.

Change-Id: I02a2e8dc40a7f8417fd77afcf8a12ac3df117988
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2074
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-08 18:38:08 +00:00
Vincent Ambo
e84f9ef0ad fix(whitby): Use new IRC bouncer location for clbot
... I found this location in the logs, because the certs are now valid
for this, but I'm not actually sure if it's right.

Change-Id: I5ac88073e3bf6a95fead4c1d34515622c4416c6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2070
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-11-05 14:22:01 +00:00
Griffin Smith
e39b7e002c feat(ops/nixos/paroxysm): Set Restart = "always"
Sometimes (like today) paroxysm crashes. We'd like it to restart if that
happens.

Change-Id: I98841096bcd6605c4279744ae5c65a9c92092a21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2069
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-05 13:30:42 +00:00
Elis Hirwing
2a6be2b484 feat(tvl-slapd): add etu to slapd
Change-Id: I39ecf2167fd65f305853bf0e48c6208d94a5bf1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2055
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-10-22 13:26:25 +00:00
htbf
41f1b01ba4 feat(tvl-slapd): add htbf
Change-Id: I6da03700708bcafc4f476b01c0a27d27fb85cc4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2050
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-10-18 02:20:19 +00:00
Cynthia Revström
df6a1d47d2 chore(tvl-slapd): use ARGON2 for cynthia
Change-Id: I81efffe384644cc2d2a625fc96ef5264dedd76ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2019
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-10-05 13:00:59 +00:00
Jonas Höglund
b2870615f7 feat(whitby): add firefly user
Change-Id: Ib785577c173795d5cc6ccd7a3ee7e6a568439a0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2013
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-28 13:42:08 +00:00
Jonas Höglund
3a5e908111 feat(tvl-slapd): add firefly
Change-Id: I28b71a429f2093e2ff3d7148cfaa2425f322dfea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2010
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-27 23:38:48 +00:00
Cynthia Revström
8e31495b9d feat(whitby): add cynthia owo
Change-Id: Id9e06ce8645ec2dbe1167d2b0b023159d3e91487
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2008
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-27 23:27:23 +00:00
Ben Cartwright-Cox
e3d0585fcd feat(tvl-slapd): add ben to slapd
Change-Id: Iadec1d04b086f878c408a8867778f6bd75254dfc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2005
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-27 22:56:09 +00:00
Vincent Ambo
afc07b1b74 chore(whitby): Double the number of build agents again
The main bottleneck of our builds right now is Nix evaluation, which
means that most of the time is spent idling during builds.

Since we're evaluating in parallel, lets give it a few more builders.

I don't want to go all the way to 64 immediately to first see if we
get any adverse effects from highly concurrent builds running
concurrently (if we do we could group them into different "concurrency
groups" in Buildkite).

Change-Id: Ibc3f89fb59cb4ee471b152ff36887ffe2b39f8f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1932
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-06 14:42:40 +00:00
Griffin Smith
7a83b8fe40 fix(ops/monorepo-gerrit): Escape escape characters with more escape characters
Escape!

Change-Id: Idec7d17feda13ccf1ec03c41106ec6ed581d9e47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1901
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-31 23:21:45 +00:00
Vincent Ambo
2afa22e512 feat(whitby): Double the number of Buildkite agents
Allow 16 things to happen in parallel, which is useful now that the
CI granularity is on a per-target level.

Change-Id: Ie65dd119ea0666618fbb249613e70a68276db834
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1902
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-31 23:21:16 +00:00
Vincent Ambo
9c482d6238 feat(ci): Add subtarget support for builds
We have naturally evolved a distinction between logical and physical
targets.

Physical targets are those which correspond directly to a tree
location on disk and can be built with `-A path.to.files`, while
logical targets are those that are exported from within an expression
but do not have a corresponding file on disk.

This change adds support for exporting logical targets from any tree
location by adding a `meta.targets` attribute containing keys into
itself, which will be consumed by the CI target gathering logic and
included in the generated pipeline.

Note that the labels for subtargets are syntactically different to
emphasise that they do not correspond to a file location. For example,
this change enables 'ops.nixos.whitbySystem' as a subtarget, which is
labeled in CI as `ops/nixos:whitbySystem`.

Change-Id: Ied09647a62c2ba98e3914548e3742ad422c63ecf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1893
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-31 23:14:11 +00:00
Vincent Ambo
61d2d2d503 feat(ops/pipelines): Dynamically generate CI pipeline from targets
Create the pipeline by outputting a file that contains nix-build
invocations for each target's *derivation path*.

Each invocation has a generated Nix expression passed to it with `-E`
which fetches the correct target from the tree while correctly
handling targets with strange characters (such as in Go-packages).

This makes it possible to run target-level granular pipelines. We're
getting somewhere!

Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-31 23:14:11 +00:00
Griffin Smith
9b4986c5ed feat(monorepo-gerrit): Auto-link panettone issues
Auto-link b/\d+ in gerrit comments + patch comments to panettone issues.
This is a departure from the previous syntax of #\d+ that we were using
historically, but has been requested by multiple people

Fixes: b/40
Change-Id: I8e556eb6b2f55ae80d945b86b51041751ecb053b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1888
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-30 22:16:50 +00:00
Vincent Ambo
4ff9d5dee8 feat: Implement automatic CI target detection for the depot
Automatically walk the entire depot tree and pick out things that are
"buildable", then include them in the attribute `ci.targets` (which is
now also the target for CI builds).

A long time ago, in a land far away, we (well, I, at the time) had a
prototype of this which ran into constant issues with infinite
recursions while trying to walk the tree. In fact, this is why
readTree originally gained the `__readTree`-attribute which marks
things that were imported automatically.

Based on some code edef whipped up earlier (with the breakthrough
being that we also add the attribute to top-level folders, which
suddenly resolves a whole bunch of problems), I've now implemented
this actually working version.

At the moment all builds still happen as one big bag of builds, but at
some point we will granularise this.

Change-Id: I86f12ce7f63dae98e7e5c6646a4e9d220de783f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1854
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-26 23:49:32 +00:00
Vincent Ambo
88317aea0d fix(besadii): Remove unused Nix code from the derivation
This was used previously when build granularity was besadii's task,
which it no longer is.

Change-Id: I6df2db1ed4730a7953199b7b48aa9ad916418b22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1853
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
5ad61ee408 chore: Add __readTree attributes on systems that need to be built
This is a temporary state (TODO added) to be picked up by the new CI
logic.

Change-Id: Id4702740ffd18325088e2a8a0c6157a8cee7ccf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-26 23:49:32 +00:00
Vincent Ambo
d4824b64f6 feat(whitby): Enable log forwarding via journaldriver
Change-Id: I474159acfe514f6f2eb7867e4eba854016590ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1836
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-08-23 20:45:18 +00:00
tazjin
674d921202 revert(whitby): Use Tvix as the system Nix on whitby
This reverts commit 8fa3087067.

Reason for revert:

This almost worked. We discovered two important issues:

- The daemon startup does not correctly handle the socket passed in by systemd.

- There is some issue with chunking of large calls, running a build for ci-builds resulted in:

tazjin@whitby /depot (canon)> nix-build -A ciBuilds.__allTargets
E20200821 01:42:22.846053 12601 shared.cc:306] error: Rpc call addTextToStore to unix:///nix/var/nix/daemon-socket/socket failed (RESOURCE_EXHAUSTED): Received message larger than max (10889961 vs. 4194304)

Change-Id: Ic5ba4ef06a4953cf71a36b139fe25ea673cb6fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1802
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 02:01:33 +00:00
Vincent Ambo
8fa3087067 feat(whitby): Use Tvix as the system Nix on whitby
... this is going to break so much stuff. Lets have some fun.

Change-Id: If0185e0323391c7055d47b797083bb5afde57cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1829
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 01:37:25 +00:00