The main bottleneck of our builds right now is Nix evaluation, which
means that most of the time is spent idling during builds.
Since we're evaluating in parallel, lets give it a few more builders.
I don't want to go all the way to 64 immediately to first see if we
get any adverse effects from highly concurrent builds running
concurrently (if we do we could group them into different "concurrency
groups" in Buildkite).
Change-Id: Ibc3f89fb59cb4ee471b152ff36887ffe2b39f8f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1932
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Allow 16 things to happen in parallel, which is useful now that the
CI granularity is on a per-target level.
Change-Id: Ie65dd119ea0666618fbb249613e70a68276db834
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1902
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
We have naturally evolved a distinction between logical and physical
targets.
Physical targets are those which correspond directly to a tree
location on disk and can be built with `-A path.to.files`, while
logical targets are those that are exported from within an expression
but do not have a corresponding file on disk.
This change adds support for exporting logical targets from any tree
location by adding a `meta.targets` attribute containing keys into
itself, which will be consumed by the CI target gathering logic and
included in the generated pipeline.
Note that the labels for subtargets are syntactically different to
emphasise that they do not correspond to a file location. For example,
this change enables 'ops.nixos.whitbySystem' as a subtarget, which is
labeled in CI as `ops/nixos:whitbySystem`.
Change-Id: Ied09647a62c2ba98e3914548e3742ad422c63ecf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1893
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Create the pipeline by outputting a file that contains nix-build
invocations for each target's *derivation path*.
Each invocation has a generated Nix expression passed to it with `-E`
which fetches the correct target from the tree while correctly
handling targets with strange characters (such as in Go-packages).
This makes it possible to run target-level granular pipelines. We're
getting somewhere!
Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Auto-link b/\d+ in gerrit comments + patch comments to panettone issues.
This is a departure from the previous syntax of #\d+ that we were using
historically, but has been requested by multiple people
Fixes: b/40
Change-Id: I8e556eb6b2f55ae80d945b86b51041751ecb053b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1888
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Automatically walk the entire depot tree and pick out things that are
"buildable", then include them in the attribute `ci.targets` (which is
now also the target for CI builds).
A long time ago, in a land far away, we (well, I, at the time) had a
prototype of this which ran into constant issues with infinite
recursions while trying to walk the tree. In fact, this is why
readTree originally gained the `__readTree`-attribute which marks
things that were imported automatically.
Based on some code edef whipped up earlier (with the breakthrough
being that we also add the attribute to top-level folders, which
suddenly resolves a whole bunch of problems), I've now implemented
this actually working version.
At the moment all builds still happen as one big bag of builds, but at
some point we will granularise this.
Change-Id: I86f12ce7f63dae98e7e5c6646a4e9d220de783f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1854
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
This was used previously when build granularity was besadii's task,
which it no longer is.
Change-Id: I6df2db1ed4730a7953199b7b48aa9ad916418b22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1853
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This is a temporary state (TODO added) to be picked up by the new CI
logic.
Change-Id: Id4702740ffd18325088e2a8a0c6157a8cee7ccf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This reverts commit 8fa3087067.
Reason for revert:
This almost worked. We discovered two important issues:
- The daemon startup does not correctly handle the socket passed in by systemd.
- There is some issue with chunking of large calls, running a build for ci-builds resulted in:
tazjin@whitby /depot (canon)> nix-build -A ciBuilds.__allTargets
E20200821 01:42:22.846053 12601 shared.cc:306] error: Rpc call addTextToStore to unix:///nix/var/nix/daemon-socket/socket failed (RESOURCE_EXHAUSTED): Received message larger than max (10889961 vs. 4194304)
Change-Id: Ic5ba4ef06a4953cf71a36b139fe25ea673cb6fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1802
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
... this is going to break so much stuff. Lets have some fun.
Change-Id: If0185e0323391c7055d47b797083bb5afde57cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1829
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Adds a lot more information about what is actually going on here, as
well as links to relevant things such as our Monorepo document.
I'm aware that I didn't make all `//...` links clickable, but I
realised at some point that it might be easier to just update cheddar
to support highlighting those natively :-)
Change-Id: Icbb212a6c07a5cf38ab8e65d83a64bec783eb8d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1768
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Adds the ability to post to multiple channels by simply running
multiple instances of clbot.
We should probably implement support for this in clbot itself, but
right now I can't be bothered to write Go.
Change-Id: I5cffd0dc10a7f6cc19c37c5834c5610166b4ae23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1771
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
This uses the not very documented ignore_default_attention_set_rules review API.
Change-Id: I650777bbbd24a1922f26967fbbd7da06d14b6782
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1513
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
It tries to write this to ~/.cache otherwise, which worked for the git
user but does not work for root (??)
Change-Id: I02d04da7d8e2b8782ce70bc72bce0b90c3961aa0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1546
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Changes the restic backup service to run as root, rather than git, and
include the PostgreSQL dumps in its scope.
The on-machine credentials have already been placed in the right
location in /var/backup/restic
Fixes: 27
Change-Id: Iae76357442f07596a2297ce7b6d51aae392d2074
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1541
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
... daily is just the default cron pattern for this, but we might also
want this to happen more frequently. Not sure yet.
Change-Id: I4e433fefebd93488891e765b5842fdb6537e3c6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1518
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Now that we've migrated over all the data to postgresql, we can get rid
of cl-prevalence as a dependency from Panettone along with all code that
mentions it.
Change-Id: I945f50a88fea5770aac5b4a058342b8269c0bea2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1495
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
It appears this didn't even *work* without a password, so we've been
forced into being more secure.
Change-Id: I4ff9d04961a703a85299dafb79e8447b0a933fc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1491
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is how panettone is currently connecting, so this needs to be here
in order for it to work. Shortly I'll update all of this to use
passwords, but for now this gets things up and running again
Change-Id: If87f4dbce0800dcbc4f7bf10e88f3e591410b416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1488
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Create a running Postgres database server along with a user and database
for Panettone, and pass configuration for it to the panettone module
Change-Id: I333994288131be328e62069382d6d40f8034c400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1466
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Otherwise I have to set TERM to something else so that I can actually use the
machine when I'm booted into Linux and it's incredibly tedious and I hate it.
Change-Id: Icfb5aacfea8cd6227743d29d9b07dc1b745d22c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1435
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Deploy Panettone to whitby as a systemd service, proxied to from an
nginx virtual host listening at b.tvl.fyi
Change-Id: I69755566151a45120e6b3453751af0e9291fa241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1339
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
I don't have time for bash's history.
Change-Id: I741107d33f09999ef43a7609079ad926e8127e69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1362
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
... also bootstraps her user directory to store the key in.
Change-Id: Iecd341c655adc7d81be5ce9eb765c531b7512e80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1361
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
This is inline with how other user keys are managed.
Change-Id: Ica0b3b30336aee02a78e019b13e1cf576e4e1943
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1360
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Note that this is not yet updated automatically, so the page will be
stale until somebody rebuilds whitby.
Change-Id: I91f4b03c9309aed289df055fac292a214dca7668
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1297
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
Not having this set led to gerrit setting the committer to
"qyliss <hi@alyssa.is>", which is wrong.
Change-Id: I3fe02264e22dd6d739575b34ceb1221d1d6a9d98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1267
Tested-by: BuildkiteCI
Reviewed-by: qyliss <hi@alyssa.is>
Breaking change detection will run but not enforce.
Emoji of water buffalo was chosen by @pedge fiat in the bufbuild slack.
Change-Id: Ie292f2bfddc0e3bc512e4a138c0b5d0fa2603bad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1247
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
The Hetzner DNS servers were unhappy after today's Cloudflare outage,
and that broke some of our builds - this wouldn't have happened with
Google DNS!
Change-Id: Ib74c6de9526e739f55d4a9830d945ece35b72138
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1259
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
... and enable it on whitby
Change-Id: Ife45f15227f9d95823ebd3b97d2a17175b84eaff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1064
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
There is only one minor configuration change: CLBot now connects to
cl.tvl.fyi, instead of localhost, because Gerrit is still on camden.
Change-Id: Ibd8d46ec2c18312a270471a2f0be3e58eaf0cbab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1062
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
This is required for the Gerrit setup.
Change-Id: I02e03dafe36e6c47ffabf4d590e0c6f1dea027e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1061
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
This prevents a request that takes >1s on each page load.
Change-Id: Ic91bb602e3059b1f17681aa468739bb0a103f8cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1003
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
The upstream module is not flexible enough for my needs, so I made my
own.
Change-Id: Ie9f786da7eb8c878e0782b07a075c064ad8cd253
Reviewed-on: https://cl.tvl.fyi/c/depot/+/953
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
- X-Forwarded-Proto support so it knows it's behind TLS
- Remove extraneous logs and just log to stdout so it's caught be systemd
Change-Id: I650777bbfd24a1922f26967ffff7da06d14b6639
Reviewed-on: https://cl.tvl.fyi/c/depot/+/952
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This adds a first crack at one idea for a generic, non-user-specific
rebuild-system script to ops.nixos.rebuild-system. The idea here is that
we enumerate all the nixos systems stored in the monorepo (similarly to
what we do for ci-builds right now) then search through them by hostname
to find the one matching the hostname of the current system, which is an
attempt at a more generic version of tazjin's rebuilder script which
does the same thing but with an explicit case block.
As a caveat, it feels like there's a slight possibility that this way of
finding systems is going to get slow to evaluate - on my system it feels
fine but if it grows out of hand it's probably feasible to just bake
this into the built script as a dynamically generated case statement.
Change-Id: I2e4c5401913b6f4d936ab48ba2f95f96e0e78eb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/894
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
We now use the actual 'Verified' label instead of Code-Review from
Buildkite, this workaround is no longer required.
This reverts commit d3f9cb0ec3.
Change-Id: Ib8c1680eae844cb7b45bf8837acf2af03d4ed344
Reviewed-on: https://cl.tvl.fyi/c/depot/+/909
Reviewed-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
This *should* translate to the required invocation to make sudo allow
nopasswd for users in the wheel group.
Change-Id: I3713862b8df9087cfbaa72d7e824bc43469f7c1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/857
Reviewed-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
This is the point of the machine, afterall.
Change-Id: I15c11600c1c18fa8962d57f75f99a72e1553f9c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/853
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: BuildkiteCI
Tested-by: BuildkiteCI
systemd gets sad otherwise and it is very difficult to console it
Change-Id: Ic6405489532c407273e5634474185f2947420b37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/851
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: BuildkiteCI
Tested-by: BuildkiteCI
it's not glittershark because grfn is the username I have on my laptop
and I want to be able to ssh without an `@`.
Change-Id: Ie1fb6f5e12f3ac52a44680704179bd27a00a7768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/850
Reviewed-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
This adds NixOS configuration for the machine whitby.tvl.fyi.
No interesting services are configured yet, so this configuration is
quite plain.
Change-Id: I67b7c75ebd6e298719b52e6b3bd83cc3be3c45d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/843
Tested-by: BuildkiteCI
Reviewed-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
This does not work for ARGON2 hashes.
Change-Id: I1e070fa0ff17ef21632e94e6777da637deb6f54f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/834
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: BuildkiteCI
Tested-by: BuildkiteCI
This makes it possible to use {ARGON2} hashes instead of the current
salted SHA hashes, which is a much better idea.
Unfortunately the nixpkgs module does not have an option for
overridding the package used, so it is overlaid into the system
package set - this causes widespread rebuilds.
This is fine for us for now, but I have opened a PR upstream to add a
package option: https://github.com/NixOS/nixpkgs/pull/91963
Change-Id: Ib4be931d88e74b91566639f8656742cf096f6cc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/831
Reviewed-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
Before this change, besadii would skip further processing of meta refs (which happen for every CL metadata change), but it would still schedule a build by returning an update - which would then inevitably fail.
This change makes besadii skip meta refs the same way it skips non-depot builds, i.e. completely.
Move *on* from meta refs, do *not* collect $100.
Change-Id: I269d2299f4d3cb1f9c041da8c92fa00ae7794b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/825
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: BuildkiteCI
Tested-by: BuildkiteCI
These are updated for all sorts of things and should just be silently
ignored by besadii.
Change-Id: I0a6de373b21d6bef5fd31d0a1d3f72c501073bba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/801
Reviewed-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
This removes almost all of the GCP-infrastructure leftovers from my
previous setup.
The DNS configuration is retained, but moves to my user folder
instead.
Change-Id: I1867acd379443882f11a3c645846c9902eadd5b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/782
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: isomer <isomer@tvl.fyi>
Besadii already adds 'Verified'-labels, which are used to signal CI
status on CLs, however we don't actually use these labels (yet) which
also means that they are not displayed in the Gerrit UI.
This change temporarily introduces the Code-Review label *in
addition* (with the same values as Verified), providing a build status
signal on the CL but without being required for submission.
Change-Id: I2c3a37c59aceb426815ad4e400c80ab85be482dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/781
Tested-by: BuildkiteCI
Reviewed-by: ericvolp12 <ericvolp12@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
