The main bottleneck of our builds right now is Nix evaluation, which
means that most of the time is spent idling during builds.
Since we're evaluating in parallel, lets give it a few more builders.
I don't want to go all the way to 64 immediately to first see if we
get any adverse effects from highly concurrent builds running
concurrently (if we do we could group them into different "concurrency
groups" in Buildkite).
Change-Id: Ibc3f89fb59cb4ee471b152ff36887ffe2b39f8f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1932
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This has somehow stopped being a thing in newer vterm versions,
causing the weird behaviour with my term switcher - buffers with the
correct name were sticking around, but no longer in the right mode.
Change-Id: Ie641eb3db91808d7d1016de1e8ef3ad271c8995e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1931
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Since upgrading to Emacs 27 I have observed a strange behaviour where
this terminal switcher sometimes fails to select a valid buffer, in
which case it falls through to the case that just opens a new buffer
instead.
This is kind of annoying and to aid in debugging this change makes the
creation of new buffers explicit and fails if no matching buffer is
found.
Note that this is likely not a fix for the issue itself, but it will
help debug what is going on.
Change-Id: I906869aba7d25156aaf92c090b169ce02785b85e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1930
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Auto-generate org agenda commands from project tags of the form
`project__foo_bar`, prefixed with `p` and named based on the first
letters of the words in the project. there is (obviously) some overlap
here but that can be fixed by just adding extra underscores to things to
disambiguate.
Change-Id: If07b15a21d8bcb6df6245e8c6e4731041930ecc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1926
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
These were not used anyway, and *may* have been causing a weird series
of problems leading to things sometimes aborting. Or maybe it has
nothing to do with it - the only correlation we've found is from
disassembling the output of a core dump. Regardless, this feels
stylistically like a good idea.
Change-Id: I288b50945e74ea90c118a21767fa4387c9f47f7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1921
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Rather than printing "Nix" in the version string, print "Tvix". This is
cosmetically nice, but also is very useful when making sure I actually
have installed tvix on a system successfully.
Change-Id: Idd1a9954bd66509327f52b7457cfba2f090ab30c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1924
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
This allows distinguishing between an empty value and no value.
Patch ported from upstream at
ba87b08f85
Change-Id: I061cc8e16b1a7a0341adfc3b0edca1c0c51d5c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1884
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
When the nix daemon starts up, first check (using sd_listen_fds) whether
we have been systemd socket-activated. If so, instead of passing the nix
daemon socket path to grpc, start a manual accept(2) loop, passing the
client file descriptors to grpc via AddInsecureChannelFromFd. There's an
open grpc issue at https://github.com/grpc/grpc/issues/19133 for
building support into grpc to do this automatically, but as of right now
this appears to be the only way to make this happen.
Making this happen, by the way, was a bit of a journey - at one point I
attempted to ServerBuilder's experimental AddExternalConnectionAcceptor
API, and that didn't work either - it appears that the final missing
piece to getting this working was explicitly fcntl(2)ing the client file
descriptors to set O_NONBLOCK before passing them into gRPC. With that
set, this all works inside of the test vm.
Fixes: b/56
Change-Id: I5d2ab2b5b02eb570249b30a9674e115c61b0ab0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1882
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Expose depot to the test vm via NIX_PATH, so we can test the build of
depot packages from inside the test vm.
Change-Id: I26314634895ab1339f5715c0b0b261028eefdaa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1923
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Allow 16 things to happen in parallel, which is useful now that the
CI granularity is on a per-target level.
Change-Id: Ie65dd119ea0666618fbb249613e70a68276db834
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1902
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
We have naturally evolved a distinction between logical and physical
targets.
Physical targets are those which correspond directly to a tree
location on disk and can be built with `-A path.to.files`, while
logical targets are those that are exported from within an expression
but do not have a corresponding file on disk.
This change adds support for exporting logical targets from any tree
location by adding a `meta.targets` attribute containing keys into
itself, which will be consumed by the CI target gathering logic and
included in the generated pipeline.
Note that the labels for subtargets are syntactically different to
emphasise that they do not correspond to a file location. For example,
this change enables 'ops.nixos.whitbySystem' as a subtarget, which is
labeled in CI as `ops/nixos:whitbySystem`.
Change-Id: Ied09647a62c2ba98e3914548e3742ad422c63ecf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1893
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Create the pipeline by outputting a file that contains nix-build
invocations for each target's *derivation path*.
Each invocation has a generated Nix expression passed to it with `-E`
which fetches the correct target from the tree while correctly
handling targets with strange characters (such as in Go-packages).
This makes it possible to run target-level granular pipelines. We're
getting somewhere!
Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Wrap all ldap access in a macro that automatically reconnects and
retries operations that fail due to a connection error, to handle the
case where the ldap server restarts while we still have an open
connection.
Fixes: #44
Change-Id: I4859cf509106e480f97fed17e7f08e0eea909352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1871
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
These are unused, and I'm not sure they ever actually worked.
Change-Id: I6bb6b4257b7815932581ded7929437dba80961c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1885
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Auto-link b/\d+ in gerrit comments + patch comments to panettone issues.
This is a departure from the previous syntax of #\d+ that we were using
historically, but has been requested by multiple people
Fixes: b/40
Change-Id: I8e556eb6b2f55ae80d945b86b51041751ecb053b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1888
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
The various bits (including the unix socket that we listen on) append
/nix to this - so it isn't expected to be in there.
Change-Id: I0d95f2cc5641f610c1bdb03b036fac61dac9842d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1890
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
The nix package is expected to install /etc/profile.d/nix.sh and
/etc/profile.d/nix-daemon.sh into its output - these set the environment
variables expected by the client-side nix user commands, eg NIX_DAEMON.
Fixes: b/52
Change-Id: I5fb964c076f092d0ef7e1d49aca37875c866fb90
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1889
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
The absence of the navbar containing the "all issues" and "log out"
links from the top of the page has been a common complaint - initially I
disagreed, but after some time thinking about it I've come around. This
adds the same nav - with the "All Issues" link and the "Log Out" link -
to the top of every page, and also fixes a bug where query params would
prevent the "All Issues" link from being hidden on the "All Issues"
page, which looked especially weird when they were right next to each other.
Change-Id: I1d07175fa07aee057ddd140a6864d01342fbb7ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1868
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Rather than sending the entire AddTextToStore request along in a single
message, send it in a stream of chunks using the same metadata-first
approach we've been using for the other store gRPC requests. This fixes
a bug where certain builds could send more data than the maximum gRPC
request size (4194304 bytes, it would appear), resulting in a
RESOURCE_EXHAUSTED error.
The initial chunk size, which is currently constant but should be made
dynamic at some point in the future, has been chosen based on the IPC
bandwidth delay product for tazjin's desktop, rounded up.
Change-Id: I6f0232cdbc98653484816b39855126873fc59a03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1835
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
These are the scripts I use to test the nix daemon interaction with a
non-/nix store directory during development, copied almost verbatim from
my cmake build directory. As such, there's likely a *lot* of cleanup and
deduplication to be done here, but I'm committing these as is in the
hope that others can benefit from them somehow.
Change-Id: I42a10a85e6731fa2014c7ea9738224d678a8376b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1881
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Style blockquotes (which show up in rendered markdown) similarly to how
github does, by rendering a 5px-wide margin to the left with some
padding.
Fixes: #48
Change-Id: I79aa3b6cda5d928885c2cc36f504009232252c17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1869
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
- Currently, asking for all your quotes either stalls the bot, or
doesn't result in you getting all your quotes, or both. This aims
to resolve this oversight by shoving them all in a pastebin.
- This uses the lovely `crimp` library by tazjin, which is really
good at just doing HTTP stuff with minimal fuss. Amazing!
(although we should probably actually use the depot version)
- Everything is hard coded for now, but we probably don't care.
- Stuff expires after 24 hours, for privacy reasons?
- We also had to add a function to format entries without colours,
and took the opportunity to clean up the format!() a bit.
Change-Id: I6e75968c7da48a51fff327355b8fa2c025d0db75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1872
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
The default was really annoyingly short - 90 days feels perfectly fine
for what we want, though we may want to increase even further.
Fixes: #19
Change-Id: I917abd95c4925f8491cd2be7cd87d91bb6621153
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1867
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Load a SESSION_SECRET env var and set it as the hunchentoot session
secret if present, so that restarting panettone doesn't destroy all
sessions due to the secret getting regenerated.
Refs: #19
Change-Id: Ia2c633fa998e128ecece66e824df01c430da8235
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1866
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
This reverts commit 475d41f698. I'd like
my derivations back, please.
Changes necessary to get this working:
- Don't depend on `nixpkgs` being in the NIX_PATH for my website - it's
not necessary anyway since emacs 27 is mainline now
- .skip-subtrees on things that shouldn't be evaluated anyway
- Get rid of system/pkgs, and move the one thing in there that *wasn't*
already in third_party (alsi) to third_party
- Drop notifymuch for now - it's not working, and I'll probably get it
landed in nixpkgs before I manage to get it working
- Add __readTree = true to my systems so they get built.
- explicitly disable ci for xanthous, which is failing to build and had
been omitted previously
Change-Id: I20f5e81d6eb7ffe040091a08d75d0cb15304f707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1864
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
The command line options --arg and --argstr that are used by a bunch of
CLI commands to pass arguments to top-level functions in files go
through the same code-path as auto-calling top-level functions with
their default arguments - this, however, was only passing the arguments
that were *explicitly* mentioned in the formals of the function - in the
case of an as-pattern with an ellipsis (eg args @ { ... }) extra passed
arguments would get omitted. This fixes that to instead pass *all*
specified auto args in the case that our function has an ellipsis.
Submitted upstream at https://github.com/NixOS/nix/pull/3965Fixes: #46
Change-Id: I32b7ee0e5bacf75b2bc43a3f0796f533f4bd5959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1863
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Previously, MixEvalArgs (a generic data type used to handle --arg,
--argstr, and -I arguments to `nix-build`, `nix eval`, etc.) was storing
the difference between --arg and --argstr by prepending a single
character (either 'E' or 'S') to the value of the arg. This is messy and
un-type-safe, so this commit refactors that to use a proper enum and a
std::pair, which allows us to add a switch and get totality checking.
yay, types!
Change-Id: I883f8db3136d05dda190fac0a1b494386c2ff87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1862
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Automatically walk the entire depot tree and pick out things that are
"buildable", then include them in the attribute `ci.targets` (which is
now also the target for CI builds).
A long time ago, in a land far away, we (well, I, at the time) had a
prototype of this which ran into constant issues with infinite
recursions while trying to walk the tree. In fact, this is why
readTree originally gained the `__readTree`-attribute which marks
things that were imported automatically.
Based on some code edef whipped up earlier (with the breakthrough
being that we also add the attribute to top-level folders, which
suddenly resolves a whole bunch of problems), I've now implemented
this actually working version.
At the moment all builds still happen as one big bag of builds, but at
some point we will granularise this.
Change-Id: I86f12ce7f63dae98e7e5c6646a4e9d220de783f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1854
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
This was used previously when build granularity was besadii's task,
which it no longer is.
Change-Id: I6df2db1ed4730a7953199b7b48aa9ad916418b22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1853
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This is a temporary state (TODO added) to be picked up by the new CI
logic.
Change-Id: Id4702740ffd18325088e2a8a0c6157a8cee7ccf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
These projects, which are not currently included in CI runs, don't
build at the moment.
Upcoming logic changes would mean that we would start including them
in CI, which is undesirable until they're fixed - but I'm not going to
be doing that now.
Change-Id: I7c337e098be8bff00db6d99fc7236a695f5a85f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1850
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This file is not a readTree-compatible Nix file, but rather a NixOS
module. At some point it should be moved elsewhere and .skip-subtree'd
to avoid this issue.
Change-Id: If1b3f7cc80084af1f44036b8b9272f7b76438c2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1849
Tested-by: BuildkiteCI
Reviewed-by: multi <depot@in-addr.xyz>
This is required to automatically walk the tree (see subsequent
commits).
Note: Lisp packages are removed from the CI builds in this commit
because the attrValues of third_party.lisp will contain an element
that is simply `true`, which causes a type error.
These packages are re-added when CI refactoring is complete.
Change-Id: I21e2b719e6c7161c23d2867a216f4daa1c6c8394
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1848
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
