Commit graph

212 commits

Author SHA1 Message Date
Vincent Ambo
876b71f641 fix(nix/buildkite): Forbid 'prompt' in build phase steps
This would block CI on human-approval if people were allowed to do it,
so they're just not.

Change-Id: I8a9b657d5c91636a7b4de249b977e24fc0941a1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5826
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
56a97a0337 refactor(nix/buildkite): Explicit support for build phases
Previously the extra steps were roughly divided into steps that run
"at build time" (i.e. before we publish results to Gerrit), and
"post-build" (i.e. later on).

In practice, these are something like a build/release pairing, where
steps running after the build results are returned are mostly run for
side-effects (e.g. publishing git subtrees to external repos).

This refactoring makes this distinction explicit in //nix/buildkite
and changes the extraSteps API with an explicit `phases` attribute
instead of the previous `postStep` attribute.

In practice the previous API is still supported, but will throw
evaluation warnings until an arbitrarily chosen cutoff date of
2022-10-01 at which point we will change using it into a hard error.

This uncovered a few strange behaviours which we only accidentally
avoided, most of which I have left TODOs about and will clean up in
subsequent commits.

The purpose of this commit is to allow for separate evaluations of
only build or only release steps, for example if release steps are
evaluated in a slightly different context (e.g. with overridden
versioning that is not relevant to standard CI functionality).

Change-Id: I0b0186e3824273c15a774260708702d4a5974dac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5825
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
a027ee9f03 refactor(nix/buildkite): Rename "post" steps to "release" steps
This is in preparation for a subsequent CL that will do much more
significant changes in //nix/buildkite.

Change-Id: I80a8d67d3a7d593854c8d711572483c2581e7881
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5824
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
b8301ed64b docs(nix/buildkite): Fix an out-of-date comment about chunk size
Change-Id: Ic1f874f4ca83f9088355dc0512723ea962e0db52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5823
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-06-02 20:32:46 +00:00
sterni
3c5da97609 fix(nix/buildLisp): resolve eval problem in ccl code
The isPowerPC predicate has been [removed], since it was misleadingly
named (it just matches PowerPC, 32bit, little endian). This means the
64bit code path could now actually work.

Not sure about endianess, the CCL docs don't really say much regarding
that topic.

[removed]: https://github.com/NixOS/nixpkgs/pull/168113

Change-Id: Icf4a8c6b1df95fa597ed87508f57aaa73e6185ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5796
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-06-01 10:46:03 +00:00
Vincent Ambo
0d4cf119bc feat(nix-1p): Export subtree to GitHub
We needed a derivation for that, but this can also be used in the
Nixery docs building process (which includes nix-1p).

Change-Id: If97cf785a33d703af975da3b41de9b69566dfa81
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5789
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-29 12:30:49 +00:00
Vincent Ambo
9f6215f6e0 subtree(nix/nix-1p): Merge nix-1p into depot
It's time to stop maintaining this on GitHub, and use the
export-from-depot feature instead.

Change-Id: Ic6b840bb0e8580c7214113467b4995ea3d2fae02
2022-05-29 13:31:03 +02:00
sterni
6813598c17 feat(nix/utils): add onlyDrvPath to get the drvPath w/o the outputs
I want to use this utility in a deploy script where the .drv is
nix-copy-closure-d to a remote host and realized there. Consequently it
doesn't make sense that the local deploy script depends on the
derivation's outputs which drvPath does by default.

This also came up when working on //nix/buildkite, although we didn't
end up using it there.

Change-Id: I952bbfd4d7e9de212569d5ee12182eb50d360f53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5767
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-28 12:01:46 +00:00
sterni
a6367b4bdc fix(nix/utils): remove predicates based on symlink heuristic
Due to [nix#6579] the heuristic which allowed us to determine if a
symlink points to a directory is not reliable – if restrict-eval is
enabled it _will_ return wrong results. Until upstream resolves
this (and we backport the patch) it is probably best to not expose this
functionality at all.

[nix#6579]: https://github.com/NixOS/nix/issues/6579

Change-Id: Id847c794bb279be909c5426953c4fe13c2493343
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5761
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-28 11:35:42 +00:00
sterni
0e705fedb0 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: disable CCL once again due to
  The Mysterious Runtime Bug™.

* //users/tazjin/nixos: uninstall dmd which is broken in nixpkgs atm.

Change-Id: I8dd2220af48a7e087584b6f50529fb8477e6a2fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5699
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-27 21:15:10 +00:00
Vincent Ambo
c16a18a718 chore(nix/utils): Comment out tests for detection-ish of symlinks
This is broken for (as of yet unclear reasons) with restricted
evaluation mode.

Change-Id: Idbc16e7e21dfb113995c045659fefe2c1a535741
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5691
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-05-26 16:44:52 +00:00
sterni
5fae8b0826 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: ccl dumped images have fixed themselves… again

* //3p/git: rebase patch on 2.36.0

* //3p/overlays/haskell: remove upstreamed workarounds

* Disable everything depending on cgit temporarily, since it doesn't
  compile with git 2.36 yet.

Change-Id: I9dc11c0846641341adbdcc7162cbf149a15fe0cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5519
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-04 16:18:50 +00:00
Vincent Ambo
0ce396bdab feat(nix/lazy-deps): Add function to generate lazy binary dispatcher
There is a reoccuring problem in readTree-type repositories that use
nix-shell, where evaluation of the full set of dependencies that
should be made available to users takes a noticeable amount of time,
slowing down operations when `direnv` is involved.

In depot, we have so far fixed this by maintaining a manual `//bin`
directory which contains a set of symlinks to a central dispatch
script that can dispatch to various tools in depot lazily.

This script can instead be generated ad-hoc by Nix (pretty fast if we
can make assumptions like `git` and `nix-build` existing on user's
machines already) and added to $PATH.

The function introduced in this commit implements the logic for that.
The structure of the script is based on the existing
`//bin/__dispatch`.

This does not yet switch depot's envrc to use this new method of
installing dependencies lazily.

Change-Id: I92efcd9bb6aa51aa2709ad910a464e9dac97ee89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5512
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-05-02 23:34:55 +00:00
Vincent Ambo
c05c4995ab chore(3p/sources): Bump channels and overlays
Changes:

* updated keycloak configuration for new version
* migrate to emacs28 outside of //users, re-add emacs27 but with a
  warning attached urging people to migrate

Change-Id: I3e5765a63934541f72f6c4a8673d3b4671850c93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5501
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-04-21 16:54:07 +00:00
sterni
11f9306b9a chore(3p/sources): Bump channels & overlays
Use nixos-unstable-small which fixes CVE-2018-25032
(out of bounds write while compressing).

* //users/grfn/xanthous:

  - Supporting random-fu 0.3 requires considerable changes and patching
    random-extras (https://github.com/aristidb/random-extras/pull/5).

    For now we downgrade random-fu and its dependency rvar to 0.2.*,
    forcing us to build xanthous with GHC 8.10.7, due to random-fu 0.2.*
    not supporting that version.

    Nix expressions for the downgraded packages are checked in to avoid
    the potential need to compile Haskell at pipeline eval time.

  - generic-arbitrary exposes a GenericArbitrary newtype now.
    This means we no longer have to implement it in xanthous
    downstream and patch generic-arbitrary to expose the
    GArbitrary type class.

  - Minor adjustments for lens 5.0:

    Xanthous.Game.Memo: clear needs to use ASetter' instead of Lens'

    Xanthous.Data.EntityMap: TraversableWithIndex no longer has an
    itraversed function.

  - Xanthous.Orphans: adjust for aeson's KeyMap, use KM.size explicitly
    instead of relying on MonoTraversable's length

* //nix/buildLisp: the CCL issue has resurfaced, disabling the
  implementation once again.

* //3p/arion: remove, as depot uses the nixpkgs package of it anyways.

* //users/wpcarro: accomodate GHC 9.0.1's stricter parsing of operators.

* //users/tazjin: disable rustfmt as it stopped respecting settings

* //3p/overlays: upgrade home-manager until fix for serivce generation
  has landed upstream

* //users/grfn/system: remove rr override, as the pinned commit is part
  of the 5.5.0 release shipped by nixpkgs.

Change-Id: If229e7317ba48498f85170b57ee9053f6997ff8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5428
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-04-11 19:37:01 +00:00
Vincent Ambo
e70428e75b refactor(ops/pipelines): Configurable GraphQL token location
For external users of the pipeline construction, the token might be in
a different path than `/run/agenix/buildkite-graphql-token`.

It is made configurable through the BUILDKITE_TOKEN_PATH environment
variable. This should be configured on the pipeline level to apply to
all steps.

Change-Id: I23c52e2d705e4134b8b013f8603f92e5533a6e44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5424
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
2022-03-30 11:06:49 +00:00
sterni
508a62b603 chore(3p/sources): Bump channels & overlays
* Remove use of aliases that have been removed in nixpkgs commit
  a36f455905d55838a0d284656e096fbdb857cf3a:

  - ncat
  - x11
  - nologin
  - dbus_libs
  - emacsPackagesGen
  - man-pages
  - pulseaudioLight

Change-Id: Ide603bf48bc7f77e10e4aa715ba025aece3644fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5387
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-03-19 17:11:59 +00:00
Vincent Ambo
b8660386eb fix(nix/buildkite): Do not set 'branches' attribute in step groups
This is no longer accepted by the Buildkite API and causes build
failures.

Functionality is unchanged since we also set the property on the step
itself.

Change-Id: Ib0e0908e4093ca4522711170a7179ce4bacafdc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5324
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-02-24 16:50:32 +00:00
Vincent Ambo
c7bc8aac49 docs(yants): Update josh cloning instructions
Change-Id: I4e9e31488f91465df6b2c6800a3940118107524b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5315
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-19 11:37:00 +00:00
sterni
235eda6649 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: re-enable CCL, as the crash has been fixed upstream,
  although it is unclear what exactly caused / fixed it.

* //ops/whitby: the kitty build broke upstream, so we can't install the
  terminfo on whitby for a bit.

Change-Id: I5710acbe837fbc936e334b2e81f9cf00ed6ae280
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5274
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-13 09:15:09 +00:00
Vincent Ambo
3d8ee62087 style(rust): Format all Rust code with rustfmt
Change-Id: Iab7e00cc26a4f9727d3ab98691ef379921a33052
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5240
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-08 12:06:39 +00:00
Vincent Ambo
a25675804c chore(nix/buildLisp): meta.targets -> meta.ci.targets
Change-Id: Ic8effcc94f3fcfa283436be3e5c0477ef89e9e94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5247
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-07 22:28:41 +00:00
Vincent Ambo
8b8c98380e refactor(readTree): deprecate meta.targets for meta.ci.targets
This means that we use the meta.ci attribute more consistently.

The meta.targets attribute is still read, but prints a big, red
warning telling people to migrate to the new one.

Fixes b/176

Change-Id: Ifb4452f529cfc6bbd5018ad7374cac1c83b10045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5238
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-07 15:13:52 +00:00
Vincent Ambo
3bde425860 refactor(nix): Move mkLabel from buildkite to readTree
This function is more generically useful than just for pipeline
construction.

A subsequent commit will use it inside of readTree itself.

Change-Id: I5eabd6f659726484667e060958865dddbc205762
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5237
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-07 15:13:52 +00:00
Vincent Ambo
b5150fbacf fix(nix/buildkite): read & print the store path of built derivations
This was useful to have in CI, e.g. when targeting a specific NixOS
system. The actual result symlink which is printed is not useful.

Alternative solution would be to change the wrapping of this so that
we conditionally create the symlink for extra steps, but I think it's
not worth the complexity of evaluating the step twice.

Change-Id: Id86eb5114bec935c63a2907ec5f169fc5d41a6cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5227
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-04 19:49:47 +00:00
sterni
0e45761853 chore(3p): Bump NixOS channels and emacs overlay to 2022-02-03
* //nix/buildLisp: This channel bump brought a bizarre regression
  in ccl, causing binaries to crash on thread clean up. This was
  likely caused by a glibc update in nixpkgs. We'll disable emitting CI
  targets for ccl until we can find out and fix what's going on.

Change-Id: I37629f384fa99ec4ef96ce7127fa7569adecb687
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5207
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
2022-02-04 14:39:41 +00:00
sterni
b57854e24d fix(nix/buildLisp): make uiop:argv0 work also for ccl and sbcl
Detection is broken there, too, as UIOP relies on setting the variable
before dumping the image in its portability wrapper dump-image which we
don't use at all.

Change-Id: If7bea5a8522a2e64707b1ee88d62d420bd00a952
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5112
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-02-02 18:21:06 +00:00
Vincent Ambo
bb140e16a2 feat(nix/buildkite): Add parentOverride argument to extra steps
This can be used to override the parent derivation if its output is
required, for example to inject versions which are only used during
releases to avoid cache-busting.

Change-Id: I2211496efa8f9bc98ea43b23e4f3f92c61a6da73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5184
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-02-02 13:36:58 +00:00
Vincent Ambo
ba7197c271 fix(nix/readTree): Use overrideAttrs with derivations
Derivations that support overrideAttrs now have their readTree
markers merged in using it, as passthru attributes.

This makes the significant difference that overriding readTree targets
using `overrideAttrs` keeps their readTree data intact.

Change-Id: Ieef635f048781bf4782c1a28532b89a66d9ca24d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5186
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-02 13:36:58 +00:00
Vincent Ambo
a49358d8aa fix(nix/buildGo): Use overrideAttrs to add metadata
Required for using overrideAttrs in readTree (cl/5186). Since this
uses pkgs.runCommand we know that overrideAttrs is available.

Change-Id: I18fdcc34cc79872834052caf4bf74555fdb766ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5187
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-02 13:36:58 +00:00
Vincent Ambo
aa122cbae7 style: format entire depot with nixpkgs-fmt
This CL can be used to compare the style of nixpkgs-fmt against other
formatters (nixpkgs, alejandra).

Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: cynthia <cynthia@tvl.fyi>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: grfn <grfn@gws.fyi>
2022-01-31 16:11:53 +00:00
Vincent Ambo
d7b2637f59 chore(nix/buildkite): reduce chunk size to 192
We've seen the famous 1 minute timeout on Buildkite again, probably
due to something (keys in targets?) increasing the overall payload
size of our chunks.

This reduces the chunk size by 25%. Lets keep an eye on it with this
value ...

Change-Id: I6bf0e9e4ab0d5b8de22773e6cd5da8d0959cc448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5105
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-29 09:17:07 +00:00
sterni
6908d960b2 feat(3p/overlays/ecl-static): 21.2.1 -> 1c98924
Seems like some issues to do with bytecode compilation have been fixed
at HEAD. closer-mop compiles again and an ironclad failure with the
next quicklisp/channel bump is avoided.

In this change pathname handling in ECL also changed somehow, causing it
to make the :directory part absolute by prefixing it with a slash which
made ld.bfd unhappy while linking an output path that began with a
double slash. This problem can be avoided by constructing the path as
ANSI Common Lisp intended. The truename on the out path is important to
make it recognize that it is indeed a directory.

Change-Id: I5e744022b92502f99ac0b33411a6be443707e200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5076
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-01-28 10:43:01 +00:00
Vincent Ambo
9e28029705 feat(nix/buildkite): Support un-gated post-build extra steps
Allows users to define steps with `postBuild = true` which always run
after 🦆, but do not require human approvals.

This can be useful for things like unconditional release steps.

Change-Id: Idbf6c48a9dedcfc6cc9b7f098423364e2fa72d2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5052
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
4d7dcf10ed refactor(nix/buildkite): Restrict step conditionals to refs only
The previous `condition` abstraction which allowed the full set of
Buildkite conditionals is way too leaky (it lets users to very
Buildkite-specific things which we may not want to allow, and which
are mostly not relevant to a pure evaluation).

Supporting only the `branches` condition (native to Buildkite) should
make it possible to port this to other future CI systems later.

Change-Id: Ib8adcc41db4f1a3566cbeecf13a4228403105c1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5051
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
1a1d706125 feat(nix/buildkite): Add support for gated steps
Adds support for extra build steps that specify a `prompt`. These
steps will be run at the end of the pipeline and will be gated by
human approval.

This mechanism can be used to, for example, stage releases of software
released from depot that are subject to approval.

Change-Id: I97bb505664a2ccf01142286f14e20a370afaa345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5033
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
5a88e47b71 refactor(ops/pipelines): Split build/post steps into separate chunks
This will create `build-chunk-$n.json` files for steps that should run
_before_ duck, and `post-chunk-$n.json` files for steps that should
run after duck.

The post steps are not yet uploaded to Buildkite, but we also don't
have any right now.

Change-Id: I7e1b59cf55a8bf1d97266f6e988aa496959077bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5047
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-22 11:59:08 +00:00
Vincent Ambo
4e82a9dc86 feat(nix/buildkite): Add support for extra steps
This introduces a new feature to our CI system in which targets can
declare extra steps in `meta.ci.extraSteps`.

See the comment in //nix/buildkite/default.nix for an explanation of
how these extra steps are defined.

Change-Id: Icce2890c743286dd37f43024cd390dcebac8cdba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5008
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-21 11:49:03 +00:00
Vincent Ambo
16d72933fc chore(nix/buildkite): Drop --fork-point parameter
The --fork-point parameter is dependent on reflog data which may get
garbage collected. This can lead to flaky behaviour where it returns
no results and fails if `git gc` recently ran (Buildkite will do this
occasionally).

Though the parameter is semantically closer to what we're looking for,
the output is *usually* the same commit since we're not dealing with
more than one thing to compare.

Change-Id: Idc31e7a26fda2b7113edfa162d9d3811b1a01bf6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5032
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-20 19:00:28 +00:00
Vincent Ambo
d14c81007e fix(nix/buildkite): prevent swallowing errors from git-merge-base
The `local` usage we had before would silently swallow non-zero exit
statuses from the substituted git commands.

For some (as of yet unknown) reason, `git merge-base` seems to
sometimes silently fail and produce no output, which broke the rest of
the script logic.

This change will lead to an earlier error, but we don't know if it is
a fix for the actual cause of the git-merge-base problem because the
shape of that problem is unclear.

Change-Id: I4555c8638da450263fa2fd2c274dfdb69f65578e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5012
Tested-by: BuildkiteCI
Reviewed-by: kn <klemens@posteo.de>
2022-01-20 11:04:35 +00:00
Vincent Ambo
0652ac0ace refactor(nix/buildkite): Move fetch-parent-targets script here
This is no longer TVL-specific and should live here with the other
generalised stuff.

Change-Id: I95a1b4c0321f34812162d6fd40568269abf639dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5006
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-19 17:22:36 +00:00
sterni
307eea8e27 fix(nix/buildLisp): use filtered deps for native deps in testSuite
allDeps filters the lisp deps according to the given implementation,
processing any implementation conditional attribute sets. These are not
understood by allNative, so we need to pass it the already filtered
input or evaluation would fail.

Change-Id: I9eb2d0c3b2bf70d759d03490cf31fc585283ce7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5001
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-01-19 13:07:57 +00:00
Vincent Ambo
1c81ccb996 fix(3p): Move away from builtins.fetchGit
Moves to the derivation-based git fetchers everywhere in third-party.
This might help with forward-compatibility with newer Nix versions,
though that's not our primary concern right now.

Change-Id: I565bb72585b8639893e9ea3a9e233338aede63a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3903
Tested-by: BuildkiteCI
Reviewed-by: zseri <zseri.devel@ytrizja.de>
2022-01-17 14:18:37 +00:00
Vincent Ambo
0779f96687 feat(nix/buildkite): Check target map of parent to determine skips
This changes the logic for build pipeline generation to inspect
an (optional) parentTargetMap attribute which contains the derivation
map of a target commit.

Targets that existed in a parent commit with the same drv hash will be
skipped, as they are not considered to have changed.

This does not yet wire up any logic for retrieving the target map from
storage, meaning that at this commit all targets are always built.

The intention is that we will have logic to fetch the target
map (initially from Buildkite artefact storage), which we then pass to
the depot via externalArgs when actually generating the pipeline.

Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 11:49:01 +00:00
Vincent Ambo
0a21da2bb4 feat(ops/pipelines): Create drvmap structure for each commit
Always create a structure that maps all targets to derivations, and
persist it as a JSON file.

This relates to some of the ideas expressed in:

https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit

The file is always uploaded to Buildkite as an artifact. This allows
for retrieving it based on the commit ID in a Buildkite GraphQL query.

By default, Buildkite stores artefacts for 6 months. Storage location
can be overridden (with custom retention) through some environment
variables, but for now at TVL the Buildkite-managed storage is fine.
See also: https://buildkite.com/docs/pipelines/artifacts

In the subsequent filtering implementation, when diffing commits
across a time-range that exceeds artefact retention time, we should
simply default to building everything.

Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 10:26:08 +00:00
Vincent Ambo
b7ef2a579b refactor: Generalise pipeline generation in //nix/buildkite
Extracts the logic for generating our Buildkite pipeline (which has
been copy&pasted and slightly modified in some places outside of
depot) into a generic //nix/buildkite library.

This should cause no change in functionality.

Change-Id: Iad3201713945de41279b39e4f1b847f697c179f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4726
Autosubmit: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-12-28 15:34:39 +00:00
zseri
00adb6e8f3 feat(nix/yants): improve error message for errornous predicate
while trying to yantsify `mkSecrets` in https://cl.tvl.fyi/c/depot/+/4688,
I(zseri) needed to debug a failing evaluation which boiled down
to a result.ok containing something which wasn't boolean,
but the error message didn't indicate where that value came from.

I debugged yants and found that the only place which didn't
simply combine boolean values or use functions which always
return booleans, I managed to isolate the error to the
`pred v` expression. To avoid the necessity to debug yants
to find this, I improve the error message for this case
to mention that
- a restriction predicate is invalid
- what's the name of the failing restriction
- the unexpected predicate return value

Change-Id: I6c570a33ccc5afc445f208e2e8855c49fb37abaf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4698
Tested-by: BuildkiteCI
Reviewed-by: zseri <zseri.devel@ytrizja.de>
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: zseri <zseri.devel@ytrizja.de>
2021-12-27 08:20:58 +00:00
sterni
b9f510d989 fix(nix/buildLisp): don't emit a target for impl used at top level
These targets would be the same derivation, but named differently which
is noisy and causes a few, mostly subtle issues:

* Buildkite struggles with large pipelines a bit, we can save quite a
  few steps by removing these.

* Having two jobs for the same derivation sometimes causes the annoying
  situation that an agent would do nothing except waiting for a lock.

* Non-nix CI we add in the future may not be able to recognize that
  these targets are the same and do extra work unnecessarily.

Change-Id: I1103e719ade1d3859d222b713969ac34a8765cba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4515
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-20 12:46:13 +00:00
sterni
887ac4d3d4 feat(nix/utils): handle toString-able sets produced by builtins.path
When preparing cl/4381 I noticed that we actually handle this case
properly. depot.nix.utils.storePathName depot.path now works as
expected.

Change-Id: Ice9329c67b2e2210852012f5abe82fbbb13193de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4382
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: sterni <sternenseemann@systemli.org>
2021-12-17 10:47:34 +00:00
Griffin Smith
ca56cb2d9f docs(nix/readTree): Tweak grammar of docs for gather
This seemed to be missing a word previously.

Change-Id: Ifa860051d6b692a626dbaddbaee44b761f2274ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4386
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-16 21:14:15 +00:00