This is supposed to help with resource wasting in situations when
multiple dynamic steps depend on a failing derivation. Cost of
failure currently is `C = T * D` where:
- T -> time to compile the failing derivation
- D -> amount of CI targets depending on a failing derivation
Switching to `Fail Fast` limits it to just T (time of a single
failure). Which helps a lot, especially while upgrading nixpkgs.
Fast fail at this moment is in preview, so to enable it:
- Enable `Fail Fast` feature at pipeline or organization level
- Set `cancelOnBuildFailing` parameter to true for `mkPipeline`
Change-Id: I4373a46633522d21e94cfa8bac35243b4eeb0b9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6243
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
To run an extra step for all branches, user don't set the
`branches` attribute. This change avoid setting `branches` to null in such a case.
Change-Id: Iabf2f3d0411b037ece5584f30b29c7e65420b63f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5975
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Using the `agents` attribute, it lets the user target specific agents to run a step.
Change-Id: Id6fc0981d4879b77598854e1f296dffffb33a405
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5974
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Using the `skip` attribute, it lets the user decide to temporarily skip a step.
Change-Id: If63ce60a2b4148f041655912730acc4ae9e28534
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5971
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Generating a release-only pipeline skips a bigger chunk of eval this
way (the step itself is never actually evaluated, which means we never
actually compute the drv), which can be quite beneficial in terms of
evaluation time.
Change-Id: I2739026ddd1c6a86f82627ac26a046c5fe7359ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5830
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Extra steps that use `depends_on` (e.g. if they need output from their
parent) should not actually depend on their parents build step if the
build phase is not active.
This is required to actually decouple the phases.
Change-Id: I398da9a8a53e97ca3c635342259fc722d54b8e4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5829
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Using the `activePhases` attribute, the set of phases included in an
evaluation can be modified.
This lets users generate e.g. ONLY the release steps of a pipeline.
Change-Id: Ib0c38826dd69666094d619f5f324d1baafce8134
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5828
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
This will avoid things like extra steps being accidentally ignored
because of typos.
Change-Id: Ic4fa5925e42a7a449f89b4cde1510e216e91da6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5827
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
This would block CI on human-approval if people were allowed to do it,
so they're just not.
Change-Id: I8a9b657d5c91636a7b4de249b977e24fc0941a1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5826
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Previously the extra steps were roughly divided into steps that run
"at build time" (i.e. before we publish results to Gerrit), and
"post-build" (i.e. later on).
In practice, these are something like a build/release pairing, where
steps running after the build results are returned are mostly run for
side-effects (e.g. publishing git subtrees to external repos).
This refactoring makes this distinction explicit in //nix/buildkite
and changes the extraSteps API with an explicit `phases` attribute
instead of the previous `postStep` attribute.
In practice the previous API is still supported, but will throw
evaluation warnings until an arbitrarily chosen cutoff date of
2022-10-01 at which point we will change using it into a hard error.
This uncovered a few strange behaviours which we only accidentally
avoided, most of which I have left TODOs about and will clean up in
subsequent commits.
The purpose of this commit is to allow for separate evaluations of
only build or only release steps, for example if release steps are
evaluated in a slightly different context (e.g. with overridden
versioning that is not relevant to standard CI functionality).
Change-Id: I0b0186e3824273c15a774260708702d4a5974dac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5825
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
This is in preparation for a subsequent CL that will do much more
significant changes in //nix/buildkite.
Change-Id: I80a8d67d3a7d593854c8d711572483c2581e7881
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5824
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
For external users of the pipeline construction, the token might be in
a different path than `/run/agenix/buildkite-graphql-token`.
It is made configurable through the BUILDKITE_TOKEN_PATH environment
variable. This should be configured on the pipeline level to apply to
all steps.
Change-Id: I23c52e2d705e4134b8b013f8603f92e5533a6e44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5424
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
This is no longer accepted by the Buildkite API and causes build
failures.
Functionality is unchanged since we also set the property on the step
itself.
Change-Id: Ib0e0908e4093ca4522711170a7179ce4bacafdc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5324
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
This function is more generically useful than just for pipeline
construction.
A subsequent commit will use it inside of readTree itself.
Change-Id: I5eabd6f659726484667e060958865dddbc205762
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5237
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This was useful to have in CI, e.g. when targeting a specific NixOS
system. The actual result symlink which is printed is not useful.
Alternative solution would be to change the wrapping of this so that
we conditionally create the symlink for extra steps, but I think it's
not worth the complexity of evaluating the step twice.
Change-Id: Id86eb5114bec935c63a2907ec5f169fc5d41a6cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5227
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
This can be used to override the parent derivation if its output is
required, for example to inject versions which are only used during
releases to avoid cache-busting.
Change-Id: I2211496efa8f9bc98ea43b23e4f3f92c61a6da73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5184
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
We've seen the famous 1 minute timeout on Buildkite again, probably
due to something (keys in targets?) increasing the overall payload
size of our chunks.
This reduces the chunk size by 25%. Lets keep an eye on it with this
value ...
Change-Id: I6bf0e9e4ab0d5b8de22773e6cd5da8d0959cc448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5105
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Allows users to define steps with `postBuild = true` which always run
after 🦆, but do not require human approvals.
This can be useful for things like unconditional release steps.
Change-Id: Idbf6c48a9dedcfc6cc9b7f098423364e2fa72d2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5052
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
The previous `condition` abstraction which allowed the full set of
Buildkite conditionals is way too leaky (it lets users to very
Buildkite-specific things which we may not want to allow, and which
are mostly not relevant to a pure evaluation).
Supporting only the `branches` condition (native to Buildkite) should
make it possible to port this to other future CI systems later.
Change-Id: Ib8adcc41db4f1a3566cbeecf13a4228403105c1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5051
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
Adds support for extra build steps that specify a `prompt`. These
steps will be run at the end of the pipeline and will be gated by
human approval.
This mechanism can be used to, for example, stage releases of software
released from depot that are subject to approval.
Change-Id: I97bb505664a2ccf01142286f14e20a370afaa345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5033
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
This will create `build-chunk-$n.json` files for steps that should run
_before_ duck, and `post-chunk-$n.json` files for steps that should
run after duck.
The post steps are not yet uploaded to Buildkite, but we also don't
have any right now.
Change-Id: I7e1b59cf55a8bf1d97266f6e988aa496959077bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5047
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
This introduces a new feature to our CI system in which targets can
declare extra steps in `meta.ci.extraSteps`.
See the comment in //nix/buildkite/default.nix for an explanation of
how these extra steps are defined.
Change-Id: Icce2890c743286dd37f43024cd390dcebac8cdba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5008
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
The --fork-point parameter is dependent on reflog data which may get
garbage collected. This can lead to flaky behaviour where it returns
no results and fails if `git gc` recently ran (Buildkite will do this
occasionally).
Though the parameter is semantically closer to what we're looking for,
the output is *usually* the same commit since we're not dealing with
more than one thing to compare.
Change-Id: Idc31e7a26fda2b7113edfa162d9d3811b1a01bf6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5032
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
The `local` usage we had before would silently swallow non-zero exit
statuses from the substituted git commands.
For some (as of yet unknown) reason, `git merge-base` seems to
sometimes silently fail and produce no output, which broke the rest of
the script logic.
This change will lead to an earlier error, but we don't know if it is
a fix for the actual cause of the git-merge-base problem because the
shape of that problem is unclear.
Change-Id: I4555c8638da450263fa2fd2c274dfdb69f65578e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5012
Tested-by: BuildkiteCI
Reviewed-by: kn <klemens@posteo.de>
This is no longer TVL-specific and should live here with the other
generalised stuff.
Change-Id: I95a1b4c0321f34812162d6fd40568269abf639dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5006
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
This changes the logic for build pipeline generation to inspect
an (optional) parentTargetMap attribute which contains the derivation
map of a target commit.
Targets that existed in a parent commit with the same drv hash will be
skipped, as they are not considered to have changed.
This does not yet wire up any logic for retrieving the target map from
storage, meaning that at this commit all targets are always built.
The intention is that we will have logic to fetch the target
map (initially from Buildkite artefact storage), which we then pass to
the depot via externalArgs when actually generating the pipeline.
Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Always create a structure that maps all targets to derivations, and
persist it as a JSON file.
This relates to some of the ideas expressed in:
https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit
The file is always uploaded to Buildkite as an artifact. This allows
for retrieving it based on the commit ID in a Buildkite GraphQL query.
By default, Buildkite stores artefacts for 6 months. Storage location
can be overridden (with custom retention) through some environment
variables, but for now at TVL the Buildkite-managed storage is fine.
See also: https://buildkite.com/docs/pipelines/artifacts
In the subsequent filtering implementation, when diffing commits
across a time-range that exceeds artefact retention time, we should
simply default to building everything.
Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Extracts the logic for generating our Buildkite pipeline (which has
been copy&pasted and slightly modified in some places outside of
depot) into a generic //nix/buildkite library.
This should cause no change in functionality.
Change-Id: Iad3201713945de41279b39e4f1b847f697c179f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4726
Autosubmit: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
