Commit graph

51 commits

Author SHA1 Message Date
sterni
8fed8982b7 feat(nix/buildkite): reflect deps between derivations in pipelines
Most of the steps in our buildkite pipeline build derivations without
doing anything else. A lot of those derivations depend on each other.
Consequently, buildkite will schedule builds of derivations whose
dependencies are still in the process of being built. The result is many
buildkite agents doing nothing but blocking on other derivations being
built. We can easily prevent this by using the dependency information we
can get from the derivation (files) of the targets we want to build and
translating them into buildkite step dependencies.

The hard part of this has already been done for a while:
//nix/dependency-analyzer finds the dependencies between a list of
“known” derivations (even if they only depend on each other through
intermediate derivations) without depending on a specific derivation
builder convention, but rather relying on `.drv` files. It still has a
few rough edges, but has been working reliably for our purposes.

Since our steps are identified by derivation hashes, we can just
directly use the available dependency data. Luckily, buildkite seems to
just takes a step as if it was completed if it is skipped, so we don't
even have to check whether dependencies have been skipped or not.

On whitby it seems that the dependency analysis costs about a minute
additionally (which is how long it takes to run
//nix/dependency-analyzer in isolation just about).

Supersedes cl/5063, cl/5060, cl/5064 and cl/5065.

Change-Id: I91d2eb2b43d60811cac0d26fa94467298f622970
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11116
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2024-04-16 16:16:05 +00:00
sterni
dce1a7480c feat(nix/buildkite): use keys based on drvPaths where possible
This will make it easier to emit intra pipeline dependencies based on
the dependencies between derivations contained therein later.

A consequence of this change is that it is no longer possible to have
the same derivation be exposed as multiple steps in the pipeline. I
doubt that having this is very useful, though.

Keys for extraSteps are not changed significantly, but are distinguished
from derivation based steps (with prefix `drv-*`) by their prefix
`extra-step-*`.

Change-Id: I4165900e512b3967fa3ca4cd5bffd44bc15915fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11115
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2024-04-16 16:15:35 +00:00
sterni
c59e11dc3e fix(nix/buildkite): solve bogus realpath errors if build steps fail
- Set pipefail to make sure the correct exit code gets returned.

- Don't let xargs(1) call realpath(1) if the build fails (and nothing is
  printed to stdout).

Change-Id: I526a18df877601d884b83862da5f37336cdf52cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11312
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-03 11:52:10 +00:00
sterni
16a7c4a1be feat(buildkite): avoid building extraSteps in pipeline construction
In principle we don't want to build any (later) pipeline target during
pipeline evaluation insofar they appear in extraSteps. For this reason,
we have the needsOutput mechanism which prevents the parent target of an
extraStep from being built in 🦙.

Unfortunately, this mechanism is not general purpose enough, as we use
other (i.e. non parent) targets from depot in extraSteps. As a
consequence, kind of expensive builds need to happen during pipeline
construction at the moment. The solution is to use the fact that the
command script we want to run is exposed via the readTree interface to
depot and build the script proper only when the extra step is executed.

To facilitate this, some prerequisite changes need to be made:

- We need to use a symlink different to result in case needsOutput is
  true which needs support in mkBuildCommand. We also need to avoid this
  symlink being picked up by git, as many extra steps check whether the
  tree is dirty or not. (Is there a way to have it outside the depot
  tree?)

- Since we rely on the build command printing a single store path we
  store in $command_script, we need to avoid it printing two paths
  in cases where nix-store(1) is used (nix-store(1) prints the symlink
  and readlink(1) would print the store path in a separate line).

Future work would be to remove/deprecate the needsOutput mechanism:
After this change the parent target wouldn't be built right away even if
it appeared in the script via string interpolation. Thus we could,
instead of expecting the target being available as `./result`, make our
extra steps nix-ier.

Change-Id: Idd2e88a865eadabe229ce1e05406e8cc4cb63f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10850
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-03-07 15:39:56 +00:00
sterni
1608f935aa refactor(nix/buildkite): unify target attr path calculation
After the previous refactor that made mkBuildExpr use attribute paths,
mkStep and mkDrvMap can now share the code for calculating the attribute
path of any given target.

Change-Id: I50f9687e85c3b4407e4a55ff7e645db882bb80d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10849
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-03-02 23:39:50 +00:00
sterni
206742949b refactor(nix/buildkite): use attr path over target drv in mkBuild*
mkBuild* is independent of the target abstraction, we can just use
attribute paths. That is useful because we'll want to reuse that code in
order to avoid building derivations (apart from unavoidable IfD) during
pipeline construction for extraSteps.

Breaking Change for mkBuildExpr and mkBuildCommand.

Change-Id: I03646310192087d3e50f358a714472d1ac1a652f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10848
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-02 23:32:38 +00:00
sterni
aa14e36a91 refactor(nix/buildkite): make more use of formal arguments
When changing the buildkite pipeline code I found that some functions
have a lot of arguments where the order is not necessarily clear. To
ease further refactors / new features, I've ported them over to taking
attribute sets.

Note that this technically is a breaking change, as these functions are
all exposed. Not sure how often they'd be called from the outside,
though.

Change-Id: I118c8c5242922403d12f6e5a61beaf68f636b40a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10847
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-02 23:01:05 +00:00
Vincent Ambo
ba9a62ca9f feat(nix/buildkite): fetch drvmap from *any* default branch builds
With this change, the query we make to Buildkite changes from
explicitly querying for a specific number of *ancestor* builds, to
any latest builds of the default branch that have a drvmap.

This is not really supported by Buildkite, it seems, and the query to
do it feels very wonky and requires a lot of fiddling with `jq` to get
the output into the right shape.

We lose the information about which build we downloaded this from in
the output. Adding that information back would make the `jq` query
much more complex.

Change-Id: I9e7cecdffa9ac09f9e0339eb24d98c0e8dd82292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9168
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2023-10-22 12:29:52 +00:00
Florian Klink
8b870c7ec9 feat(nix/buildkite): add meta.ci.buildkiteExtraStepArgs
This allows setting / overwriting arbitrary step arguments, such as setting
[Retry attributes](https://buildkite.com/docs/pipelines/command-step#retry-attributes)
for a specific readTree target.

It's intended to be used for cases where modelling each and every option
in a custom meta.ci attribute would be unfeasible.

Change-Id: I3352d5353b26a41a16760a7df37cd5ffee1665bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8983
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-07-24 20:51:14 +00:00
Florian Klink
f41b611761 feat(nix/buildkite): add meta.ci.buildkiteExtraDeps
This allows setting a buildkiteExtraDeps in `meta.ci`, which will get
added to `depends_on` (which can also be a list).

It allows explicitly describing a dependency on another buildkite step,
either generated by nix/buildkite, or part of the static pipeline.

At some point we might want to expose our key calculation function too,
similar to how readTree exposes mkLabel already, but that's left for
a followup.

Change-Id: I793170401ccd3907ad8bf232b80ca7a492e1c942
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8980
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-24 20:51:14 +00:00
Florian Klink
728de762fd feat(nix/buildkite): support meta.timeout
This uses the nixpkgs convention of meta.timeout, and adds a
timeout_in_minutes field to the pipeline step in case its set.

Fixes https://b.tvl.fyi/issues/285.

Change-Id: Ia72e3832f14bf9172319bce070c5b0944f1c96fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8970
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-07-18 16:47:58 +00:00
Florian Klink
6c630aadc8 feat(nix/buildkite): drop postBuild logic
This has been scheduled for removal in 2022-10-01, but it didn't happen
so far. Let's remove it now, better late than never.

Closes https://b.tvl.fyi/issues/286.

Change-Id: I26fe07360c694c1c93418f3310b4067051e08c87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8969
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-18 16:47:58 +00:00
Florian Klink
4ddfdc2ae0 feat(nix/buildkite): extraSteps: include parentLabel into hash
Contrary to the normal steps, extra steps only show the attribute name
in their label, not the whole attribute path.

To make sure these are still unique, also incorporate the parentLabel
into the string that's hashed.

Change-Id: Ic7a1f76aaac159ab5bd30054e422234882947990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8708
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-03 14:09:28 +00:00
Florian Klink
ed159e3a1d feat(nix/buildkite): add step key to release steps
This uses the same logic ("sha1sum of label") as already used in the
"build" chunks.

Fixes https://b.tvl.fyi/issues/276.

Change-Id: I76e59a141bd05f7d1d997a6471ea65e8cc7b4c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8707
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-03 13:47:22 +00:00
Florian Klink
4cf0ee8ea3 docs(nix/buildkite): escape "release" in warning
When using the buildkite tooling with `postBuild` parameter set, the
following error message is shown:

```
This replaces the previous boolean postBuild API in extra
step definitions. Please remove the postBuild parameter from
this step and instead set phase = release;.
```

However, `release` is no symbol, this should be a string.

Update the message to reflect that.

Change-Id: I341c81d862204db06946512428910cada28481ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8690
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-01 10:34:12 +00:00
William Carroll
1cd7678bdb feat(ops/buildkite): Support "soft failures"
Example Usage:

```nix
{ pkgs, ... }:

(pkgs.writeText "foo" "bar").overrideAttrs (_: {
  meta.ci.extraSteps.bar = {
    label = "testing soft_fail";
    softFail = [
      { exit_status = 253; }
      { exit_status = 251; }
    ];
    command = pkgs.writeShellScript "fail" ''
      # Trigger a soft failure by exiting 253 or 251
      exit 253
    '';
  };
})
```

Buildkite Documentation: https://buildkite.com/changelog/56-command-steps-can-now-be-made-to-soft-fail

Change-Id: Idfce15a8d9cde568aa24ebd03956ada399bba5d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7076
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-01 22:11:05 +00:00
Evgeny Zemtsov
b5b968e103 fix(nix/buildkite) don't fail if gated step has no branches
Change-Id: Id3365c18bc3904b42e8b129a513cff4480a2cb07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7051
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 15:27:31 +00:00
Evgeny Zemtsov
b319e00831 fix(nix/buildkite): split extra steps based on all known phases
Fix a bug introduced by 2ca153141 (I merged the wrong patchset).

Issue happens when pipeline is split by phases into independent
evals (e.g. build/release/deploy).

Splitting extra steps requires knowledge of all known phases,
otherwise pipeline evaluation fails due to extra steps from inactive
phases.

Change-Id: Iab0f2dc3eadda281e483055e26f00a95442e15b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6942
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-11 09:17:25 +00:00
Evgeny Zemtsov
2ca153141d feat(nix/buildkite): allow custom phases
This change automatically extends the list of known phases as soon as
they are added to active phase list.

This is great when a user wants to design pipelines with multiple
groups of dynamic steps.

For example in Resoptima we want to design deployment pipeline where
first only staging k8s namespaces are updated/tested and only after,
we update production.

Change-Id: Iab0f2dc3eadda281e483055e26f00a95442e15b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6923
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-10 15:40:12 +00:00
Evgeny Zemtsov
335bf6900d fix(nix/buildkite): follow parent skip behavior in extra steps
We found a bug after updating to latest tvl-kit which broke
incremental releases.

Bug was related to the fact that extra steps skip attribute had
precedence over parent configuration. This is a desired behavior when
extra step is explicitly set to `skip=true` but otherwise it must
follow parent.

Due to extra step normalization skip parameter is always set to false
if not explicitly set.

Along the way, I'm adding support for setting skip attribute on extra
steps as string so that people can define skip reasons.

The bug was introduced by commit:
b9d79109d feat(ops/buildkite): Allow skip of individual steps

Change-Id: I8a46d0926a749d2434412b309c661b749e9dbf37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6827
Autosubmit: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-30 13:14:54 +00:00
sterni
0c178a0ef6 chore(3p/sources): Bump channels & overlays
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.

* runCommandNoCC -> runCommand
* gmailieer -> lieer
  We also need to work around the fact that home-manager hasn't catched
  on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
  This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy

Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.

Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-09-28 08:02:31 +00:00
Evgeny Zemtsov
8fa3bc7137 feat(buildkite/mkPipeline): allow fast failure for build steps
This is supposed to help with resource wasting in situations when
multiple dynamic steps depend on a failing derivation. Cost of
failure currently is `C = T * D` where:
 - T -> time to compile the failing derivation
 - D -> amount of CI targets depending on a failing derivation

Switching to `Fail Fast` limits it to just T (time of a single
failure). Which helps a lot, especially while upgrading nixpkgs.

Fast fail at this moment is in preview, so to enable it:
 - Enable `Fail Fast` feature at pipeline or organization level
 - Set `cancelOnBuildFailing` parameter to true for `mkPipeline`

Change-Id: I4373a46633522d21e94cfa8bac35243b4eeb0b9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6243
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-01 14:35:03 +00:00
Jean-François Roche
282ad0015b fix(ops/buildkite): do not set branch if empty
To run an extra step for all branches, user don't set the
`branches` attribute. This change avoid setting `branches` to null in such a case.

Change-Id: Iabf2f3d0411b037ece5584f30b29c7e65420b63f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5975
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 14:41:39 +00:00
Jean-François Roche
45c794f1b3 feat(ops/buildkite): Allow agents targeting
Using the `agents` attribute, it lets the user target specific agents to run a step.

Change-Id: Id6fc0981d4879b77598854e1f296dffffb33a405
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5974
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 09:29:58 +00:00
Jean-François Roche
b9d79109dd feat(ops/buildkite): Allow skip of individual steps
Using the `skip` attribute, it lets the user decide to temporarily skip a step.

Change-Id: If63ce60a2b4148f041655912730acc4ae9e28534
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5971
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 09:29:58 +00:00
Vincent Ambo
b9c31d6658 refactor(nix/buildkite): Optimise steps if builds are not enabled
Generating a release-only pipeline skips a bigger chunk of eval this
way (the step itself is never actually evaluated, which means we never
actually compute the drv), which can be quite beneficial in terms of
evaluation time.

Change-Id: I2739026ddd1c6a86f82627ac26a046c5fe7359ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5830
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-06-06 12:02:26 +00:00
Vincent Ambo
df83e3d556 fix(nix/buildkite): Disable depends_on if build phase is not run
Extra steps that use `depends_on` (e.g. if they need output from their
parent) should not actually depend on their parents build step if the
build phase is not active.

This is required to actually decouple the phases.

Change-Id: I398da9a8a53e97ca3c635342259fc722d54b8e4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5829
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-06-06 12:02:26 +00:00
Vincent Ambo
302f7e6623 feat(nix/buildkite): Allow toggling of individual phases
Using the `activePhases` attribute, the set of phases included in an
evaluation can be modified.

This lets users generate e.g. ONLY the release steps of a pipeline.

Change-Id: Ib0c38826dd69666094d619f5f324d1baafce8134
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5828
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-06-06 12:02:26 +00:00
Vincent Ambo
cf00365f84 feat(nix/buildkite): Validate available phases in extra steps
This will avoid things like extra steps being accidentally ignored
because of typos.

Change-Id: Ic4fa5925e42a7a449f89b4cde1510e216e91da6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5827
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
876b71f641 fix(nix/buildkite): Forbid 'prompt' in build phase steps
This would block CI on human-approval if people were allowed to do it,
so they're just not.

Change-Id: I8a9b657d5c91636a7b4de249b977e24fc0941a1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5826
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
56a97a0337 refactor(nix/buildkite): Explicit support for build phases
Previously the extra steps were roughly divided into steps that run
"at build time" (i.e. before we publish results to Gerrit), and
"post-build" (i.e. later on).

In practice, these are something like a build/release pairing, where
steps running after the build results are returned are mostly run for
side-effects (e.g. publishing git subtrees to external repos).

This refactoring makes this distinction explicit in //nix/buildkite
and changes the extraSteps API with an explicit `phases` attribute
instead of the previous `postStep` attribute.

In practice the previous API is still supported, but will throw
evaluation warnings until an arbitrarily chosen cutoff date of
2022-10-01 at which point we will change using it into a hard error.

This uncovered a few strange behaviours which we only accidentally
avoided, most of which I have left TODOs about and will clean up in
subsequent commits.

The purpose of this commit is to allow for separate evaluations of
only build or only release steps, for example if release steps are
evaluated in a slightly different context (e.g. with overridden
versioning that is not relevant to standard CI functionality).

Change-Id: I0b0186e3824273c15a774260708702d4a5974dac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5825
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
a027ee9f03 refactor(nix/buildkite): Rename "post" steps to "release" steps
This is in preparation for a subsequent CL that will do much more
significant changes in //nix/buildkite.

Change-Id: I80a8d67d3a7d593854c8d711572483c2581e7881
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5824
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
b8301ed64b docs(nix/buildkite): Fix an out-of-date comment about chunk size
Change-Id: Ic1f874f4ca83f9088355dc0512723ea962e0db52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5823
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-06-02 20:32:46 +00:00
Vincent Ambo
e70428e75b refactor(ops/pipelines): Configurable GraphQL token location
For external users of the pipeline construction, the token might be in
a different path than `/run/agenix/buildkite-graphql-token`.

It is made configurable through the BUILDKITE_TOKEN_PATH environment
variable. This should be configured on the pipeline level to apply to
all steps.

Change-Id: I23c52e2d705e4134b8b013f8603f92e5533a6e44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5424
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
2022-03-30 11:06:49 +00:00
Vincent Ambo
b8660386eb fix(nix/buildkite): Do not set 'branches' attribute in step groups
This is no longer accepted by the Buildkite API and causes build
failures.

Functionality is unchanged since we also set the property on the step
itself.

Change-Id: Ib0e0908e4093ca4522711170a7179ce4bacafdc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5324
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-02-24 16:50:32 +00:00
Vincent Ambo
3bde425860 refactor(nix): Move mkLabel from buildkite to readTree
This function is more generically useful than just for pipeline
construction.

A subsequent commit will use it inside of readTree itself.

Change-Id: I5eabd6f659726484667e060958865dddbc205762
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5237
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-07 15:13:52 +00:00
Vincent Ambo
b5150fbacf fix(nix/buildkite): read & print the store path of built derivations
This was useful to have in CI, e.g. when targeting a specific NixOS
system. The actual result symlink which is printed is not useful.

Alternative solution would be to change the wrapping of this so that
we conditionally create the symlink for extra steps, but I think it's
not worth the complexity of evaluating the step twice.

Change-Id: Id86eb5114bec935c63a2907ec5f169fc5d41a6cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5227
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-04 19:49:47 +00:00
Vincent Ambo
bb140e16a2 feat(nix/buildkite): Add parentOverride argument to extra steps
This can be used to override the parent derivation if its output is
required, for example to inject versions which are only used during
releases to avoid cache-busting.

Change-Id: I2211496efa8f9bc98ea43b23e4f3f92c61a6da73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5184
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-02-02 13:36:58 +00:00
Vincent Ambo
aa122cbae7 style: format entire depot with nixpkgs-fmt
This CL can be used to compare the style of nixpkgs-fmt against other
formatters (nixpkgs, alejandra).

Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: cynthia <cynthia@tvl.fyi>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: grfn <grfn@gws.fyi>
2022-01-31 16:11:53 +00:00
Vincent Ambo
d7b2637f59 chore(nix/buildkite): reduce chunk size to 192
We've seen the famous 1 minute timeout on Buildkite again, probably
due to something (keys in targets?) increasing the overall payload
size of our chunks.

This reduces the chunk size by 25%. Lets keep an eye on it with this
value ...

Change-Id: I6bf0e9e4ab0d5b8de22773e6cd5da8d0959cc448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5105
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-29 09:17:07 +00:00
Vincent Ambo
9e28029705 feat(nix/buildkite): Support un-gated post-build extra steps
Allows users to define steps with `postBuild = true` which always run
after 🦆, but do not require human approvals.

This can be useful for things like unconditional release steps.

Change-Id: Idbf6c48a9dedcfc6cc9b7f098423364e2fa72d2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5052
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
4d7dcf10ed refactor(nix/buildkite): Restrict step conditionals to refs only
The previous `condition` abstraction which allowed the full set of
Buildkite conditionals is way too leaky (it lets users to very
Buildkite-specific things which we may not want to allow, and which
are mostly not relevant to a pure evaluation).

Supporting only the `branches` condition (native to Buildkite) should
make it possible to port this to other future CI systems later.

Change-Id: Ib8adcc41db4f1a3566cbeecf13a4228403105c1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5051
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
1a1d706125 feat(nix/buildkite): Add support for gated steps
Adds support for extra build steps that specify a `prompt`. These
steps will be run at the end of the pipeline and will be gated by
human approval.

This mechanism can be used to, for example, stage releases of software
released from depot that are subject to approval.

Change-Id: I97bb505664a2ccf01142286f14e20a370afaa345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5033
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
5a88e47b71 refactor(ops/pipelines): Split build/post steps into separate chunks
This will create `build-chunk-$n.json` files for steps that should run
_before_ duck, and `post-chunk-$n.json` files for steps that should
run after duck.

The post steps are not yet uploaded to Buildkite, but we also don't
have any right now.

Change-Id: I7e1b59cf55a8bf1d97266f6e988aa496959077bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5047
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-22 11:59:08 +00:00
Vincent Ambo
4e82a9dc86 feat(nix/buildkite): Add support for extra steps
This introduces a new feature to our CI system in which targets can
declare extra steps in `meta.ci.extraSteps`.

See the comment in //nix/buildkite/default.nix for an explanation of
how these extra steps are defined.

Change-Id: Icce2890c743286dd37f43024cd390dcebac8cdba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5008
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-21 11:49:03 +00:00
Vincent Ambo
16d72933fc chore(nix/buildkite): Drop --fork-point parameter
The --fork-point parameter is dependent on reflog data which may get
garbage collected. This can lead to flaky behaviour where it returns
no results and fails if `git gc` recently ran (Buildkite will do this
occasionally).

Though the parameter is semantically closer to what we're looking for,
the output is *usually* the same commit since we're not dealing with
more than one thing to compare.

Change-Id: Idc31e7a26fda2b7113edfa162d9d3811b1a01bf6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5032
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-20 19:00:28 +00:00
Vincent Ambo
d14c81007e fix(nix/buildkite): prevent swallowing errors from git-merge-base
The `local` usage we had before would silently swallow non-zero exit
statuses from the substituted git commands.

For some (as of yet unknown) reason, `git merge-base` seems to
sometimes silently fail and produce no output, which broke the rest of
the script logic.

This change will lead to an earlier error, but we don't know if it is
a fix for the actual cause of the git-merge-base problem because the
shape of that problem is unclear.

Change-Id: I4555c8638da450263fa2fd2c274dfdb69f65578e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5012
Tested-by: BuildkiteCI
Reviewed-by: kn <klemens@posteo.de>
2022-01-20 11:04:35 +00:00
Vincent Ambo
0652ac0ace refactor(nix/buildkite): Move fetch-parent-targets script here
This is no longer TVL-specific and should live here with the other
generalised stuff.

Change-Id: I95a1b4c0321f34812162d6fd40568269abf639dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5006
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-19 17:22:36 +00:00
Vincent Ambo
0779f96687 feat(nix/buildkite): Check target map of parent to determine skips
This changes the logic for build pipeline generation to inspect
an (optional) parentTargetMap attribute which contains the derivation
map of a target commit.

Targets that existed in a parent commit with the same drv hash will be
skipped, as they are not considered to have changed.

This does not yet wire up any logic for retrieving the target map from
storage, meaning that at this commit all targets are always built.

The intention is that we will have logic to fetch the target
map (initially from Buildkite artefact storage), which we then pass to
the depot via externalArgs when actually generating the pipeline.

Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 11:49:01 +00:00
Vincent Ambo
0a21da2bb4 feat(ops/pipelines): Create drvmap structure for each commit
Always create a structure that maps all targets to derivations, and
persist it as a JSON file.

This relates to some of the ideas expressed in:

https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit

The file is always uploaded to Buildkite as an artifact. This allows
for retrieving it based on the commit ID in a Buildkite GraphQL query.

By default, Buildkite stores artefacts for 6 months. Storage location
can be overridden (with custom retention) through some environment
variables, but for now at TVL the Buildkite-managed storage is fine.
See also: https://buildkite.com/docs/pipelines/artifacts

In the subsequent filtering implementation, when diffing commits
across a time-range that exceeds artefact retention time, we should
simply default to building everything.

Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 10:26:08 +00:00