This introduces a new feature to our CI system in which targets can
declare extra steps in `meta.ci.extraSteps`.
See the comment in //nix/buildkite/default.nix for an explanation of
how these extra steps are defined.
Change-Id: Icce2890c743286dd37f43024cd390dcebac8cdba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5008
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
The --fork-point parameter is dependent on reflog data which may get
garbage collected. This can lead to flaky behaviour where it returns
no results and fails if `git gc` recently ran (Buildkite will do this
occasionally).
Though the parameter is semantically closer to what we're looking for,
the output is *usually* the same commit since we're not dealing with
more than one thing to compare.
Change-Id: Idc31e7a26fda2b7113edfa162d9d3811b1a01bf6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5032
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
The `local` usage we had before would silently swallow non-zero exit
statuses from the substituted git commands.
For some (as of yet unknown) reason, `git merge-base` seems to
sometimes silently fail and produce no output, which broke the rest of
the script logic.
This change will lead to an earlier error, but we don't know if it is
a fix for the actual cause of the git-merge-base problem because the
shape of that problem is unclear.
Change-Id: I4555c8638da450263fa2fd2c274dfdb69f65578e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5012
Tested-by: BuildkiteCI
Reviewed-by: kn <klemens@posteo.de>
This is no longer TVL-specific and should live here with the other
generalised stuff.
Change-Id: I95a1b4c0321f34812162d6fd40568269abf639dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5006
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
This changes the logic for build pipeline generation to inspect
an (optional) parentTargetMap attribute which contains the derivation
map of a target commit.
Targets that existed in a parent commit with the same drv hash will be
skipped, as they are not considered to have changed.
This does not yet wire up any logic for retrieving the target map from
storage, meaning that at this commit all targets are always built.
The intention is that we will have logic to fetch the target
map (initially from Buildkite artefact storage), which we then pass to
the depot via externalArgs when actually generating the pipeline.
Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Always create a structure that maps all targets to derivations, and
persist it as a JSON file.
This relates to some of the ideas expressed in:
https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit
The file is always uploaded to Buildkite as an artifact. This allows
for retrieving it based on the commit ID in a Buildkite GraphQL query.
By default, Buildkite stores artefacts for 6 months. Storage location
can be overridden (with custom retention) through some environment
variables, but for now at TVL the Buildkite-managed storage is fine.
See also: https://buildkite.com/docs/pipelines/artifacts
In the subsequent filtering implementation, when diffing commits
across a time-range that exceeds artefact retention time, we should
simply default to building everything.
Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Extracts the logic for generating our Buildkite pipeline (which has
been copy&pasted and slightly modified in some places outside of
depot) into a generic //nix/buildkite library.
This should cause no change in functionality.
Change-Id: Iad3201713945de41279b39e4f1b847f697c179f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4726
Autosubmit: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
