0a21da2bb4
Always create a structure that maps all targets to derivations, and persist it as a JSON file. This relates to some of the ideas expressed in: https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit The file is always uploaded to Buildkite as an artifact. This allows for retrieving it based on the commit ID in a Buildkite GraphQL query. By default, Buildkite stores artefacts for 6 months. Storage location can be overridden (with custom retention) through some environment variables, but for now at TVL the Buildkite-managed storage is fine. See also: https://buildkite.com/docs/pipelines/artifacts In the subsequent filtering implementation, when diffing commits across a time-range that exceeds artefact retention time, we should simply default to building everything. Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
93 lines
3.2 KiB
YAML
93 lines
3.2 KiB
YAML
# This file defines the static Buildkite pipeline which attempts to
|
|
# create the dynamic pipeline of all depot targets.
|
|
#
|
|
# If something fails during the creation of the pipeline, the fallback
|
|
# is executed instead which will simply report an error to Gerrit.
|
|
---
|
|
steps:
|
|
- label: ":llama:"
|
|
key: "pipeline-gen"
|
|
command: |
|
|
set -ue
|
|
|
|
if test -n "$${GERRIT_CHANGE_URL-}"; then
|
|
echo "This is a build of [cl/$$GERRIT_CHANGE_ID]($$GERRIT_CHANGE_URL) (at patchset #$$GERRIT_PATCHSET)" | \
|
|
buildkite-agent annotate
|
|
fi
|
|
|
|
nix-build -A ops.pipelines.depot -o pipeline --show-trace
|
|
|
|
# Steps need to be uploaded in reverse order because pipeline
|
|
# upload prepends instead of appending.
|
|
ls pipeline/chunk-*.json | tac | while read chunk; do
|
|
buildkite-agent pipeline upload $$chunk
|
|
done
|
|
|
|
buildkite-agent artifact upload pipeline/drvmap.json
|
|
|
|
# Wait for all previous steps to complete.
|
|
- wait: null
|
|
continue_on_failure: true
|
|
|
|
# Exit with success or failure depending on whether any other steps
|
|
# failed.
|
|
#
|
|
# This information is checked by querying the Buildkite GraphQL API
|
|
# and fetching the count of failed steps.
|
|
#
|
|
# This step must be :duck: (yes, really!) because the post-command
|
|
# hook will inspect this name.
|
|
#
|
|
# Note that this step has requirements for the agent environment, which
|
|
# are enforced in our NixOS configuration:
|
|
#
|
|
# * curl and jq must be on the $PATH of build agents
|
|
# * besadii configuration must be readable to the build agents
|
|
- label: ":duck:"
|
|
key: ":duck:"
|
|
command: |
|
|
set -ueo pipefail
|
|
|
|
readonly FAILED_JOBS=$(curl 'https://graphql.buildkite.com/v1' \
|
|
--silent \
|
|
-H "Authorization: Bearer $(cat /run/agenix/buildkite-graphql-token)" \
|
|
-d "{\"query\": \"query BuildStatusQuery { build(uuid: \\\"$BUILDKITE_BUILD_ID\\\") { jobs(passed: false) { count } } }\"}" | \
|
|
jq -r '.data.build.jobs.count')
|
|
|
|
echo "$$FAILED_JOBS build jobs failed."
|
|
|
|
if (( $$FAILED_JOBS > 0 )); then
|
|
exit 1
|
|
fi
|
|
|
|
# After duck, on success, create a gcroot if the build branch is
|
|
# canon.
|
|
#
|
|
# We care that this anchors *most* of the depot, in practice it's
|
|
# unimportant if there is a build race and we get +-1 of the
|
|
# targets.
|
|
#
|
|
# Unfortunately this requires a third evaluation of the graph, but
|
|
# since it happens after :duck: it should not affect the timing of
|
|
# status reporting back to Gerrit.
|
|
- label: ":anchor:"
|
|
if: "build.branch == 'refs/heads/canon'"
|
|
command: |
|
|
nix-build -A ci.gcroot --out-link /nix/var/nix/gcroots/depot/canon
|
|
depends_on:
|
|
- step: ":duck:"
|
|
allow_failure: false
|
|
|
|
# Create a revision number for the current commit for builds on
|
|
# canon.
|
|
#
|
|
# This writes data back to Gerrit using the Buildkite agent
|
|
# credentials injected through a git credentials helper.
|
|
#
|
|
# Revision numbers are defined as the number of commits in the
|
|
# lineage of HEAD, following only the first parent of merges.
|
|
- label: ":git:"
|
|
if: "build.branch == 'refs/heads/canon'"
|
|
command: |
|
|
git -c 'credential.helper=gerrit-creds' \
|
|
push origin "HEAD:refs/r/$(git rev-list --count --first-parent HEAD)"
|