mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/ops/pipelines/depot.nix

48 lines
1.5 KiB
Nix
Raw Normal View History

feat(ops/pipelines): Add Buildkite pipeline configuration This adds configuration which generates the structure expected for Buildkite pipelines, which can then be dynamically ingested by Buildkite when a pipeline is triggered. Change-Id: I61e3dc3affb19c1f2550ef827fa73b17f8d8ae47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/627 Reviewed-by: ericvolp12 <ericvolp12@gmail.com> Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-27 18:14:58 +02:00
# This file configures the primary build pipeline used for the
# top-level list of depot targets.
feat(nix/buildkite): Check target map of parent to determine skips This changes the logic for build pipeline generation to inspect an (optional) parentTargetMap attribute which contains the derivation map of a target commit. Targets that existed in a parent commit with the same drv hash will be skipped, as they are not considered to have changed. This does not yet wire up any logic for retrieving the target map from storage, meaning that at this commit all targets are always built. The intention is that we will have logic to fetch the target map (initially from Buildkite artefact storage), which we then pass to the depot via externalArgs when actually generating the pipeline. Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-16 13:55:36 +01:00
{ depot, pkgs, externalArgs, ... }:
feat(ops/pipelines): Add Buildkite pipeline configuration This adds configuration which generates the structure expected for Buildkite pipelines, which can then be dynamically ingested by Buildkite when a pipeline is triggered. Change-Id: I61e3dc3affb19c1f2550ef827fa73b17f8d8ae47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/627 Reviewed-by: ericvolp12 <ericvolp12@gmail.com> Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-27 18:14:58 +02:00
let
feat(ops/pipelines): Dynamically generate CI pipeline from targets Create the pipeline by outputting a file that contains nix-build invocations for each target's *derivation path*. Each invocation has a generated Nix expression passed to it with `-E` which fetches the correct target from the tree while correctly handling targets with strange characters (such as in Go-packages). This makes it possible to run target-level granular pipelines. We're getting somewhere! Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi> Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-27 02:05:45 +02:00
# Protobuf check step which validates that changes to .proto files
# between revisions don't cause backwards-incompatible or otherwise
# flawed changes.
protoCheck = {
command = "${depot.nix.bufCheck}/bin/ci-buf-check";
label = ":water_buffalo:";
};
refactor(tools/depotfmt): Move depotfmt check into a real build step Produces more useful output and also makes for a good target for the upcoming extraSteps logic. Change-Id: Ifd389d433d9e27f97940a48999f4fba35646e37a Reviewed-on: https://cl.tvl.fyi/c/depot/+/4727 Tested-by: BuildkiteCI Autosubmit: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-12-28 11:38:03 +01:00
feat(ops/pipelines): Create drvmap structure for each commit Always create a structure that maps all targets to derivations, and persist it as a JSON file. This relates to some of the ideas expressed in: https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit The file is always uploaded to Buildkite as an artifact. This allows for retrieving it based on the commit ID in a Buildkite GraphQL query. By default, Buildkite stores artefacts for 6 months. Storage location can be overridden (with custom retention) through some environment variables, but for now at TVL the Buildkite-managed storage is fine. See also: https://buildkite.com/docs/pipelines/artifacts In the subsequent filtering implementation, when diffing commits across a time-range that exceeds artefact retention time, we should simply default to building everything. Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-10-01 12:43:53 +02:00
pipeline = depot.nix.buildkite.mkPipeline {
headBranch = "refs/heads/canon";
drvTargets = depot.ci.targets;
refactor(depotfmt): Move formatting check into an extra step Change-Id: I7e4cf6bb2351bd11a5396f1663c0d4cc97c0d94e Reviewed-on: https://cl.tvl.fyi/c/depot/+/5009 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-19 19:23:36 +01:00
additionalSteps = [ protoCheck ];
feat(nix/buildkite): Check target map of parent to determine skips This changes the logic for build pipeline generation to inspect an (optional) parentTargetMap attribute which contains the derivation map of a target commit. Targets that existed in a parent commit with the same drv hash will be skipped, as they are not considered to have changed. This does not yet wire up any logic for retrieving the target map from storage, meaning that at this commit all targets are always built. The intention is that we will have logic to fetch the target map (initially from Buildkite artefact storage), which we then pass to the depot via externalArgs when actually generating the pipeline. Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-16 13:55:36 +01:00
parentTargetMap = if (externalArgs ? parentTargetMap)
then builtins.fromJSON (builtins.readFile externalArgs.parentTargetMap)
else {};
refactor(ops/pipelines): Move :anchor: into postBuildSteps There is no need for this step to be part of the static pipeline (it should not run if the build fails anyways). Change-Id: I71400a452d6f8f4708d146b346eaffda5da2f766 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5049 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-22 12:47:32 +01:00
postBuildSteps = [
# After successful builds, create a gcroot for builds on canon.
#
# This anchors *most* of the depot, in practice it's unimportant
# if there is a build race and we get +-1 of the targets.
#
# Unfortunately this requires a third evaluation of the graph, but
# since it happens after :duck: it should not affect the timing of
# status reporting back to Gerrit.
{
label = ":anchor:";
branches = "refs/heads/canon";
command = ''
nix-build -A ci.gcroot --out-link /nix/var/nix/gcroots/depot/canon
'';
}
];
feat(ops/pipelines): Create drvmap structure for each commit Always create a structure that maps all targets to derivations, and persist it as a JSON file. This relates to some of the ideas expressed in: https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit The file is always uploaded to Buildkite as an artifact. This allows for retrieving it based on the commit ID in a Buildkite GraphQL query. By default, Buildkite stores artefacts for 6 months. Storage location can be overridden (with custom retention) through some environment variables, but for now at TVL the Buildkite-managed storage is fine. See also: https://buildkite.com/docs/pipelines/artifacts In the subsequent filtering implementation, when diffing commits across a time-range that exceeds artefact retention time, we should simply default to building everything. Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-10-01 12:43:53 +02:00
};
drvmap = depot.nix.buildkite.mkDrvmap depot.ci.targets;
in pkgs.runCommandNoCC "depot-pipeline" {} ''
mkdir $out
cp -r ${pipeline}/* $out
cp ${drvmap} $out/drvmap.json
''
Reference in a new issue Copy permalink