Commit graph

266 commits

Author SHA1 Message Date
Florian Klink
32f41458c0 refactor(tvix): move castore into tvix-castore crate
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.

Notable changes:
 - `fixtures` and `utils` had to be moved out of the `test` cfg, so they
   can be imported from tvix-store.
 - Some ad-hoc fixtures in the test were moved to proper fixtures in the
   same step.
 - The protos are now created by a (more static) recipe in the protos/
   directory.

The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)

Bug: https://b.tvl.fyi/issues/301

Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:51:21 +00:00
Vincent Ambo
c5770e65dd feat(nix/renderMarkdown): add optional tagfilter argument
Makes it possible to disable tag filtering for rendered content (on by
default, of course).

Change-Id: I74ecfee97eaa7abf32049172b28705e7a3f21548
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9276
Tested-by: BuildkiteCI
Reviewed-by: Mark Shevchenko <markshevchenko@gmail.com>
2023-09-08 13:51:26 +00:00
Vincent Ambo
fcda068235 feat(nix/sparseTree): add optional name argument
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.

I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.

Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-08 20:11:56 +00:00
Vincent Ambo
9f762bf623 docs(nix-1p): explain scope of with
This fixes https://github.com/tazjin/nix-1p/issues/13

Due to with's use of a semicolon instead of some kind of bracing
statement, people can get confused about what the scope of a `with`
is.

Hopefully this change should make it more explicit.

Change-Id: Ifaff28e09a2e8b8b244a9c11d44b2780dee284ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8951
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-01 19:53:31 +00:00
Florian Klink
8b870c7ec9 feat(nix/buildkite): add meta.ci.buildkiteExtraStepArgs
This allows setting / overwriting arbitrary step arguments, such as setting
[Retry attributes](https://buildkite.com/docs/pipelines/command-step#retry-attributes)
for a specific readTree target.

It's intended to be used for cases where modelling each and every option
in a custom meta.ci attribute would be unfeasible.

Change-Id: I3352d5353b26a41a16760a7df37cd5ffee1665bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8983
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-07-24 20:51:14 +00:00
Florian Klink
f41b611761 feat(nix/buildkite): add meta.ci.buildkiteExtraDeps
This allows setting a buildkiteExtraDeps in `meta.ci`, which will get
added to `depends_on` (which can also be a list).

It allows explicitly describing a dependency on another buildkite step,
either generated by nix/buildkite, or part of the static pipeline.

At some point we might want to expose our key calculation function too,
similar to how readTree exposes mkLabel already, but that's left for
a followup.

Change-Id: I793170401ccd3907ad8bf232b80ca7a492e1c942
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8980
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-24 20:51:14 +00:00
Florian Klink
728de762fd feat(nix/buildkite): support meta.timeout
This uses the nixpkgs convention of meta.timeout, and adds a
timeout_in_minutes field to the pipeline step in case its set.

Fixes https://b.tvl.fyi/issues/285.

Change-Id: Ia72e3832f14bf9172319bce070c5b0944f1c96fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8970
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-07-18 16:47:58 +00:00
Florian Klink
6c630aadc8 feat(nix/buildkite): drop postBuild logic
This has been scheduled for removal in 2022-10-01, but it didn't happen
so far. Let's remove it now, better late than never.

Closes https://b.tvl.fyi/issues/286.

Change-Id: I26fe07360c694c1c93418f3310b4067051e08c87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8969
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-18 16:47:58 +00:00
Vincent Ambo
5b1c327c82 docs(nix-1p): suggest using nix repl to experiment with commands
This fixes https://github.com/tazjin/nix-1p/issues/14

I guess some people also don't discover the REPL for a while unless
told about it.

Change-Id: I76366c1cb02b5612a5aa87fe33b40b7bb0db3ff3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8950
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-07-11 16:06:23 +00:00
Vincent Ambo
09486d483c docs(nix-1p): import is a builtin, not a keyword
Pointed out by @Gaelan on GH in
https://github.com/tazjin/nix-1p/issues/12

As noted there, some Nix syntax highlighters will colour `import`
differently (e.g. the Emacs nix-mode), but it's not technically a
keyword in the language.

Change-Id: I1bbd36261cda31deb9ba47380a2810e64ba03ea1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8832
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-19 22:20:39 +00:00
Florian Klink
4ddfdc2ae0 feat(nix/buildkite): extraSteps: include parentLabel into hash
Contrary to the normal steps, extra steps only show the attribute name
in their label, not the whole attribute path.

To make sure these are still unique, also incorporate the parentLabel
into the string that's hashed.

Change-Id: Ic7a1f76aaac159ab5bd30054e422234882947990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8708
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-03 14:09:28 +00:00
Florian Klink
ed159e3a1d feat(nix/buildkite): add step key to release steps
This uses the same logic ("sha1sum of label") as already used in the
"build" chunks.

Fixes https://b.tvl.fyi/issues/276.

Change-Id: I76e59a141bd05f7d1d997a6471ea65e8cc7b4c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8707
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-03 13:47:22 +00:00
Florian Klink
4cf0ee8ea3 docs(nix/buildkite): escape "release" in warning
When using the buildkite tooling with `postBuild` parameter set, the
following error message is shown:

```
This replaces the previous boolean postBuild API in extra
step definitions. Please remove the postBuild parameter from
this step and instead set phase = release;.
```

However, `release` is no symbol, this should be a string.

Update the message to reflect that.

Change-Id: I341c81d862204db06946512428910cada28481ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8690
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-01 10:34:12 +00:00
sterni
830bc9961a chore: remove comments formerly meant for alignment
/**/ is a nice way to align if statements which doesn't work with
nixpkgs-fmt, since it'll reflow the comment to the line preceding the
if. Consequently, we can delete these comments now.

Change-Id: Ifa5327f846a903e07607b21f8eedbc32fc36f758
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8689
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-05-31 10:55:07 +00:00
Vincent Ambo
159646787a feat(nix/lazy-deps): add override pattern for deps
Introduces a `.overrideDeps` attribute with which additional tools can
be supplied. This works like `.override` in nixpkgs.

Change-Id: I69a009b51f7f073a2d030eda5e3b5310e0f8e883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8491
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-04-24 10:56:40 +00:00
Vincent Ambo
44a5e14200 feat(nix/readTree): implement .skip-tree marker for subtree ignoring
With this change, readTree gains the ability to notice a `.skip-tree`
marker in addition to the `.skip-subtree` marker.

The behaviour of the new marker will completely ignore the folder that
the marker is located in (i.e. no node will be present for it in the
parent at all).

To make this work, the recursive function in readTree had to be
modified to return a sentinel value (noting that a tree has requested
to be skipped) which is then filtered out when constructing the list
of children. The actual `readTree` function is now a wrapper around
this inner, sentinel-yielding implementation which unwraps the result
set.

For obvious reasons, `.skip-tree` is not allowed at the top-level and
readTree will throw an error if it encounters it there.

Fixes: b/244
Change-Id: Ica731bc1af356e881fd3d31c7109f62ffd2762ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8185
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-28 12:15:33 +00:00
Florian Klink
7df27dad12 feat(nix/bufCheck): always run from repo root
This doesn't work when run from subdirectories, and going there manually before
running `mg run //nix/bufCheck` is annoying.

Change-Id: Icd30a7596ff0dfe2781f7cfa1b4085cbfdebd6ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7324
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2022-12-27 13:27:40 +00:00
Florian Klink
a2ac8a66c1 feat(nix/bufCheck): ensure .pb.go is up to date
Change-Id: I03e1c3aced266966f6959dbce05c1f1fb1480b5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7323
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 13:27:40 +00:00
Florian Klink
54642c822d chore(nix/buildGo): drop thing.proto from example
This was missed from cl/7536.

Change-Id: I12c12bf91bad00c30bb63518d65c320a61cbf442
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7641
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
7fbac93940 chore(nix/buildGo): drop buildGo.proto and buildGo.grpc
As described in  https://b.tvl.fyi/issues/221#comment-344, buildGo.proto was a
mistake and should be removed.

Change-Id: Ic588a5e8eea58e83e3ec9a37ac681ce526028718
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7536
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-26 11:17:22 +00:00
zimbatm
66a8ce5900 feat(nix/lazy-deps): add passthru.devShell
Allows to easily convert a lazy-deps into a nix-shell compatible
derivation.

Change-Id: I65a0609642f12e0d00658c69cc394a73dddccd16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5855
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 10:31:34 +00:00
zimbatm
99030d10ce refactor(nix/lazy-deps): use runCommand
writeTextFile is nice, but not flexible enough to allow the passthru
argument needed for a follow-up change.

Change-Id: I4f0cffd0f29b2c06b0155101d3806c9c5745c37a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5854
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 10:31:34 +00:00
sterni
19ddb1466e revert(nix/utils): onlyDrvPath to get the drvPath w/o the outputs
This reverts commit 6813598c17.

Reason for revert: This function is a re-implementation of
builtins.unsafeDiscardOutputDependency which I missed at the time.

Change-Id: I5bb52bfd5e8d51defaf90ee795b0fe99be84f6db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7265
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-09 15:36:29 +00:00
sterni
57cf952ea9 chore(3p/sources): Bump channels & overlays (OpenSSL edition)
* //ops/machines/whitby: Disable grafana, since the grafana module was
  changed upstream in a way that our configuration no longer works.
  Since the OpenSSL security update is relatively pressing, adapting the
  grafana configuration beforehand is not a hard requirement. See
  https://github.com/NixOS/nixpkgs/pull/191768.

* //tools/depotfmt: keep Go at version 1.18 to forgo a reformat of the
  tree.

* //nix/buildGo: keep Go at version 1.18, as 1.19 changed the CLI
  interface (?) in a way that breaks buildGo.

* //3p/overlays/tvl: drop upstreamed tdlib upgrade.

* //3p/overlays/tvl: patch buf to work around breakage due to git 2.38.1

TODO items for Go are tracked in b/215.

Change-Id: Ie08fef49cf3db12e6b5225a8b992a990ddc5b642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7141
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-03 15:10:39 +00:00
William Carroll
1cd7678bdb feat(ops/buildkite): Support "soft failures"
Example Usage:

```nix
{ pkgs, ... }:

(pkgs.writeText "foo" "bar").overrideAttrs (_: {
  meta.ci.extraSteps.bar = {
    label = "testing soft_fail";
    softFail = [
      { exit_status = 253; }
      { exit_status = 251; }
    ];
    command = pkgs.writeShellScript "fail" ''
      # Trigger a soft failure by exiting 253 or 251
      exit 253
    '';
  };
})
```

Buildkite Documentation: https://buildkite.com/changelog/56-command-steps-can-now-be-made-to-soft-fail

Change-Id: Idfce15a8d9cde568aa24ebd03956ada399bba5d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7076
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-01 22:11:05 +00:00
Florian Klink
6025242fc7 chore(buf): Use nixpkgs-provided buf
The version of buf used is quite old.

nixpkgs provides a more recent version, but it requires us to migrate
config to the latest version.

depot_scanner.proto doesn't honor some of the conventions, so we need
allow_comment_ignores and drop a bunch of comments in there.

Change-Id: Ic978fe92fb7c8471f58c137497528f18aad8f3ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7053
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
2022-10-21 18:39:03 +00:00
sterni
5753fc14f5 refactor(nix/dependency-analyzer): use unsafeDiscardOutputDependency
Turns out that pathContextDrvPath already exists as a builtin which
is very convenient. Actually somewhat embarassing that I missed this
for so long.

Change-Id: Ieb5e113d70dec548b3053911ff9dbe9ed48402be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7050
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 18:21:25 +00:00
Evgeny Zemtsov
b5b968e103 fix(nix/buildkite) don't fail if gated step has no branches
Change-Id: Id3365c18bc3904b42e8b129a513cff4480a2cb07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7051
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 15:27:31 +00:00
sterni
8149ca5c38 fix(nix/dependency-analyzer): don't read from non-drv store paths
Emitting dependencies on non-drv store paths from drv directDrvDeps is
fine and actually correct, even though the Nix 2.3 version can't do it
at the moment (but this would change when the placeholder implementation
is replaced using a drv parser). However, we can't necessarily determine
the dependencies of non-drv store paths because such store paths may be
binary files that can't be read in by readFile due to NUL bytes.

Change-Id: Ifbd101adaee4f32f10c010fa79e19b9b1127fc6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6945
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-11 18:35:50 +00:00
sterni
81fadbe6eb fix(nix/buildGo): fix for Nix >= 2.6 readFile changes
The string context retained by readFile would leak into attribute keys
in fromJSON which may not have string context in any Nix version. We
don't need Nix >= 2.6 support, but buildGo may have external users and
this change is simple enough.

Change-Id: I593f1ef513502691119428d26d508a5f4d378543
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6946
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-11 17:38:42 +00:00
Evgeny Zemtsov
b319e00831 fix(nix/buildkite): split extra steps based on all known phases
Fix a bug introduced by 2ca153141 (I merged the wrong patchset).

Issue happens when pipeline is split by phases into independent
evals (e.g. build/release/deploy).

Splitting extra steps requires knowledge of all known phases,
otherwise pipeline evaluation fails due to extra steps from inactive
phases.

Change-Id: Iab0f2dc3eadda281e483055e26f00a95442e15b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6942
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-11 09:17:25 +00:00
Griffin Smith
b0547ccfa5 feat(nix/readTree): Handle a builtins w/o scopedImport
We (might) not want to implement scopedImport in tvix given it's
considered a bit of a misfeature; this makes readTree work with a
`builtins` set that doesn't have it (and if we decide we do want tvix to
have scopedImport, we can revert this pretty easily).

Change-Id: Ia3bbc847514672063a607d977ce167d489fa1131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6915
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-10 17:42:48 +00:00
Evgeny Zemtsov
2ca153141d feat(nix/buildkite): allow custom phases
This change automatically extends the list of known phases as soon as
they are added to active phase list.

This is great when a user wants to design pipelines with multiple
groups of dynamic steps.

For example in Resoptima we want to design deployment pipeline where
first only staging k8s namespaces are updated/tested and only after,
we update production.

Change-Id: Iab0f2dc3eadda281e483055e26f00a95442e15b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6923
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-10 15:40:12 +00:00
sterni
5174c21637 fix(nix/tag): correct no match check in discr
It uses discrDef internally, but passes `null` as the default tag name,
causing Nix to drop the attribute and return an empty attribute set if
the default case is hit. Consequently we need to check for the empty
attribute set, not `null` to figure out if there was no match found.

We can also test this behavior using `assertThrows` which was introduced
after the tag library was originally written.

Change-Id: I45adb2f9602762dfc867956323fb3f5ae4c8bd1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6904
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2022-10-09 13:25:14 +00:00
sterni
70427fd934 refactor(nix/stateMonad): optimize after
This should save on one function application which can be a big deal for
bigger for_ loops, I suspect. It's not really complicated, so why not.

Change-Id: I2bfcd254e55f1bea366b09de294b2bef9f5b5dda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6834
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-08 10:59:45 +00:00
sterni
3fab3b780b feat(nix/dependency-analyzer): improved directDrvDeps for Nix >= 2.6
This codepath will basically never be used in depot, but I want to add
it as kind of a note to myself. It's kind of a neat feature, although
I'm not quite sure it is going to stick around.

Change-Id: If0e26ef47bdedc6dbf3d048ad4fc9a3a1fd6c5a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6833
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-08 10:59:45 +00:00
sterni
57d5988b34 feat(nix/dependency-analyzer): find deps among a list of known drvs
This was written with the same intention (and reuses a little of its
code) as cl/5060 and cl/5063: We want to be able to emit dependencies
between //nix/buildkite pipeline steps, so that no agent is occupied
with waiting on locks for derivations built by a different agent.

This dependency information is already available to the Nix store
implementation (e.g. via `nix-store --query --references`) and can also
be obtained in the Nix language which is important, since the pipeline
is generated at evaluation time. (Note: For Nix 2.3, you either need a
strong convention about how derivations expose their dependencies (which
we don't) or rely on store implementation internals (drv files).
For Nix 2.6 there is a better trick, but it also relies on the existence
of drv files.)

The actual task can be formulated as follows: Given a set of
derivations, calculate the the closest derivations also in the input
each derivation depends on. (We call these (next) known dependencies.)
This is crucial because pipeline step often depend on each other only
indirectly with any number of intermediate derivations. For cl/5064 I
determined that 6 intermediate layers is quite common for dependencies
that are perceived to be “direct”.

This problem is solved as follows:

1. Calculate the dependency graph of the combined dependency closure of
   all input derivations. This is quite easy and fairly quick thanks to
   the C++ implementation of builtins.genericClosure. One weak point of
   the current implementation is that the function to determine the
   direct derivation dependencies for Nix < 2.6 is quite hacky.

2. Take the graph from 1. and calculate a dependency graph that only
   connects the known derivations of the input, but retains all
   connections between them (minus intermediate nodes).

In practice the dependency graph is represented as an attribute set
mapping derivation paths to a list of derivation paths it depends on.
The second step is performed by adding a second list of known derivation
paths it depends on.

The main improvements over the previous concept (cl/5060 and cl/5063):

* We only try to find the closest known dependencies in the dependency
  graph whereas we would traverse emit dependencies for the entire
  dependency closure.

* We immediately store the calculation of the closest known dependency
  in the dependency graph, even for intermediate nodes. This avoids
  recalculating the connection (which was a big drawback of the previous
  approach) and makes the calculation itself cheaper.

You can run `mg build //nix/dependency-analyzer:example` to build a
visualization of the internal dependencies between `depot.ci.targets` as
discovered by dependency-analyzer.

Change-Id: If8c0cdfc8470d4b337336257d9818aaa0d51110f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6832
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-08 10:59:45 +00:00
vpfeiffer
55ea15ab83 docs(nix/nix-1p): Update to use final and prev
The convention has changed to use `final` instead of `self` and `prev`
instead of `super`. This new convention is much easier to understand
especially for users that are new to Nix and Nixpkgs.

This change is notable in the Nixpkgs project code. I added a note
mentioning that the official documentation hasn't been update with
this information. The documentation is currently undergoing an
overhaul and that is why I believe this hasn't been reflected in the
nixpkgs documentation as of yet.

Imported-From: https://github.com/tazjin/nix-1p/pull/9/commits
Change-Id: I901df69a564969b1784b3d7ac629cfaf85093acb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6859
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-05 11:14:38 +00:00
sterni
5e097aa8e9 feat(nix/stateMonad): simple Nix state monad implementation
In the absence of do syntactic sugar relatively tedious to write, but
useful to express certain types of algorithms. I found it useful to
memoize intermediate results as they are calculated in order to avoid
recomputing them later in a drv dependency analyzer I've written.

Change-Id: I47cf3c644a96952c70276c9fa4cb3190b1c1e027
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6828
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-01 17:47:07 +00:00
Evgeny Zemtsov
335bf6900d fix(nix/buildkite): follow parent skip behavior in extra steps
We found a bug after updating to latest tvl-kit which broke
incremental releases.

Bug was related to the fact that extra steps skip attribute had
precedence over parent configuration. This is a desired behavior when
extra step is explicitly set to `skip=true` but otherwise it must
follow parent.

Due to extra step normalization skip parameter is always set to false
if not explicitly set.

Along the way, I'm adding support for setting skip attribute on extra
steps as string so that people can define skip reasons.

The bug was introduced by commit:
b9d79109d feat(ops/buildkite): Allow skip of individual steps

Change-Id: I8a46d0926a749d2434412b309c661b749e9dbf37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6827
Autosubmit: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-30 13:14:54 +00:00
sterni
0c178a0ef6 chore(3p/sources): Bump channels & overlays
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.

* runCommandNoCC -> runCommand
* gmailieer -> lieer
  We also need to work around the fact that home-manager hasn't catched
  on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
  This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy

Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.

Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-09-28 08:02:31 +00:00
sterni
f39a4e4fa7 refactor(nix/buildLisp/tests/argv0): use derivation for tests
There is no need to use an extraStep, actually, and using derivations
reduces noise on CI.

Change-Id: I897c3c3f7e0acee8f051fcc01450ff57176726f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6573
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-20 10:43:54 +00:00
Luke Granger-Brown
f190712b7f chore(gerrit): migrate OWNERS files to code-owners style
Change-Id: Iacc521dfdd4b4a2d5cef3920cf8189bcce35a488
2022-09-19 11:13:28 +00:00
Evgeny Zemtsov
8fa3bc7137 feat(buildkite/mkPipeline): allow fast failure for build steps
This is supposed to help with resource wasting in situations when
multiple dynamic steps depend on a failing derivation. Cost of
failure currently is `C = T * D` where:
 - T -> time to compile the failing derivation
 - D -> amount of CI targets depending on a failing derivation

Switching to `Fail Fast` limits it to just T (time of a single
failure). Which helps a lot, especially while upgrading nixpkgs.

Fast fail at this moment is in preview, so to enable it:
 - Enable `Fail Fast` feature at pipeline or organization level
 - Set `cancelOnBuildFailing` parameter to true for `mkPipeline`

Change-Id: I4373a46633522d21e94cfa8bac35243b4eeb0b9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6243
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-01 14:35:03 +00:00
sterni
049ec22943 feat(nix/buildLisp): re-enable CCL
The problem went away once again, let's see how long it'll last this time.

As it turns out, CCL has a Unicode Standard conforming string
implementation that doesn't allow the use of (lone) surrogate code
points, requiring us to disable a test in cl-json which tested the
behavior of en- and decoding of such a (technically illegal) string.

Change-Id: I8bfa482934bbf94f86cecdde02d5c3d4e77950a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6204
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-08-30 17:51:35 +00:00
Jean-François Roche
282ad0015b fix(ops/buildkite): do not set branch if empty
To run an extra step for all branches, user don't set the
`branches` attribute. This change avoid setting `branches` to null in such a case.

Change-Id: Iabf2f3d0411b037ece5584f30b29c7e65420b63f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5975
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 14:41:39 +00:00
Jean-François Roche
45c794f1b3 feat(ops/buildkite): Allow agents targeting
Using the `agents` attribute, it lets the user target specific agents to run a step.

Change-Id: Id6fc0981d4879b77598854e1f296dffffb33a405
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5974
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 09:29:58 +00:00
Jean-François Roche
b9d79109dd feat(ops/buildkite): Allow skip of individual steps
Using the `skip` attribute, it lets the user decide to temporarily skip a step.

Change-Id: If63ce60a2b4148f041655912730acc4ae9e28534
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5971
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 09:29:58 +00:00
Vincent Ambo
544d72189c chore(nixery): use nix-1p from within the depot
Since the source of nix-1p is checked in under //nix/nix-1p, we should
use it from there if Nixery is being built inside of depot.

Change-Id: Iddd54f7b93b398b2f909db6ee105366a9914a2ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5882
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-06-16 15:08:27 +00:00
Vincent Ambo
b9c31d6658 refactor(nix/buildkite): Optimise steps if builds are not enabled
Generating a release-only pipeline skips a bigger chunk of eval this
way (the step itself is never actually evaluated, which means we never
actually compute the drv), which can be quite beneficial in terms of
evaluation time.

Change-Id: I2739026ddd1c6a86f82627ac26a046c5fe7359ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5830
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-06-06 12:02:26 +00:00