tvl-depot/nix
sterni 57d5988b34 feat(nix/dependency-analyzer): find deps among a list of known drvs
This was written with the same intention (and reuses a little of its
code) as cl/5060 and cl/5063: We want to be able to emit dependencies
between //nix/buildkite pipeline steps, so that no agent is occupied
with waiting on locks for derivations built by a different agent.

This dependency information is already available to the Nix store
implementation (e.g. via `nix-store --query --references`) and can also
be obtained in the Nix language which is important, since the pipeline
is generated at evaluation time. (Note: For Nix 2.3, you either need a
strong convention about how derivations expose their dependencies (which
we don't) or rely on store implementation internals (drv files).
For Nix 2.6 there is a better trick, but it also relies on the existence
of drv files.)

The actual task can be formulated as follows: Given a set of
derivations, calculate the the closest derivations also in the input
each derivation depends on. (We call these (next) known dependencies.)
This is crucial because pipeline step often depend on each other only
indirectly with any number of intermediate derivations. For cl/5064 I
determined that 6 intermediate layers is quite common for dependencies
that are perceived to be “direct”.

This problem is solved as follows:

1. Calculate the dependency graph of the combined dependency closure of
   all input derivations. This is quite easy and fairly quick thanks to
   the C++ implementation of builtins.genericClosure. One weak point of
   the current implementation is that the function to determine the
   direct derivation dependencies for Nix < 2.6 is quite hacky.

2. Take the graph from 1. and calculate a dependency graph that only
   connects the known derivations of the input, but retains all
   connections between them (minus intermediate nodes).

In practice the dependency graph is represented as an attribute set
mapping derivation paths to a list of derivation paths it depends on.
The second step is performed by adding a second list of known derivation
paths it depends on.

The main improvements over the previous concept (cl/5060 and cl/5063):

* We only try to find the closest known dependencies in the dependency
  graph whereas we would traverse emit dependencies for the entire
  dependency closure.

* We immediately store the calculation of the closest known dependency
  in the dependency graph, even for intermediate nodes. This avoids
  recalculating the connection (which was a big drawback of the previous
  approach) and makes the calculation itself cheaper.

You can run `mg build //nix/dependency-analyzer:example` to build a
visualization of the internal dependencies between `depot.ci.targets` as
discovered by dependency-analyzer.

Change-Id: If8c0cdfc8470d4b337336257d9818aaa0d51110f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6832
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-08 10:59:45 +00:00
..
binify style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
bufCheck refactor(bufCheck): Assume bufCheck is run in depot checkout 2021-08-29 12:41:07 +00:00
buildGo fix(nix/buildGo): Use overrideAttrs to add metadata 2022-02-02 13:36:58 +00:00
buildkite fix(nix/buildkite): follow parent skip behavior in extra steps 2022-09-30 13:14:54 +00:00
buildLisp chore(3p/sources): Bump channels & overlays 2022-09-28 08:02:31 +00:00
buildManPages chore(gerrit): migrate OWNERS files to code-owners style 2022-09-19 11:13:28 +00:00
dependency-analyzer feat(nix/dependency-analyzer): find deps among a list of known drvs 2022-10-08 10:59:45 +00:00
drvSeqL style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
emptyDerivation chore(gerrit): migrate OWNERS files to code-owners style 2022-09-19 11:13:28 +00:00
escapeExecline style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
getBins style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
lazy-deps feat(nix/lazy-deps): Add function to generate lazy binary dispatcher 2022-05-02 23:34:55 +00:00
mergePatch style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
netstring style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
nint chore(gerrit): migrate OWNERS files to code-owners style 2022-09-19 11:13:28 +00:00
nix-1p docs(nix/nix-1p): Update to use final and prev 2022-10-05 11:14:38 +00:00
readTree refactor(readTree): deprecate meta.targets for meta.ci.targets 2022-02-07 15:13:52 +00:00
renderMarkdown chore(3p/sources): Bump channels & overlays 2022-09-28 08:02:31 +00:00
runExecline style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
runTestsuite style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
sparseTree chore(3p/sources): Bump channels & overlays 2022-09-28 08:02:31 +00:00
stateMonad feat(nix/stateMonad): simple Nix state monad implementation 2022-10-01 17:47:07 +00:00
tag style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
tailscale style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
utils chore(gerrit): migrate OWNERS files to code-owners style 2022-09-19 11:13:28 +00:00
writeElispBin chore(3p/sources): Bump channels and overlays 2022-04-21 16:54:07 +00:00
writeExecline style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
writers style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
writeScript style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
writeScriptBin refactor(nix): use our own writeScript(Bin) 2020-06-30 00:19:27 +00:00
yants docs(yants): Update josh cloning instructions 2022-02-19 11:37:00 +00:00
OWNERS chore(gerrit): migrate OWNERS files to code-owners style 2022-09-19 11:13:28 +00:00