No description
Find a file
sterni 57d5988b34 feat(nix/dependency-analyzer): find deps among a list of known drvs
This was written with the same intention (and reuses a little of its
code) as cl/5060 and cl/5063: We want to be able to emit dependencies
between //nix/buildkite pipeline steps, so that no agent is occupied
with waiting on locks for derivations built by a different agent.

This dependency information is already available to the Nix store
implementation (e.g. via `nix-store --query --references`) and can also
be obtained in the Nix language which is important, since the pipeline
is generated at evaluation time. (Note: For Nix 2.3, you either need a
strong convention about how derivations expose their dependencies (which
we don't) or rely on store implementation internals (drv files).
For Nix 2.6 there is a better trick, but it also relies on the existence
of drv files.)

The actual task can be formulated as follows: Given a set of
derivations, calculate the the closest derivations also in the input
each derivation depends on. (We call these (next) known dependencies.)
This is crucial because pipeline step often depend on each other only
indirectly with any number of intermediate derivations. For cl/5064 I
determined that 6 intermediate layers is quite common for dependencies
that are perceived to be “direct”.

This problem is solved as follows:

1. Calculate the dependency graph of the combined dependency closure of
   all input derivations. This is quite easy and fairly quick thanks to
   the C++ implementation of builtins.genericClosure. One weak point of
   the current implementation is that the function to determine the
   direct derivation dependencies for Nix < 2.6 is quite hacky.

2. Take the graph from 1. and calculate a dependency graph that only
   connects the known derivations of the input, but retains all
   connections between them (minus intermediate nodes).

In practice the dependency graph is represented as an attribute set
mapping derivation paths to a list of derivation paths it depends on.
The second step is performed by adding a second list of known derivation
paths it depends on.

The main improvements over the previous concept (cl/5060 and cl/5063):

* We only try to find the closest known dependencies in the dependency
  graph whereas we would traverse emit dependencies for the entire
  dependency closure.

* We immediately store the calculation of the closest known dependency
  in the dependency graph, even for intermediate nodes. This avoids
  recalculating the connection (which was a big drawback of the previous
  approach) and makes the calculation itself cheaper.

You can run `mg build //nix/dependency-analyzer:example` to build a
visualization of the internal dependencies between `depot.ci.targets` as
discovered by dependency-analyzer.

Change-Id: If8c0cdfc8470d4b337336257d9818aaa0d51110f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6832
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-08 10:59:45 +00:00
.gcroots feat(.envrc): gcroot third_party.sources 2022-09-15 11:27:53 +00:00
.nixery feat(ops/modules): Add module for running Nixery 2021-08-12 14:55:59 +00:00
corp feat(tvix/eval): add method for emitting runtime warnings 2022-10-07 14:24:36 +00:00
docs docs(CONTRIBUTING): Mention new subtree commit type 2021-11-30 22:42:34 +00:00
fun chore(3p/sources): Bump channels & overlays 2022-09-28 08:02:31 +00:00
lisp chore(gerrit): migrate OWNERS files to code-owners style 2022-09-19 11:13:28 +00:00
net chore(3p/sources): Bump channels & overlays 2022-09-28 08:02:31 +00:00
nix feat(nix/dependency-analyzer): find deps among a list of known drvs 2022-10-08 10:59:45 +00:00
ops feat(ops/pipelines): allow accessing the nix store 2022-10-08 10:59:45 +00:00
third_party chore(3p/sources): Bump channels & overlays 2022-10-06 23:23:42 +00:00
tools chore(nixery): use ldflags parameter instead of buildFlagsArray 2022-10-02 13:39:28 +00:00
tvix refactor(tvix/eval): implement ToSpan directly for rnix::TextRange 2022-10-08 10:58:42 +00:00
users fix(wpcarro/blog): typos, grammatical errors 2022-10-07 19:08:18 +00:00
views feat(tvl-kit): Expose //tools/checks in tvl-kit 2022-06-07 09:40:06 +00:00
web fix(web/static/css): Improve paragraph & line heights 2022-10-06 10:32:02 +00:00
.envrc feat(.envrc): gcroot third_party.sources 2022-09-15 11:27:53 +00:00
.git-blame-ignore-revs fix: add cl/4397 (treewide nixpkgs-fmt) to git-blame-ignore-revs 2022-02-07 18:15:09 +00:00
.gitignore feat(.envrc): gcroot third_party.sources 2022-09-15 11:27:53 +00:00
.hgignore chore(hgignore): ignore .git for hg 2020-06-14 18:23:13 +00:00
.mailmap chore(mailmap): add my name to mailmap 2020-07-18 18:15:05 +00:00
.rgignore chore: Only exclude //third_party/git from ripgrep 2020-05-17 23:58:22 +01:00
buf.yaml feat(buf.yaml): add protobuf linter config 2020-07-17 20:01:29 +00:00
default.nix feat: Add top-level system parameter 2022-06-03 23:27:03 +00:00
LICENSE docs(readTree): Add SPDX license header on readTree 2021-04-12 13:56:44 +00:00
OWNERS chore(gerrit): migrate OWNERS files to code-owners style 2022-09-19 11:13:28 +00:00
README.md docs(README.md): reflect recent upheaval in depot 2022-05-27 23:24:28 +00:00
RULES feat(whitby): Let sterni bear the wheel 2021-05-23 19:06:15 +00:00
rustfmt.toml feat(depotfmt): Check & format Rust code with rustfmt 2022-02-08 12:06:39 +00:00

depot

Build status

This repository is the monorepo for the community around The Virus Lounge, containing our personal tools and infrastructure. Everything in here is built using Nix.

A large portion of the software here is very self-referential, meaning that it exists to sustain the operation of the repository. This is the case because we partially see this as an experiment in tooling for monorepos.

Highlights

Services

  • Source code is available primarily through Sourcegraph on cs.tvl.fyi, where it is searchable and even semantically indexed. A lower-tech view of the repository is also available via cgit-pink on code.tvl.fyi.

    The repository can be cloned using git from https://cl.tvl.fyi/depot.

  • All code in the depot, with the exception of code that is checked in to individual //users folders, needs to be reviewed. We use Gerrit on cl.tvl.fyi for this.

  • Issues are tracked via our own issue tracker on b.tvl.fyi. Its source code lives at //web/panettone/.

  • Smaller todo-list entries which do not warrant a separate issue are listed at todo.tvl.fyi.

  • We use Buildkite for CI. Recent builds are listed on tvl.fyi/builds and pipelines are configured dynamically via //ops/pipelines.

  • A search service that makes TVL services available via textual shortcuts is available: atward

All services that we host are deployed on NixOS machines that we manage. Their configuration is tracked in //ops/{modules,machines}.

Nix

  • //nix/readTree contains the Nix code which automatically registers projects in our Nix attribute hierarchy based on their in-tree location
  • //tools/nixery contains the source code of Nixery, a container registry that can build images ad-hoc from Nix packages
  • //nix/yants contains Yet Another Nix Type System, which we use for a variety of things throughout the repository
  • //nix/buildGo implements a Nix library that can build Go software in the style of Bazel's rules_go. Go programs in this repository are built using this library.
  • //nix/buildLisp implements a Nix library that can build Common Lisp software. Currently only SBCL is supported. Lisp programs in this repository are built using this library.
  • //web/blog and //web/atom-feed: A Nix-based static site generator which generates the web page and Atom feed for tazj.in (//users/tazjin/homepage) and tvl.fyi (//web/tvl)
  • //web/bubblegum contains a CGI-based web framework written in Nix.
  • //nix/nint: A shebang-compatible interpreter wrapper for Nix.
  • //tvix contains initial work towards a modular architecture for Nix.

We have a variety of other tools and libraries in the //nix folder which may be of interest.

Packages / Libraries

  • //net/alcoholic_jwt contains an easy-to-use JWT-validation library for Rust
  • //net/crimp contains a high-level HTTP client using cURL for Rust
  • //tools/emacs-pkgs contains various useful Emacs libraries, for example:
    • dottime.el provides dottime in the Emacs modeline
    • nix-util.el provides editing utilities for Nix files
    • term-switcher.el is an ivy-function for switching between vterm buffers
    • tvl.el provides helper functions for interacting with the TVL monorepo
  • //lisp/klatre provides a grab-bag utility library for Common Lisp

User packages

Contributors to the repository have user directories under //users, which can be used for personal or experimental code that does not require review.

Some examples:

  • //users/grfn/xanthous: A (WIP) TUI RPG, written in Haskell.
  • //users/tazjin/emacs: tazjin's Emacs & EXWM configuration
  • //users/tazjin/finito: A persistent finite-state machine library for Rust.

Licensing

Unless otherwise stated in a subdirectory, all code is licensed under the MIT license. See LICENSE for details.

Contributing

If you'd like to contribute to any of the tools in here, please check out the contribution guidelines and our code of conduct.

IRC users can find us in #tvl on hackint, which is also reachable via XMPP at #tvl@irc.hackint.org (sic!).

Hackint also provide a web chat.