tvl-depot/nix/readTree/README.md
Vincent Ambo 44a5e14200 feat(nix/readTree): implement .skip-tree marker for subtree ignoring
With this change, readTree gains the ability to notice a `.skip-tree`
marker in addition to the `.skip-subtree` marker.

The behaviour of the new marker will completely ignore the folder that
the marker is located in (i.e. no node will be present for it in the
parent at all).

To make this work, the recursive function in readTree had to be
modified to return a sentinel value (noting that a tree has requested
to be skipped) which is then filtered out when constructing the list
of children. The actual `readTree` function is now a wrapper around
this inner, sentinel-yielding implementation which unwraps the result
set.

For obvious reasons, `.skip-tree` is not allowed at the top-level and
readTree will throw an error if it encounters it there.

Fixes: b/244
Change-Id: Ica731bc1af356e881fd3d31c7109f62ffd2762ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8185
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-28 12:15:33 +00:00

3.3 KiB

readTree

This is a Nix program that builds up an attribute set tree for a large repository based on the filesystem layout.

It is in fact the tool that lays out the attribute set of this repository.

As an example, consider a root (.) of a repository and a layout such as:

.
├── third_party
│   ├── default.nix
│   └── rustpkgs
│       ├── aho-corasick.nix
│       └── serde.nix
└── tools
    ├── cheddar
    │   └── default.nix
    └── roquefort.nix

When readTree is called on that tree, it will construct an attribute set with this shape:

{
    tools = {
        cheddar = ...;
        roquefort = ...;
    };

    third_party = {
        # the `default.nix` of this folder might have had arbitrary other
        # attributes here, such as this:
        favouriteColour = "orange";

        rustpkgs = {
            aho-corasick = ...;
            serde = ...;
        };
    };
}

Every imported Nix file that yields an attribute set will have a __readTree = true; attribute merged into it.

Traversal logic

readTree will follow any subdirectories of a tree and import all Nix files, with some exceptions:

  • If a folder contains a default.nix file, no sibling Nix files will be imported - however children are traversed as normal.
  • If a folder contains a default.nix it is loaded and, if it evaluates to a set, merged with the children. If it evaluates to anything other than a set, else the children are not traversed.
  • A folder can opt out from readTree completely by containing a .skip-tree file. The content of the file is not read. These folders will be missing completely from the readTree structure.
  • A folder can declare that its children are off-limit by containing a .skip-subtree file. Since the content of the file is not checked, it can be useful to leave a note for a human in the file.
  • The default.nix of the top-level folder on which readTree is called is not read to avoid infinite recursion (as, presumably, this file is where readTree itself is called).

Traversal is lazy, readTree will only build up the tree as requested. This currently has the downside that directories with no importable files end up in the tree as empty nodes ({}).

Import structure

readTree is called with an argument set containing a few parameters:

  • path: Initial path at which to start the traversal.
  • args: Arguments to pass to all imports.
  • filter: (optional) A function to filter the argument set on each import based on the location in the tree. This can be used to, for example, implement a "visibility" system inside of a tree.
  • scopedArgs: (optional) An argument set that is passed to all imported files via builtins.scopedImport. This will forcefully override the given values in the import scope, use with care!

The package headers in this repository follow the form { pkgs, ... }: where pkgs is a fixed-point of the entire package tree (see the default.nix at the root of the depot).

In theory readTree can pass arguments of different shapes, but I have found this to be a good solution for the most part.

Note that readTree does not currently make functions overridable, though it is feasible that it could do that in the future.