tvl-depot/nix/readTree
sterni d7f60bcb04 feat(nix/readTree): record list of children added by readTree
This change adds a new attribute to readTree nodes, `__readTreeChildren`
which is a list of attribute names added to this node by readTree.

This is then used by `gather` for `ci.targets` to avoid evaluating
attributes unnecessarily. Especially since Nix is not as lazy as we'd
like when determining types (i. e. child ? __readTree needs to force
`child` even when it's not an attribute set), evaluating attributes
unnecessarily is sometimes problematic.

Change-Id: I0a98691d41f987e23ee7e9ba21fbe465da5fe402
2021-09-15 22:37:25 +00:00
..
tests refactor(readTree): Pass all readTree parameters as function args 2021-09-08 17:58:53 +00:00
default.nix feat(nix/readTree): record list of children added by readTree 2021-09-15 22:37:25 +00:00
README.md feat(readTree): Support scoped import arguments 2021-09-09 11:37:06 +00:00

readTree

This is a Nix program that builds up an attribute set tree for a large repository based on the filesystem layout.

It is in fact the tool that lays out the attribute set of this repository.

As an example, consider a root (.) of a repository and a layout such as:

.
├── third_party
│   ├── default.nix
│   └── rustpkgs
│       ├── aho-corasick.nix
│       └── serde.nix
└── tools
    ├── cheddar
    │   └── default.nix
    └── roquefort.nix

When readTree is called on that tree, it will construct an attribute set with this shape:

{
    tools = {
        cheddar = ...;
        roquefort = ...;
    };

    third_party = {
        # the `default.nix` of this folder might have had arbitrary other
        # attributes here, such as this:
        favouriteColour = "orange";

        rustpkgs = {
            aho-corasick = ...;
            serde = ...;
        };
    };
}

Every imported Nix file that yields an attribute set will have a __readTree = true; attribute merged into it.

Traversal logic

readTree will follow any subdirectories of a tree and import all Nix files, with some exceptions:

  • A folder can declare that its children are off-limit by containing a .skip-subtree file. Since the content of the file is not checked, it can be useful to leave a note for a human in the file.
  • If a folder contains a default.nix file, no sibling Nix files will be imported - however children are traversed as normal.
  • If a folder contains a default.nix it is loaded and, if it evaluates to a set, merged with the children. If it evaluates to anything else the children are not traversed.
  • The default.nix of the top-level folder on which readTree is called is not read to avoid infinite recursion (as, presumably, this file is where readTree itself is called).

Traversal is lazy, readTree will only build up the tree as requested. This currently has the downside that directories with no importable files end up in the tree as empty nodes ({}).

Import structure

readTree is called with an argument set containing a few parameters:

  • path: Initial path at which to start the traversal.
  • args: Arguments to pass to all imports.
  • filter: (optional) A function to filter the argument set on each import based on the location in the tree. This can be used to, for example, implement a "visibility" system inside of a tree.
  • scopedArgs: (optional) An argument set that is passed to all imported files via builtins.scopedImport. This will forcefully override the given values in the import scope, use with care!

The package headers in this repository follow the form { pkgs, ... }: where pkgs is a fixed-point of the entire package tree (see the default.nix at the root of the depot).

In theory readTree can pass arguments of different shapes, but I have found this to be a good solution for the most part.

Note that readTree does not currently make functions overridable, though it is feasible that it could do that in the future.