tvl-depot/nix/readTree
Griffin Smith b0547ccfa5 feat(nix/readTree): Handle a builtins w/o scopedImport
We (might) not want to implement scopedImport in tvix given it's
considered a bit of a misfeature; this makes readTree work with a
`builtins` set that doesn't have it (and if we decide we do want tvix to
have scopedImport, we can revert this pretty easily).

Change-Id: Ia3bbc847514672063a607d977ce167d489fa1131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6915
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-10 17:42:48 +00:00
..
tests style: format entire depot with nixpkgs-fmt 2022-01-31 16:11:53 +00:00
default.nix feat(nix/readTree): Handle a builtins w/o scopedImport 2022-10-10 17:42:48 +00:00
README.md feat(readTree): Support scoped import arguments 2021-09-09 11:37:06 +00:00

readTree

This is a Nix program that builds up an attribute set tree for a large repository based on the filesystem layout.

It is in fact the tool that lays out the attribute set of this repository.

As an example, consider a root (.) of a repository and a layout such as:

.
├── third_party
│   ├── default.nix
│   └── rustpkgs
│       ├── aho-corasick.nix
│       └── serde.nix
└── tools
    ├── cheddar
    │   └── default.nix
    └── roquefort.nix

When readTree is called on that tree, it will construct an attribute set with this shape:

{
    tools = {
        cheddar = ...;
        roquefort = ...;
    };

    third_party = {
        # the `default.nix` of this folder might have had arbitrary other
        # attributes here, such as this:
        favouriteColour = "orange";

        rustpkgs = {
            aho-corasick = ...;
            serde = ...;
        };
    };
}

Every imported Nix file that yields an attribute set will have a __readTree = true; attribute merged into it.

Traversal logic

readTree will follow any subdirectories of a tree and import all Nix files, with some exceptions:

  • A folder can declare that its children are off-limit by containing a .skip-subtree file. Since the content of the file is not checked, it can be useful to leave a note for a human in the file.
  • If a folder contains a default.nix file, no sibling Nix files will be imported - however children are traversed as normal.
  • If a folder contains a default.nix it is loaded and, if it evaluates to a set, merged with the children. If it evaluates to anything else the children are not traversed.
  • The default.nix of the top-level folder on which readTree is called is not read to avoid infinite recursion (as, presumably, this file is where readTree itself is called).

Traversal is lazy, readTree will only build up the tree as requested. This currently has the downside that directories with no importable files end up in the tree as empty nodes ({}).

Import structure

readTree is called with an argument set containing a few parameters:

  • path: Initial path at which to start the traversal.
  • args: Arguments to pass to all imports.
  • filter: (optional) A function to filter the argument set on each import based on the location in the tree. This can be used to, for example, implement a "visibility" system inside of a tree.
  • scopedArgs: (optional) An argument set that is passed to all imported files via builtins.scopedImport. This will forcefully override the given values in the import scope, use with care!

The package headers in this repository follow the form { pkgs, ... }: where pkgs is a fixed-point of the entire package tree (see the default.nix at the root of the depot).

In theory readTree can pass arguments of different shapes, but I have found this to be a good solution for the most part.

Note that readTree does not currently make functions overridable, though it is feasible that it could do that in the future.