With this change, readTree gains the ability to notice a `.skip-tree`
marker in addition to the `.skip-subtree` marker.
The behaviour of the new marker will completely ignore the folder that
the marker is located in (i.e. no node will be present for it in the
parent at all).
To make this work, the recursive function in readTree had to be
modified to return a sentinel value (noting that a tree has requested
to be skipped) which is then filtered out when constructing the list
of children. The actual `readTree` function is now a wrapper around
this inner, sentinel-yielding implementation which unwraps the result
set.
For obvious reasons, `.skip-tree` is not allowed at the top-level and
readTree will throw an error if it encounters it there.
Fixes: b/244
Change-Id: Ica731bc1af356e881fd3d31c7109f62ffd2762ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8185
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We (might) not want to implement scopedImport in tvix given it's
considered a bit of a misfeature; this makes readTree work with a
`builtins` set that doesn't have it (and if we decide we do want tvix to
have scopedImport, we can revert this pretty easily).
Change-Id: Ia3bbc847514672063a607d977ce167d489fa1131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6915
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This means that we use the meta.ci attribute more consistently.
The meta.targets attribute is still read, but prints a big, red
warning telling people to migrate to the new one.
Fixes b/176
Change-Id: Ifb4452f529cfc6bbd5018ad7374cac1c83b10045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5238
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This function is more generically useful than just for pipeline
construction.
A subsequent commit will use it inside of readTree itself.
Change-Id: I5eabd6f659726484667e060958865dddbc205762
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5237
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Derivations that support overrideAttrs now have their readTree
markers merged in using it, as passthru attributes.
This makes the significant difference that overriding readTree targets
using `overrideAttrs` keeps their readTree data intact.
Change-Id: Ieef635f048781bf4782c1a28532b89a66d9ca24d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5186
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
This seemed to be missing a word previously.
Change-Id: Ifa860051d6b692a626dbaddbaee44b761f2274ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4386
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This behaviour was previously confusing, since readTree's data
structure treats children from Nix files and directories as identical
but only one of them would be affected by .skip-subtree
The "subtree" to be skipped here refers to all children of the
structure.
Change-Id: Idf596c9823f09cc2acf49523916bde4b801b8519
This function is also generally useful for readTree consumers that
have the concept of subtargets.
Change-Id: Ic7fc03380dec6953fb288763a28e50ab3624d233
This is often used when bootstrapping a repository with readTree,
before lib is available. Having this definition in readTree is more
convenient than copy&pasting it around to callsites.
Change-Id: I6d5d27ed142bea704843fe289ad2674be8c4d360
This is generally useful for readTree users and should be part of
readTree itself.
This is a move towards exposing several readTree-related features from
the library itself, in the future also including logic like 'gather'.
Note that this has a small functional change: In error messages of the
function, the notation for accessing Nix attributes is now used rather
than the Perforce-style `//` notation common in TVL.
For example, an error at `//web/tvl/logo` will produce `web.tvl.logo`
in the error message (which corresponds to the readTree attribute
itself).
This makes more sense for non-TVL consumers of readTree, as the
Perforce-style notation is custom to us specifically.
Change-Id: I8e199e473843c40db40b404c20d2c71f48a0f658
Since the filters return 'args', this makes nesting of filters more
readable.
Change-Id: I775252460e3e077cc6db2fab6f3948414a95ecbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3873
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This ensures in a simple example that __readTree and __readTreeChildren
are populated correctly.
Change-Id: I69a46b2ddde0d1f9bf0dff1c4780f033ac8fc27a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3655
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This was a regression introduced in cl/3554.
Change-Id: I0721693a6eb1b28976b28499875812b1c3d1c910
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3654
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This change adds a new attribute to readTree nodes, `__readTreeChildren`
which is a list of attribute names added to this node by readTree.
This is then used by `gather` for `ci.targets` to avoid evaluating
attributes unnecessarily. Especially since Nix is not as lazy as we'd
like when determining types (i. e. child ? __readTree needs to force
`child` even when it's not an attribute set), evaluating attributes
unnecessarily is sometimes problematic.
Change-Id: I0a98691d41f987e23ee7e9ba21fbe465da5fe402
This gives a slightly prettier error message and won't leak the error
message when builtins.tryEval is used. Currently an error message from
the tests is always part of the pipeline evaluation log.
Change-Id: I9b488a440368091ed42d707ba4124f592a64bd86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3502
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This makes it possible to override Nix builtins within a readTree
structure. Why would you want to do that, you might ask? Well ...
Change-Id: Icc9cb32e5db4a2eba370cf81769c642d237d4937
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3499
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Instead of having a mix of depot-passed args (for the filter) and args
to the readTree function itself, make everything a single attribute
set of arguments passed to the function.
This also makes it a bit easier to extend this in the future.
Change-Id: I633c1fc96026d137b451bb604ef92be32571a0f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3498
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Adds another argument to readTree itself which can be passed when
importing readTree (e.g. in our default.nix) to filter the arguments
passed to a target based on that target's location in the tree.
This is intentionally not yet mentioned in the docs, and also
intentionally implemented in such a way that the API surface of
readTree doesn't change. The reason for this is that I want to figure
out whether these filter functions are actually useful, e.g. within
depot by filtering user-folder passing, and then refactor the readTree
API to find a public way of exposing this as part of the readTree
function itself (and not its import).
Relates to b/143.
Change-Id: I2cdf09f67916527d2337f4bfb578749aeac51a6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3433
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Plumbs an additional internal argument through readTree that indicates
whether the top-level of a tree is being read, and avoids recursing
into itself in that case. This changes the externally visible
behaviour of readTree (it is now expected to be called a level higher
than previously).
This allows us to reduce the amount of boilerplate needed to bootstrap
the TVL repository (by not having to specify the individual folders
that need to be read).
For reasons related to an infinite recursion we could not (be bothered
to) debug, the top-level `config` key (which held the attribute set
passed on by readTree) has been removed. This is not needed, as it is
already passed on by readTree ...
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Id6e39b57b2f5b3473c4b695a72dd1d01fcfb7a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2961
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
When a file is added to the depot tree that is picked up by read-tree,
but it’s not a function like ({...}: {}), `readTree` will fail on the
function application, leading to a bad error message.
We can do slightly better, by checking the type and throwing a nicer
trace message.
`assertMsg` is copied from `nixpkgs/lib/assert.nix`, since at this
point we don’t have a reference to the lib.
There is another evaluation failure that can happen, which is when the
function we try to call does not have dots; however, nix does not
provide any inflection capabilies for checking whether a function
attrset is open (`builtins.functionArgs` only tells us the attrs it
mentions explicitly). Maybe the locality of the error could be
improved somehow.
Change-Id: Ibe38ce78bb56902075f7c31f2eeeb93485b34be3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2469
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Nix unfortunately has terrible escaping syntax: If something is an
escape sequence it does not know, like \0, it just swallows the
backslash and returns the second character (byte) as is (yes,
"\0" == "0" is true). This created the following bug in nixFileName
which should have resulted in at least a parse error: "(.*)\.nix" is
the same as "(.*).nix" which meant that nixFileName matched anything
that is at least 4 characters long and ends in "nix". This lead to
readTree creating double attributes when directories are involved or
attributes for non-nix files.
Change-Id: Ibf3be2bd189d48881c82ee795c50151bfb365627
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2535
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
This should closely match the documented behaviour. It might still be
missing some edge cases of course.
Change-Id: I5c75fa045d5f3be8cf5eab787a02644500c14522
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2466
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
First step to slowly giving readTree some coverage, so we can do
refactoring without breaking functionality.
Change-Id: If25a8c0fa9c4ac7472c0473372f10a9326cccaf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2465
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
We are going to export some tests under `nix.readTree.tests`, so in
order to do that and still have `nix.readTree` be a function, let’s
move it to `__functor`.
This requires wiring the `args` and `initPath` arguments through
explicitly.
Change-Id: Ife7956b85d35e59c22174b42dcb7cca83ed868ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2464
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
This is required to automatically walk the tree (see subsequent
commits).
Note: Lisp packages are removed from the CI builds in this commit
because the attrValues of third_party.lisp will contain an element
that is simply `true`, which causes a type error.
These packages are re-added when CI refactoring is complete.
Change-Id: I21e2b719e6c7161c23d2867a216f4daa1c6c8394
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1848
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This accepts a function for args, and passes it an attrset containing
the locatedAt parameter.
Change-Id: I3c0f8ca00605e02a787fda88f32b06f5ef9998f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1639
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This skips any directory entries starting with a dot.
Change-Id: I95767f3d35bcb2ed9b3d6e772f3924dd57612711
Reviewed-on: https://cl.tvl.fyi/c/depot/+/123
Reviewed-by: tazjin <mail@tazj.in>
