Commit graph

34 commits

Author SHA1 Message Date
Vincent Ambo
44a5e14200 feat(nix/readTree): implement .skip-tree marker for subtree ignoring
With this change, readTree gains the ability to notice a `.skip-tree`
marker in addition to the `.skip-subtree` marker.

The behaviour of the new marker will completely ignore the folder that
the marker is located in (i.e. no node will be present for it in the
parent at all).

To make this work, the recursive function in readTree had to be
modified to return a sentinel value (noting that a tree has requested
to be skipped) which is then filtered out when constructing the list
of children. The actual `readTree` function is now a wrapper around
this inner, sentinel-yielding implementation which unwraps the result
set.

For obvious reasons, `.skip-tree` is not allowed at the top-level and
readTree will throw an error if it encounters it there.

Fixes: b/244
Change-Id: Ica731bc1af356e881fd3d31c7109f62ffd2762ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8185
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-28 12:15:33 +00:00
Griffin Smith
b0547ccfa5 feat(nix/readTree): Handle a builtins w/o scopedImport
We (might) not want to implement scopedImport in tvix given it's
considered a bit of a misfeature; this makes readTree work with a
`builtins` set that doesn't have it (and if we decide we do want tvix to
have scopedImport, we can revert this pretty easily).

Change-Id: Ia3bbc847514672063a607d977ce167d489fa1131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6915
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-10 17:42:48 +00:00
Vincent Ambo
8b8c98380e refactor(readTree): deprecate meta.targets for meta.ci.targets
This means that we use the meta.ci attribute more consistently.

The meta.targets attribute is still read, but prints a big, red
warning telling people to migrate to the new one.

Fixes b/176

Change-Id: Ifb4452f529cfc6bbd5018ad7374cac1c83b10045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5238
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-07 15:13:52 +00:00
Vincent Ambo
3bde425860 refactor(nix): Move mkLabel from buildkite to readTree
This function is more generically useful than just for pipeline
construction.

A subsequent commit will use it inside of readTree itself.

Change-Id: I5eabd6f659726484667e060958865dddbc205762
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5237
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-07 15:13:52 +00:00
Vincent Ambo
ba7197c271 fix(nix/readTree): Use overrideAttrs with derivations
Derivations that support overrideAttrs now have their readTree
markers merged in using it, as passthru attributes.

This makes the significant difference that overriding readTree targets
using `overrideAttrs` keeps their readTree data intact.

Change-Id: Ieef635f048781bf4782c1a28532b89a66d9ca24d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5186
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-02 13:36:58 +00:00
Vincent Ambo
aa122cbae7 style: format entire depot with nixpkgs-fmt
This CL can be used to compare the style of nixpkgs-fmt against other
formatters (nixpkgs, alejandra).

Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: cynthia <cynthia@tvl.fyi>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: grfn <grfn@gws.fyi>
2022-01-31 16:11:53 +00:00
Griffin Smith
ca56cb2d9f docs(nix/readTree): Tweak grammar of docs for gather
This seemed to be missing a word previously.

Change-Id: Ifa860051d6b692a626dbaddbaee44b761f2274ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4386
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-16 21:14:15 +00:00
Vincent Ambo
2fa157ccd6 fix(readTree): Apply .skip-subtree to Nix-file children as well
This behaviour was previously confusing, since readTree's data
structure treats children from Nix files and directories as identical
but only one of them would be affected by .skip-subtree

The "subtree" to be skipped here refers to all children of the
structure.

Change-Id: Idf596c9823f09cc2acf49523916bde4b801b8519
2021-12-08 18:22:00 +00:00
Vincent Ambo
4f1249e46f refactor(readTree): Move 'drvTargets' into readTree
This function is also generally useful for readTree consumers that
have the concept of subtargets.

Change-Id: Ic7fc03380dec6953fb288763a28e50ab3624d233
2021-11-23 14:42:08 +00:00
Vincent Ambo
5cad3f7b81 refactor(readTree): Move 'gather' into readTree itself
Discovering CI targets is relevant to all readTree consumers and this
logic is not TVL-specific.

Change-Id: I81ed3d3f76a6c36119f04bee28ca995a013f0e35
2021-11-23 14:42:08 +00:00
Vincent Ambo
a2be05faa4 refactor(readTree): Move copy of 'fix' into readTree
This is often used when bootstrapping a repository with readTree,
before lib is available. Having this definition in readTree is more
convenient than copy&pasting it around to callsites.

Change-Id: I6d5d27ed142bea704843fe289ad2674be8c4d360
2021-11-23 12:02:53 +00:00
Vincent Ambo
bc51bd99d9 refactor(readTree): Move 'restrictFolder' function into readTree
This is generally useful for readTree users and should be part of
readTree itself.

This is a move towards exposing several readTree-related features from
the library itself, in the future also including logic like 'gather'.

Note that this has a small functional change: In error messages of the
function, the notation for accessing Nix attributes is now used rather
than the Perforce-style `//` notation common in TVL.

For example, an error at `//web/tvl/logo` will produce `web.tvl.logo`
in the error message (which corresponds to the readTree attribute
itself).

This makes more sense for non-TVL consumers of readTree, as the
Perforce-style notation is custom to us specifically.

Change-Id: I8e199e473843c40db40b404c20d2c71f48a0f658
2021-11-23 14:39:54 +03:00
Vincent Ambo
ac41d89ba5 refactor(readTree): Flip argument order of argFilter
Since the filters return 'args', this makes nesting of filters more
readable.

Change-Id: I775252460e3e077cc6db2fab6f3948414a95ecbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3873
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-11-15 12:54:32 +00:00
sterni
9dae5acf64 test(nix/readTree): test marking behavior of readTree
This ensures in a simple example that __readTree and __readTreeChildren
are populated correctly.

Change-Id: I69a46b2ddde0d1f9bf0dff1c4780f033ac8fc27a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3655
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-10-05 13:25:13 +00:00
sterni
975f23d11b fix(nix/readTree): generate the correct marker for nix file children
This was a regression introduced in cl/3554.

Change-Id: I0721693a6eb1b28976b28499875812b1c3d1c910
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3654
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-30 16:41:06 +00:00
sterni
d7f60bcb04 feat(nix/readTree): record list of children added by readTree
This change adds a new attribute to readTree nodes, `__readTreeChildren`
which is a list of attribute names added to this node by readTree.

This is then used by `gather` for `ci.targets` to avoid evaluating
attributes unnecessarily. Especially since Nix is not as lazy as we'd
like when determining types (i. e. child ? __readTree needs to force
`child` even when it's not an attribute set), evaluating attributes
unnecessarily is sometimes problematic.

Change-Id: I0a98691d41f987e23ee7e9ba21fbe465da5fe402
2021-09-15 22:37:25 +00:00
sterni
391a186d27 refactor(nix/readTree): remove unused assertMsg function
Change-Id: I4048a59ca3daeb541a7294ce72eebe32982d852c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3553
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-15 12:20:01 +00:00
sterni
5f9c85a1b5 refactor(nix/readTree): use throw instead of assertMsg
This gives a slightly prettier error message and won't leak the error
message when builtins.tryEval is used. Currently an error message from
the tests is always part of the pipeline evaluation log.

Change-Id: I9b488a440368091ed42d707ba4124f592a64bd86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3502
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-09 12:52:16 +00:00
Vincent Ambo
b1f4b530ec feat(readTree): Support scoped import arguments
This makes it possible to override Nix builtins within a readTree
structure. Why would you want to do that, you might ask? Well ...

Change-Id: Icc9cb32e5db4a2eba370cf81769c642d237d4937
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3499
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-09 11:37:06 +00:00
Vincent Ambo
aedde913d1 refactor(readTree): Pass all readTree parameters as function args
Instead of having a mix of depot-passed args (for the filter) and args
to the readTree function itself, make everything a single attribute
set of arguments passed to the function.

This also makes it a bit easier to extend this in the future.

Change-Id: I633c1fc96026d137b451bb604ef92be32571a0f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3498
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-09-08 17:58:53 +00:00
Vincent Ambo
8b851956ad feat(readTree): Add support for path-dependent args filtering
Adds another argument to readTree itself which can be passed when
importing readTree (e.g. in our default.nix) to filter the arguments
passed to a target based on that target's location in the tree.

This is intentionally not yet mentioned in the docs, and also
intentionally implemented in such a way that the API surface of
readTree doesn't change. The reason for this is that I want to figure
out whether these filter functions are actually useful, e.g. within
depot by filtering user-folder passing, and then refactor the readTree
API to find a public way of exposing this as part of the readTree
function itself (and not its import).

Relates to b/143.

Change-Id: I2cdf09f67916527d2337f4bfb578749aeac51a6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3433
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-26 20:33:52 +00:00
Vincent Ambo
a559135970 refactor(readTree): Initialise repo roots without recursing
Plumbs an additional internal argument through readTree that indicates
whether the top-level of a tree is being read, and avoids recursing
into itself in that case. This changes the externally visible
behaviour of readTree (it is now expected to be called a level higher
than previously).

This allows us to reduce the amount of boilerplate needed to bootstrap
the TVL repository (by not having to specify the individual folders
that need to be read).

For reasons related to an infinite recursion we could not (be bothered
to) debug, the top-level `config` key (which held the attribute set
passed on by readTree) has been removed. This is not needed, as it is
already passed on by readTree ...

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Id6e39b57b2f5b3473c4b695a72dd1d01fcfb7a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2961
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-04-12 21:55:07 +00:00
Vincent Ambo
f39b5a8a0d docs(readTree): Add SPDX license header on readTree
Change-Id: I9a6eafa7d4bb3d590dfa35d368adfd25aeed7f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2936
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2021-04-12 13:56:44 +00:00
Profpatsch
83e81def23 feat(nix/readTree): give better error message when not a function
When a file is added to the depot tree that is picked up by read-tree,
but it’s not a function like ({...}: {}), `readTree` will fail on the
function application, leading to a bad error message.

We can do slightly better, by checking the type and throwing a nicer
trace message.

`assertMsg` is copied from `nixpkgs/lib/assert.nix`, since at this
point we don’t have a reference to the lib.

There is another evaluation failure that can happen, which is when the
function we try to call does not have dots; however, nix does not
provide any inflection capabilies for checking whether a function
attrset is open (`builtins.functionArgs` only tells us the attrs it
mentions explicitly). Maybe the locality of the error could be
improved somehow.

Change-Id: Ibe38ce78bb56902075f7c31f2eeeb93485b34be3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2469
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-02-19 23:04:04 +00:00
sterni
2469f0b423 fix(nix/readTree): make nixFileName only match .nix extensions
Nix unfortunately has terrible escaping syntax: If something is an
escape sequence it does not know, like \0, it just swallows the
backslash and returns the second character (byte) as is (yes,
"\0" == "0" is true). This created the following bug in nixFileName
which should have resulted in at least a parse error: "(.*)\.nix" is
the same as "(.*).nix" which meant that nixFileName matched anything
that is at least 4 characters long and ends in "nix". This lead to
readTree creating double attributes when directories are involved or
attributes for non-nix files.

Change-Id: Ibf3be2bd189d48881c82ee795c50151bfb365627
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2535
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-02-18 00:07:14 +00:00
Profpatsch
5da7ee9e8a feat(nix/readTree): add tests of the traversal logic in README
This should closely match the documented behaviour. It might still be
missing some edge cases of course.

Change-Id: I5c75fa045d5f3be8cf5eab787a02644500c14522
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2466
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-01-30 12:06:54 +00:00
Profpatsch
768f5ec301 feat(nix/readTree): add tests of the README example
First step to slowly giving readTree some coverage, so we can do
refactoring without breaking functionality.

Change-Id: If25a8c0fa9c4ac7472c0473372f10a9326cccaf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2465
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-01-30 12:06:54 +00:00
Profpatsch
5f9e9a60e8 chore(nix/readTree): move function into __functor
We are going to export some tests under `nix.readTree.tests`, so in
order to do that and still have `nix.readTree` be a function, let’s
move it to `__functor`.

This requires wiring the `args` and `initPath` arguments through
explicitly.

Change-Id: Ife7956b85d35e59c22174b42dcb7cca83ed868ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2464
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-01-30 12:06:54 +00:00
Vincent Ambo
29613e7494 feat(readTree): Use tree location as the value of __readTree
Change-Id: I6a087279b5d819db475f2179cb77c0f55cbd0ac6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1860
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-31 00:46:35 +00:00
Vincent Ambo
dbda84bac3 fix(readTree): Add __readTree attribute to folders
This is required to automatically walk the tree (see subsequent
commits).

Note: Lisp packages are removed from the CI builds in this commit
because the attrValues of third_party.lisp will contain an element
that is simply `true`, which causes a type error.

These packages are re-added when CI refactoring is complete.

Change-Id: I21e2b719e6c7161c23d2867a216f4daa1c6c8394
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1848
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-26 23:49:32 +00:00
edef
58e3608e25 feat(nix/readTree): allow arguments to depend on tree location
This accepts a function for args, and passes it an attrset containing
the locatedAt parameter.

Change-Id: I3c0f8ca00605e02a787fda88f32b06f5ef9998f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1639
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 22:55:45 +00:00
edef
cec109807b fix(nix/readTree): Ignore hidden files and directories
This skips any directory entries starting with a dot.

Change-Id: I95767f3d35bcb2ed9b3d6e772f3924dd57612711
Reviewed-on: https://cl.tvl.fyi/c/depot/+/123
Reviewed-by: tazjin <mail@tazj.in>
2020-06-13 20:08:05 +00:00
Vincent Ambo
eb650eb8d8 docs(nix/readTree): Add a very descriptive README for readTree 2019-12-21 05:42:52 +00:00
Vincent Ambo
4a9c6ab6a2 refactor(nix/readTree): Move readTree to its own subfolder 2019-12-21 05:42:49 +00:00