...in `Derivation`.
This is more type-safe and should consume less memory.
This also removes some allocations in the potentially hot path of output hash calculation.
https: //b.tvl.fyi/issues/264
Change-Id: I6ad7d3cb868dc9f750894d449a6065608ef06e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
...so that we can also use `StorePath`s in
derivation.input_derivations.
Towards https://b.tvl.fyi/issues/264
Change-Id: I71d296ca273979c70f277a7f4f88a5f76de3d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10973
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
One last place where we needed to wrap the forcing of the element of a
list in `try_value!`. This fixes a previously `notyetpassing` test
Change-Id: I8827a3e39630e6959013b70bdfa9cbcb93f4e91c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10789
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
We add a new set of builtins called `import_builtins`, which
will contain import-related builtins, such as `builtins.path` and
`builtins.filterSource`. Both can import paths into the store, with
various knobs to alter the result, e.g. filtering, renaming, expected
hashes.
We introduce `filtered_ingest` which will drive the filtered ingestion
via the Nix function via the generator machinery, and then we register
the root node to the path info service inside the store.
`builtins.filterSource` is very simple, `builtins.path` is a more
complicated model requiring the same logic albeit more sophisticated
with name customization, file ingestion method and expected SHA-256.
Change-Id: I1083f37808b35f7b37818c8ffb9543d9682b2de2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10654
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
(Re-)Adds an error variant that wraps a catchable error kind, which is
used for returning the result of an evaluation.
Previously this would return the internal catchable value, which would
lead to panics if users tried to use these. Somehow this was missed; I
think we need error output tests.
Change-Id: Id6e24aa2ce4ea4358a29b2e1cf4a6749986baf8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10991
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
All these functions have an Rc of Rc<TvixStoreIO> or Rc<dyn StoreIO>, so
we can call io.clone() directly.
Change-Id: I96a5bcb9af4aca93892cb72bcfaf14540da52381
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10989
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This displays how users can configure an impure evaluation for
tvix-serde, which makes it possible to use e.g. `nixpkgs/lib`.
We might want to add an example showing how the full Nix-glue
compatibility stuff can be added here, too.
Change-Id: I2224a3fc66e739969d4c723c3d9d8127a046b6fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10994
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
The previous behaviour (enabling `import`, but not allowing e.g.
`<nixpkgs/lib>` to resolve) was very confusing.
Now imports from NIX_PATH become enabled by default, unless the user
already overrode that behaviour with something else by setting
`Evaluation::nix_path` manually.
Change-Id: Iad970beb633d9887be4b185b01e6f5858d81bea3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10993
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
There's some code path where fancy_format_str()->Error::Display, which
recurses forever and overflows the stack. This was introduced in a
previous commit today.
Change-Id: I87a59492099f6c138c752478901b9aa614bb57cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10990
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Makes use of the SourceCode field now being stored directly in
errors (see parent CL). With this change, the default `Display`
implementation can now format errors correctly, and there is no need
to keep a `SourceCode` around just for error formatting.
Updates dependent crates (CLI, serde, tvixbolt) to use this correctly.
Change-Id: Iddc5d7a6b4bab391f30a999e4c68aca34304c059
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10987
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
With this change it's no longer necessary to track the SourceCode
struct separately from the evaluation for error reporting: It's just
stored directly in the errors.
This also ends up resolving an issue in compiler::bindings, where we
cloned the Arc containing file references way too often. In fact those
clones probably compensate for all additional SourceCode clones during
error construction now.
Change-Id: Ice93bf161e61f8ea3d48103435e20c53e6aa8c3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10986
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Some implementations of DirectoryService might not allow retrieval of
intermediate Directory nodes, that are not at the "root".
Think about an object store implementation. The client is doing a
get_recursive anyways to reduce the number of roundtrips.
By documenting the fact we don't need to support looking up intermediate
Directory messages, we can just batch all directories into the same
object, keyed by the root.
Change-Id: I019d720186d03c4125cec9191e93d20586a20963
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10988
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This makes some of the work of configuring an arbitrary I/O handler
easier.
Change-Id: I158db3235fe83df6e709578ed515e0e028c20086
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10959
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Rather than debug-formatting a Vec<u8>, which gives just a sequence of
numbers (not very helpful!) use String::from_utf8_lossy to give a much
more human-readable error message
Change-Id: I6cae795ebe2ab9a656c087b82a143411e6c7c81a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10980
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Initialize a new empty builtins module `fetcher_builtins`, which will
contain the builtins which fetch URLs from the internet:
* fetchurl
* fetchGit
* fetchTarball
* fetchTree (maybe? this is experimental)
These builtins are all implemented in CPP nix at:
https://github.com/NixOS/nix/blob/2.20.2/src/libexpr/primops/fetchTree.cc
These builtins are added to the evaluation context using a similar
mechanism to the derivation builtins, and have been added everywhere
derivation builtins were previously being added.
Change-Id: I133b91cc9560f23028621414537f712e7bd8a825
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10974
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
...so that we can parse (and at the same time
validate) store paths directly to `StorePath`s.
https: //b.tvl.fyi/issues/264
Change-Id: Ide4cb5403f8e3400f7d2a5838c652ac35dbd1a4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10968
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
That's one possible abstraction to drive builds.
Whenever IO into a store path is requested, we look up the root node,
and in case we don't have it in PathInfoService, but KnownPaths gives us
a Derivation for that output path, trigger a build and await the result.
This recursively might trigger builds for parent paths if they haven't
been built yet.
Another option would be to simply expose a PathInfoService interface for
a builder too, and loop all building into IO via PathInfoService
composition - but let's start with something.
Note tvix-cli doesn't have a configurable BuildService yet, it's plugged
to the DummyBuildService, so whenever it needs to do a build, it'll fail,
but that's how it can be provoked:
```
(builtins.readFile (import <nixpkgs> {}).hello.outPath + "/bin/hello")
[…]
error[E029]: I/O error: /nix/store/cg8a576pz2yfc1wbhxm1zy4x7lrk8pix-hello-2.12.1: builds are not supported with DummyBuildService
--> [code]:1:2
|
1 | (builtins.readFile (import <nixpkgs> {}).hello.outPath + "/bin/hello")
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```
Note how this fails, while pure output path calculation
(`(import <nixpkgs> {}).hello.outPath + "/bin/hello")`) still succeeds.
Change-Id: Id2075d8a2b18554d0dd608b4b29146a8cd411e7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10793
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Log levels can be statically disabled at compile time via Cargo
features.
Do this for everything more verbose than info in release mode.
Change-Id: I57d4282b4e19a8d44f68da2f15c7534dd3e08977
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10955
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This otherwise gets a bit spammy.
Change-Id: I288350a600d79a394c239f253424ad55bc3cefc5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10954
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This causes a bit too much spam otherwise.
Change-Id: If3ced9ddfee7f49453711cd26469d1eb81983c71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Necessary, if we want to use it inside of `Derivation` etc.
Change-Id: I8888060417b2ee83ac52d7ec3e7b27c393271d8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10947
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
No otlp yet, this simply gives us structured log output and a cli
argument for the log level.
Change-Id: Ifaa60bae419640e92baebb6ee59eedd775c769c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10853
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Print store paths with their ToString implementation for brevity, and
don't log the sucessful return value of read_to_string.
Change-Id: I01b6838398acd66b8818095622f361fcca26fa77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10854
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Passing a StorePathRef is annoying if we only (already) have a
StorePath.
Change-Id: Ic3b36c0041707230515a6745a57f0d25b2bafd16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10948
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Instead of constructing a completely new derivation for hashing,
allow to call hashing with substituted input_derivations.
This
1. reduces the number of allocations,
2. prepares substituting `String`s for store paths with proper `StorePath`s without needing a separate derivation struct that allows hashes without `/nix/store` in input_derivations, and
3. keeps the change local to the `Derivation` implementation.
Change-Id: I36732c78f98fc59f0925b65823773222782017b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10935
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
This will be used inside glue, to populate the CA field.
Change-Id: I52350d66ad09c90c4e9f3bc6ea44fba4427f1576
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10946
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
We need to not only store a map from drv path to hash derivation modulo,
but also keep the original Derivation struct - because we'll use that
later to trigger builds.
Change-Id: I78e2e8431ab5ae853188866b797a79025200de98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10790
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Allow using a BUILD_SERVICE_ADDR env var, or cmdline args to configure
it. Still default to the dummy implementation.
Change-Id: I68f34f7b09eabef2b0491103857bbc798398ebfc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10846
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Populate the input_derivations struct field in-place, using
self.input_derivations.from_iter, which seems a bit more readable.
Change-Id: I4aa12cfd66e08486ef27aedab1bb731d94b63657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10934
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is was dropped accidentially, but we actually want to run tests in
debug mode, to check for overflows.
Change-Id: Ic0cf2d4a263bfd71f3453ed335d8be9dacd945e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10932
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Now that I've done a ton of things to make sure Value is small on the
stack (16 bytes, which is a perfectly reasonable size for a programming
language Value enum), add a test asserting it stays that way.
These size improvements have a measurable impact, too - here's the
`hello outpath` benchmark compared between canon (as of r/7495) and this
commit:
hello outpath time: [990.56 ms 995.83 ms 1.0070 s]
change: [-7.1397% -6.1302% -5.1651%] (p = 0.00 < 0.05)
Performance has improved.
Change-Id: If99a0976eab28eb5e516fcd2f4a0e068145af23e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10799
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
This is now the only enum variant for Value that is larger than 8
bytes (it's 16 bytes), so boxing it (especially since it's not
perf-critical) allows us to get the Value size down to only 16 bytes!
Change-Id: I98598e2b762944448bef982e8ff7da6d6683c4aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10798
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: aspen <root@gws.fyi>
This reverts commit d3d41552cf.
This was well-intentioned, but now the boxed Path values are actually
the *largest* Value enum variants, at 16 bytes (because they're
fat-pointers, with a len) instead of 8 bytes like all the other values.
Having the double reference is a reasonable price to pay (it seems; more
benchmarks may end up disagreeing) for a smaller Value repr.
Change-Id: I0d3e84f646c8f5ffd0b7259c4e456637eea360f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10797
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Storing a full BString here incurs the extra overhead of the capacity
for the inner byte-vector, which we basically never use as Nix strings
are immutable (and we don't do any mutation / sharing analysis).
Switching to a Box<BStr> cuts us from 72 bytes to 64 bytes per
string (and there are a lot of strings!)
Change-Id: I11f34c14a08fa02759f260b1c78b2a2b981714e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10794
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
These strings are allocated once and never changed, so they don't need
the additional overhead of a capacity given by String - instead, we can
use Box<str> and save on 16 bytes for each of these, *and* for each
Value since this is currently the largest Value variant.
Change-Id: I3e5cb070fe6c5bf82114c92d04f6bae775663a7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10796
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
serde_json::Value is pretty large, and is contributing (albeit not
exclusively) to the large size of the Value repr. Putting it in a box
is *especially* cheap (since it's rarely used) and allows us
to (eventually) cut down on the size of Value.
Change-Id: I005a802d8527b639beb4e938e3320b11ffa1ef23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10795
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
