mapAttrs, map and genList call Nix functions provided by the caller and
store the result of applying them in a Nix data structure that does not
force all of its contents when forced itself. This means that when such
a builtin application is forced, the Nix function calls performed by the
builtin should not be forced: They may be forced later, but it is also
possible that they will never be forced, e.g. in
builtins.length (builtins.map (builtins.add 2) [ 1 2 3 ])
it is not necessary to compute a single application of builtins.add.
Since request_call_with immediately performs the function call
requested, Tvix would compute function applications unnecessarily before
this change. Because this was not followed by a request_force, the
impact of this was relatively low in Nix code (most functions return a
new thunk after being applied), but it was enough to cause a lot of
bogus builtins.trace applications when evaluating anything from
`lib.modules`. The newly added test includes many cases where Tvix
previously incorrectly applied a builtin, breaking a working expression.
To fix this we add a new helper to construct a Thunk performing a
function application at runtime from a function and argument given as
`Value`s. This mimics the compiler's compile_apply(), but does itself
not require a compiler, since the necessary Lambda can be constructed
independently.
I also looked into other builtins that call a Nix function to verify
that they don't exhibit such a problem:
- Many builtins immediately use the resulting value in a way that makes
it necessary to compute all the function calls they do as soon as
the outer builtin application is forced:
* all
* any
* filter
* groupBy
* partition
- concatMap needs to (shallowly) force the returned list for
concatenation.
- foldl' is strict in the application of `op` (I added a comment that
makes this explicit).
- genericClosure needs to (shallowly) force the resulting list and some
keys of the attribute sets inside.
Resolves b/272.
Change-Id: I1fa53f744bcffc035da84c1f97ed25d146830446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8651
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Instead of instantiating it once in every loop iteration, put it in an
Arc, and clone that before passing it to the spawned task.
Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This feature allows the compiler to detect situations where the
created thunk is useless and can be merged into the parent chunk
instead.
The only case where the compiler does this initially is when
statically optimising a select expression.
For example, previously the expression `builtins.length` compiled into
two thunks:
1. An "inner" thunk which contained an `OpConstant` that had the
optimised `length` builtin in it.
2. An "outer" thunk which contained an `OpConstant` to access the
inner thunk, and the trailing OpForce of the top-level program.
With this change, the inner thunk is skipped completely and the outer
chunk directly contains the `length` builtin access.
This can be applied in several situations, some easier than others,
and we will add them in as we go along.
Change-Id: Ie44521445fce1199f99b5b17712833faea9bc357
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7959
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This method extends the contents of one chunk with that of another,
effectively merging the thunks together.
This will be used for the upcoming "unthunking" functionality.
Change-Id: I6ad74232cd7f3eca198ed921e455205e00d76e6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7958
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
With traverse_to not requiring a &mut anymore, we can drop the &mut self
in all these function signatures.
Change-Id: I22105376b625cb281c39e92d3206df8a6ce97a88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This operation is blocking, so it should be run inside a blocking tokio
task. Tokio panics if it detects a blocking operation inside a non-
blocking task, so cl/8619 would cause it to panic (as the GRPC clients
use spawn_blocking under the hood).
As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614),
we create a new instance of TvixStoreIO inside each loop iteration.
Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
this is slow and often overloaded, but it's kind of cool when it
works. this translation method deals much better with the kind of
slang you'd see in telegram posts than any other method.
Change-Id: I7e4c845eb382f0eac627c4237b492c8e40dae574
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8625
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This switches tvix-cli over from using `NixCompatIO` to `TvixStoreIO`.
For now, we simply instantiate in-memory services instead of getting
fancy with constructors inside tvix-store, but long-term, we might want
to support some URI syntax, to make this configurable at runtime.
nixpkgs eval tests might be fine (and fast!) with a purely in-memory
backend, but other usages might involve talking to a local tvix-store
over gRPC (using the gRPC client, either unix domain socket or even
further away remote), or running tvix-store in "embedded" mode (using
another client than the gRPC client).
Change-Id: I509afd3dc5ce3f2d52b0fb7067748fab820e26ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8572
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows closing a TvixStoreIO if it contains a SledPathInfoService.
Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We already evaluate this the same as Nix does, so worth adding it into
CI.
Change-Id: I529faccac6e5e16e0bc985ab4c3e0cd07bb23293
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8624
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This introduces a function that can be instantiated with an attribute
path to instantiate, as well as the expected path (normally interpolated
with the nix evaluator).
Check both pkgs.stdenv.drvPath and pkgs.stdenv.outPath to match.
Change-Id: Id667ed35fa159ff83fedb3017ef8d3271aa42695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8606
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We have code.namespace as a field, set to
tvix_store::directoryservice::sled, so there's no need to repeat the
name here.
Change-Id: Ic1aa8a2b24de439c6a189966bd773e9acf49d1e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8623
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This asks a remote tvix-store for the nar size and digest of a given
root node.
Change-Id: If9f916d9bfc5f8dc3166e2c6c1671c0f0124d1c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8611
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Spell out that “may diverge” is more of a “has diverged by now”. We are
essentially maintaining a fork of mime4cl.
Change-Id: I9049e8296a666c3d1b08eae28813147f360771ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8621
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This introduces a new struct, B3Digest, which internally holds a
Vec<u8>, but only allows construction with 32 bytes.
It also implements display, which will print the base64 representation.
This should reduce some boilerplate when parsing Vec<u8>.
Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
These two places didn't add the path from the context to the
ErrorKind, but simply relied on the
impl From<std::io::Error> for tvix_eval::ErrorKind, which doesn't add
the path.
Change-Id: Ifc7dbbe305d24242b0705de1dea34e8923e9d2cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8603
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
We didn't return anything useful other than ErrorKind::IO anyways.
We can use io::ErrorKind::Unsupported for DummyIO.
Fixes b/271.
Change-Id: Icb231e9b38168e8b6fa473bfa405d160357b317f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8602
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This makes publishing a bit easier without the build script
interfering and other wonkiness.
Change-Id: Iadb144aabbdeabae8899ebdc62636315239e5f08
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8601
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
One test in here assumed the server was fast, but when very busy, a
computer running these tests might not be fast.
Treat both cases that can occur separately.
Change-Id: Iba168ad39f2458c4fb8873211df33beeaff7c6c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8595
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Also, get rid of the explicit byte comparison here, but unwrap like in
the rest of the codebase. We'll deal with this in a generic manner in
b/267 and b/189.
Change-Id: Ie5f3d27ab35b7e88d67a2796c29cdd7bc7df71f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8589
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This remained around from some previous time where this type was being
cloned around directly, but it's not anymore.
Change-Id: I2c61cf9cecb8e5d7d08644ed20642ee61357bc21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8580
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a wrapper type TvixIO<T: EvalIO>, which can wrap around an
arbitrary EvalIO implementation and perform actions needed for the
Tvix CLI (marking imported paths as known, and handling __corepkgs__).
Change-Id: I5fc1ca199b9f94b21a89103b84575e0f8f58dff9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8579
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
