Commit graph

181 commits

Author SHA1 Message Date
Linus Heckemann
a3dbb60213 fix(tvix/eval): off-by-one in replaceStrings
replaceStrings would previously fail to replace the last character
in a string.

Change-Id: I43a7c960945350b2e7a5b731b7fdb617723eb38f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9151
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-24 10:07:48 +00:00
Florian Klink
def773d4d5 refactor(tvix/eval): cargo clippy (len() is usize)
Change-Id: I9d931ffcc03c6df7c0392dbc1c9a4ae0e3804213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9099
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 21:51:04 +00:00
Florian Klink
432222f098 feat(tvix/store/proto): use Bytes instead of Vec<u8>
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our
bytes field cheaper to clone.

It's a bit annoying to configure due to
https://github.com/hyperium/tonic/issues/908, but the workaround does
get the job done.

Change-Id: I25714600b041bb5432d3adf5859b151e72b12778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-22 09:03:02 +00:00
Florian Klink
72e82ffcb1 refactor(tvix/store): use bytes for node names and symlink targets
Some paths might use names that are not valid UTF-8. We should be able
to represent them.

We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.

While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?

Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-21 19:01:49 +00:00
Evgeny Zemtsov
c8fcdca4eb feat(tvix/eval): allow extending builtins outside of tvix_eval
The change allows applications that use tvix_serde for parsing
nix-based configuration to extend the language with domain-specific
set of features.

Change-Id: Ia86612308a167c456ecf03e93fe0fbae55b876a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8848
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-22 20:19:06 +00:00
sterni
4516cd09c5 fix(tvix/eval): only finalise formal arguments if defaulting
When dealing with a formal argument in a function argument pattern that
has a default expression, there are two different things that can happen
at runtime: Either we select its value from the passed attribute
successfully or we need to use the default expression. Both of these may
be thunks and both of these may need finalisers. However, in the former
case this is taken care of elsewhere, the value will always be finalised
already if necessary. In the latter case we may need to finalise the
thunk resulting from the default expression. However, the thunk
corresponding to the expression may never end up in the local's stack
slot. Since finalisation goes by stack slot (and not constants), we need
to prevent a case where we don't fall back to the default expression,
but finalise anyways.

Previously, we worked around this by making `OpFinalise` ignore
non-thunks. Since finalisation of already evaluated thunks still
crashed, the faulty compilation of function pattern arguments could
still cause a crash.

As a new approach, we reinstate the old behavior of `OpFinalise` to
crash whenever encountering something that is either not a thunk or
doesn't need finalisation. This can also help catching (similar)
miscompilations in the future. To then prevent the crash, we need to
track whether we have fallen back or not at runtime. This is done using
an additional phantom on the stack that holds a new `FinaliseRequest`
value. When it comes to finalisation we check this value and
conditionally execute `OpFinalise` based on its value.

Resolves b/261 and b/265 (partially).

Change-Id: Ic04fb80ec671a2ba11fa645090769c335fb7f58b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8705
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-20 10:07:44 +00:00
Linus Heckemann
5733274876 fix(tvix/eval): allow negative substring lengths
Nix uses string::substr without checking the sign of the length[1].
The NixOS testing infrastructure relies on this[2], and on the
implicit conversion of that to the maximum possible value for a
size_t.

[1]: ecae62020b/src/libexpr/primops.cc (L3597)
[2]: c7c2984716/nixos/lib/testing/driver.nix (L29)

Change-Id: I6d0caf6830b6bda3fdf44c40c81de6a1befeca7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8746
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-12 08:38:49 +00:00
sterni
3b33c19a9c fix(tvix): don't call function eagerly in genList, map & mapAttrs
mapAttrs, map and genList call Nix functions provided by the caller and
store the result of applying them in a Nix data structure that does not
force all of its contents when forced itself. This means that when such
a builtin application is forced, the Nix function calls performed by the
builtin should not be forced: They may be forced later, but it is also
possible that they will never be forced, e.g. in

    builtins.length (builtins.map (builtins.add 2) [ 1 2 3 ])

it is not necessary to compute a single application of builtins.add.

Since request_call_with immediately performs the function call
requested, Tvix would compute function applications unnecessarily before
this change. Because this was not followed by a request_force, the
impact of this was relatively low in Nix code (most functions return a
new thunk after being applied), but it was enough to cause a lot of
bogus builtins.trace applications when evaluating anything from
`lib.modules`. The newly added test includes many cases where Tvix
previously incorrectly applied a builtin, breaking a working expression.

To fix this we add a new helper to construct a Thunk performing a
function application at runtime from a function and argument given as
`Value`s. This mimics the compiler's compile_apply(), but does itself
not require a compiler, since the necessary Lambda can be constructed
independently.

I also looked into other builtins that call a Nix function to verify
that they don't exhibit such a problem:

- Many builtins immediately use the resulting value in a way that makes
  it necessary to compute all the function calls they do as soon as
  the outer builtin application is forced:

  * all
  * any
  * filter
  * groupBy
  * partition

- concatMap needs to (shallowly) force the returned list for
  concatenation.

- foldl' is strict in the application of `op` (I added a comment that
  makes this explicit).

- genericClosure needs to (shallowly) force the resulting list and some
  keys of the attribute sets inside.

Resolves b/272.

Change-Id: I1fa53f744bcffc035da84c1f97ed25d146830446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8651
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-26 22:35:39 +00:00
Vincent Ambo
969fbcd6d4 fix(tvix/eval): builtins.trace prints to stderr
Change-Id: Icf577396035474d6977e627058aba5805c61985e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8563
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-05-12 12:50:59 +00:00
Vincent Ambo
b78ae941a4 fix(tvix/eval): use coerce_to_string in builtins.substring
This actually uses coercion under the hood in C++ Nix. See the test
for an example.

Change-Id: Id56b364acf269225b6829d0b600e0222f8b3608d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8322
Reviewed-by: andi <andi@notmuch.email>
Tested-by: BuildkiteCI
2023-03-17 19:17:29 +00:00
Vincent Ambo
b5f5a1595d chore(tvix/eval): remove some dead code
This was commented out and forgotten during the generator refactor, oh
well.

Change-Id: I474b685159a955a846db462da0dd0067af177b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8321
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-17 11:01:07 +00:00
Vincent Ambo
5d9bfd7735 fix(tvix/eval): emit warnings from builtins.import again
Wires up generator logic to emit warnings that already have spans
attached again.

Change-Id: I9f878cec3b9d4f6f7819e7c71bab7ae70bd3f08b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8224
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-13 20:30:59 +00:00
Vincent Ambo
939cebd0f1 fix(tvix/eval): implement cppnix JSON-serialisation semantics
This drops the usage of serde::Serialize, as the trait can not be used
to implement the correct semantics (function colouring!).

Instead, a manual JSON serialisation function is written which
correctly handles toString, outPath and other similar weirdnesses.

Unexpectedly, the eval-okay-tojson test from the C++ Nix test suite
now passes, too.

This fixes an issue where serialising data structures containing
derivations to JSON would fail.

Change-Id: I5c39e3d8356ee93a07eda481410f88610f6dd9f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8209
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Vincent Ambo
dfd0066de5 fix(tvix/eval): handle toJSON on attribute sets with outPath
These are serialised as the serialisation of the value of that field.

Change-Id: Ida51708b1f43ce09b0ec835f4e265918aa31dd09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8205
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Vincent Ambo
7d339d2762 fix(tvix/eval): handle __toString when JSON-serialising attrsets
These must be serialised to a JSON string of the *result* of coercing
the function application to a string.

Change-Id: Ib7f49ccd950503ddbdbf99643cd59565e26b50da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8204
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Vincent Ambo
43d04d9b98 refactor(tvix/eval): box PathBuf
This shaves another 8 bytes off Value. How did that type get so big?!

Change-Id: I65e9b59a1636bd57e3cc4aec5fea16887070b832
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8153
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Vincent Ambo
52b7a76268 chore(tvix/eval): remove From<SmolStr> for NixString instance
No longer needed, and in some cases caused some extra work.

Change-Id: I64e8e7292573bdc92a9c7a8e470e33f8c526f311
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8152
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Vincent Ambo
4bbfeaf1cb refactor(tvix/eval): wrap NixList in Rc
The size of a `Vector<Value>` is 64 *bytes*, which is quite large, and
it bloated the entire Value type to this size.

This change adds an indirection for the inner vector through Rc.
Initially I tried to use a Box, but this breaks pointer equality
guarantees for the Vector when it is small enough to be inlined.

This reduces the size of Value from 64 to 32 bytes.

Change-Id: Ic3211e861b1966c78b2c3d536ba291fea92647fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8150
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Vincent Ambo
025c67bf4d refactor(tvix/eval): flatten call stack of VM using generators
Warning: This is probably the biggest refactor in tvix-eval history,
so far.

This replaces all instances of trampolines and recursion during
evaluation of the VM loop with generators. A generator is an
asynchronous function that can be suspended to yield a message (in our
case, vm::generators::GeneratorRequest) and receive a
response (vm::generators::GeneratorResponsee).

The `genawaiter` crate provides an interpreter for generators that can
drive their execution and lets us move control flow between the VM and
suspended generators.

To do this, massive changes have occured basically everywhere in the
code. On a high-level:

1. The VM is now organised around a frame stack. A frame is either a
   call frame (execution of Tvix bytecode) or a generator frame (a
   running or suspended generator).

   The VM has an outer loop that pops a frame off the frame stack, and
   then enters an inner loop either driving the execution of the
   bytecode or the execution of a generator.

   Both types of frames have several branches that can result in the
   frame re-enqueuing itself, and enqueuing some other work (in the
   form of a different frame) on top of itself. The VM will eventually
   resume the frame when everything "above" it has been suspended.

   In this way, the VM's new frame stack takes over much of the work
   that was previously achieved by recursion.

2. All methods previously taking a VM have been refactored into async
   functions that instead emit/receive generator messages for
   communication with the VM.

   Notably, this includes *all* builtins.

This has had some other effects:

- Some test have been removed or commented out, either because they
  tested code that was mostly already dead (nix_eq) or because they
  now require generator scaffolding which we do not have in place for
  tests (yet).

- Because generator functions are technically async (though no async
  IO is involved), we lose the ability to use much of the Rust
  standard library e.g. in builtins. This has led to many algorithms
  being unrolled into iterative versions instead of iterator
  combinations, and things like sorting had to be implemented from scratch.

- Many call sites that previously saw a `Result<..., ErrorKind>`
  bubble up now only see the result value, as the error handling is
  encapsulated within the generator loop.

  This reduces number of places inside of builtin implementations
  where error context can be attached to calls that can fail.
  Currently what we gain in this tradeoff is significantly more
  detailed span information (which we still need to bubble up, this
  commit does not change the error display).

  We'll need to do some analysis later of how useful the errors turn
  out to be and potentially introduce some methods for attaching
  context to a generator frame again.

This change is very difficult to do in stages, as it is very much an
"all or nothing" change that affects huge parts of the codebase. I've
tried to isolate changes that can be isolated into the parent CLs of
this one, but this change is still quite difficult to wrap one's mind
and I'm available to discuss it and explain things to any reviewer.

Fixes: b/238, b/237, b/251 and potentially others.
Change-Id: I39244163ff5bbecd169fe7b274df19262b515699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Vincent Ambo
bfb787a6c5 refactor(tvix/eval): remove VM argument from suspended native thunks
Because they do not use it, and it can not be passed with the coming
generator refactoring.

Change-Id: I0d96f2357a7ee79cd8a0f401583d4286230d4a6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8146
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-04 15:53:36 +00:00
Vincent Ambo
38fd3cb292 refactor(tvix/eval): insert storeDir "builtin" in eval startup
Instead of using a suspended native thunk, calculate and optionally
insert the storeDir builtin when the VM is constructed.

We already have the IO handle available at this point and can just
check whether a storeDir is present, and insert its absolute value as
a builtin.

Change-Id: If966eee6ff26dc888b6e888e7c46170c0c346b05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8145
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-04 15:53:36 +00:00
Vincent Ambo
344c119370 chore(tvix/eval): fix clippy warnings
Change-Id: I4c02f0104c455ac00a3f299c1fbf75cbb08e8972
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8142
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-03-03 11:01:52 +00:00
Aaqa Ishtyaq
fc61dff037 chore(tvix/eval): use writeln for newline string
This CL address clippy warning which expects to
use `writeln` instead of `write` for strings with
new line.

Change-Id: Ia72a07502c60cfd489ecf1e3833b9d42d44a8b17
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8030
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-13 19:45:58 +00:00
Vincent Ambo
3419f63575 fix(tvix/eval): ensure all evaluated thunks are correctly memoized
This fixes a very complicated bug (b/246). Evaluation
progresses *much* further after this, leading to several less
complicated bugs likely being uncovered by this

What was the problem?
=====================

Previously, when evaluating a thunk, we had a code path that looked
like this:

    match *thunk {
        ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => {
            let inner_repr = inner_thunk.0.borrow().clone();
            drop(thunk);
            self.0.replace(inner_repr);
        }
        /* ... */
    }

This code path created a copy of the inner `ThunkRepr` of a nested
thunk, and moved that copy into the `ThunkRepr` of the parent.

The effect of this was that the original `ThunkRepr` (unforced!) lived
on in the original thunk, without the memoization of the subsequent
forcing applying to it.

This had the result that Tvix would repeatedly evaluate these thunks
without ever memoizing them, if they occured repeatedly as shared
inner thunks. Most notably, this would *always* occur when
builtins.import was used.

What's the solution?
====================

I have completely rewritten `Thunk::force_trampoline_self` to make all
flows that can occur in it explicit. I have also removed the outer
loop inside of that function, and resorted to more use of trampolining
instead.

The function is now well-commented and it should be possible to read
it from top-to-bottom and get a general sense of what is going on,
though the trampolining itself (which is implemented in the VM) needs
to be at least partially understood for this.

What's the new problem(s)?
==========================

One new (known) problem is that we have to construct `Error` instances
in all error types here, but we do not have spans available in some
thunk-related situations. Due to b/238 we cannot ask the VM for an
arbitrary span from the callsite leading to the force. This means that
there are now code paths where, under certain conditions, causing an
evaluation error during thunk forcing will panic.

To fix this we will need to investigate and fix b/238, and/or add a
span tracking mechanism to thunks themselves.

What other impacts does this have?
==================================

With this commit, eval of nixpkgs mostly succeeds (things like stdenv
evaluate to the same hashes for us and C++ Nix, meaning we now
construct identical derivations without eval breaking).

Due to this we progress much further into nixpkgs, which lets us
uncover more additional bugs. For example, after this commit we can
quickly see that cl/7949 introduces some kind of behavioural issue and
should not be merged as-is (this was not apparent before).

Additionally, tvix-eval is now seemingly very fast. When doing
performance analysis of a nixpkgs eval, we now mostly see the code
path for shelling out to C++ Nix to add things to the store in there.
We still need those code paths, so we can not (yet) do a performance
analysis beyond that.

Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 10:47:18 +00:00
Vincent Ambo
503e6810e7 fix(tvix/eval): unsafeDiscardStringContext is a no-op
... not just a TODO.

Most use-cases of unsafeDiscardStringContext are for cases where a
string is processed in some ways and no longer contains a "physical"
reference, but still has its context attached in C++ Nix.

We don't need to do this. This does diverge in behaviour in use-cases
related to build scheduling, but that whole behaviour will be
different in Tvix.

Change-Id: I4056d4c09f62d44d6bd52b791db03fe5556672b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8016
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-02 14:30:32 +00:00
Vincent Ambo
ddf6e6cf47 fix(tvix/eval): allow builtins.toXML to serialise any function
This adds a fake argument name to builtins.toXML which allows toXML to
serialise any value instead of panicking on functions. We do still
have to fix the value itself, eventually, though.

Change-Id: I2e330ecddcd80442b4fac5eced64431ac86123ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 15:31:30 +00:00
Florian Klink
e0b05c0fa6 feat(tvix/eval): implement builtins.fromTOML
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML
testcase from nix_tests. It uses the `toml` crate, and the serde
integration it brings with it.

Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 07:49:44 +00:00
Vincent Ambo
f12f938166 refactor(tvix/eval): directly return builtin tuples from macro
All invocations of the builtin macro had to previously filter through
the `builtin_tuple` function, but it's more sensible to directly
return these from the macro.

Change-Id: I45600ba84d56c9528d3e92570461c319eea595ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7825
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 15:39:51 +00:00
Vincent Ambo
559a09c5e6 refactor(tvix/eval): remove Box in new_suspended_native
This is unnecessary, Rc already provides all the boxing we need.

Change-Id: I08cf0939c48da43f04c847526c7e5dae5336d528
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7749
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-17 10:38:06 +00:00
Vincent Ambo
d365b09226 feat(tvix/eval): implement builtins.toXML
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-16 13:43:14 +00:00
Aaqa Ishtyaq
158f4d1d69 fix(tvix/eval): len_without_is_empty clippy warn
This CL addresses clippy warning len_without_is_empty
which expects `.is_empty()` method to be present when
implementing `.len()` method for an item.

Change-Id: I8878db630b9ef5853649a906b764a33299bb5dc8
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7806
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-12 18:06:49 +00:00
Vincent Ambo
02d35e4aa6 feat(tvix/eval): implement builtins.toJSON
Implements `Serialize` for `tvix_eval::Value`. Special care is taken
with serialisation of attribute sets, and forcing of thunks.

The tests should cover both cases well.

Change-Id: I9bb135bacf6f87bc6bd0bd88cef0a42308e6c335
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-12 10:42:44 +00:00
Ryan Lahfa
805219a2fa feat(tvix/eval): implement serde::Deserialize for Value
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>

Change-Id: Ib6f7d1f4f4faac36b44f5f75cccc57bf912cf606
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7626
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-10 09:53:21 +00:00
Vincent Ambo
1d59d3ba8f fix(tvix/eval): fix last uses of Vec<Value> -> NixList in builtins
Change-Id: I0d71b82eb7ddc1e457b0996b0668006f55f56751
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7790
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-08 11:11:52 +00:00
Vincent Ambo
e0c47f32af refactor(tvix/eval): use builtins macro for placeholders
Change-Id: I30bc475e3e36a163fa169083481cdd4b4d0ca456
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7785
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-06 20:53:02 +00:00
Vincent Ambo
6b6bd30772 refactor(tvix/eval): move mocked builtins.derivation to tests
This placeholder should not live in the main crate anymore as we will
be injecting the real one from outside of eval, but there are still
language tests that depend on a (simple, mockable) version of it.

Change-Id: I68ea169db15cbdbeed320930d3069e21e376c90d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7783
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 20:53:02 +00:00
sterni
5946b47ab7 feat(tvix/eval): add builtins.{null,true,false}
Code probably rarely relies on these, but it's not hard to support them.

Change-Id: I8499fec34efaf031f9c013bbd370a13db929a2a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7772
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-01-06 12:14:42 +00:00
Vincent Ambo
3d238c350b refactor(tvix/eval): streamline construction of globals/builtins
Previously the construction of globals (a compiler-only concept) and
builtins (a (now) user-facing API) was intermingled between multiple
different modules, and kind of difficult to understand.

The complexity of this had grown in large part due to the
implementation of `builtins.import`, which required the notorious
"knot-tying" trick using Rc::new_cyclic (see cl/7097) for constructing
the set of globals.

As part of the new `Evaluation` API users should have the ability to
bring their own builtins, and control explicitly whether or not impure
builtins are available (regardless of whether they're compiled in or
not).

To streamline the construction and allow the new API features to work,
this commit restructures things by making these changes:

1. The `tvix_eval::builtins` module is now only responsible for
   exporting sets of builtins. It no longer has any knowledge of
   whether or not certain sets (e.g. only pure, or pure+impure) are
   enabled, and it has no control over which builtins are globally
   available (this is now handled in the compiler).

2. The compiler module is now responsible for both constructing the
   final attribute set of builtins from the set of builtins supplied
   by a user, as well as for populating its globals (that is
   identifiers which are available at the top-level scope).

3. The `Evaluation` API now carries a `builtins` field which is
   populated with the pure builtins by default, and can be extended by
   users.

4. The `import` feature has been moved into the compiler, as a
   special case. In general, builtins no longer have the ability to
   reference the "fix point" of the globals set.

This should not change any functionality, and in fact preserves minor
differences between Tvix/Nix that we already had (such as
`builtins.builtins` not existing).

Change-Id: Icdf5dd50eb81eb9260d89269d6e08b1e67811a2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7738
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 12:28:43 +00:00
Vincent Ambo
86361f0f4a refactor(tvix/eval): remove extra Rc<..> around Value::Attrs
The `im::OrdMap` is already small and cheap to copy while sharing
memory, so this is not required anymore.

Only the `KV` variant may have slightly larger content, but in
practice this doesn't seem to make a difference when comparing the two
variants and this one is less complicated.

Change-Id: I64a563b209a2444125653777551373cb2989ca7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7677
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
91465dc78e refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrs
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory
between different iterations of a map.

This slightly speeds up eval, but not significantly. Future work might
include benchmarking whether using a `HashMap` and only ordering in
cases where order is actually required would help.

This switches to a fork of `im` that fixes some bugs with its OrdMap
implementation.

Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
610c44ec1e refactor(tvix/eval): use im::Vector directly where possible
The conversion from im::Vector -> Vec is cheaper for NixList
construction (of course), so where possible we should make use of
that.

This updates most builtins dealing with lists to use Vector directly,
and marks the function constructing NixList from Vec as deprecated so
that we get appropriate warnings in places where it's still in use.

These places are currently inside of JSON serialisation logic which is
in flux right now, so lets leave them as-is until it's stabilised.

Change-Id: I037f12a2800f2576db4d9526bd935efd079163f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7671
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 16:33:14 +00:00
Vincent Ambo
5d73c06b1a refactor(tvix/eval): use im::Vector for NixList representation
This is a persistent, structurally sharing data structure which is
more efficient in some of our use-cases. I have verified the
efficiency improvement using `hyperfine` repeatedly over expressions
on nixpkgs.

Lists are not the most performance-critical structure in Nix (that
would be attribute sets), but we can already see a small (~5-10%)
improvement.

Note that there are a handful of cases where we still go via `Vec`
that need to be fixed, most notable for `builtins.sort` which can not
currently be implemented directly using `im::Vector` because of a
restrictive type bound.

Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-29 12:27:59 +00:00
Vincent Ambo
71174f6626 fix(tvix/eval): fix current clippy warnings
It's been a while since the last time, so quite a lot of stuff has
accumulated here.

Change-Id: I0762827c197b30a917ff470fd8ae8f220f6ba247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7597
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-25 18:25:06 +00:00
Vincent Ambo
edd13573f5 refactor(tvix/eval): use light spans in builtins.import
Change-Id: I05732073155b430575babb6f076bf465aef98857
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7581
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:59:18 +00:00
Adam Joseph
947a56c4b6 feat(tvix/eval): builtins.storeDir
Returns the store directory through EvalIO::store_dir.

Note that this is _optional_ in Tvix, as an evaluation can occur in a
context where there simply is no store directory. In those contexts,
`builtins.storeDir` returns `null` in Tvix.

This would only happen in contexts like Tvixbolt (or completely
unrelated use-cases) in practice.

Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: I5a752c7e89b2f75bd7efb082dbfa5b25e3b1ff3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7452
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:59:18 +00:00
Vincent Ambo
4cb9ada0df refactor(tvix/eval): use EvalIO::read_dir for equivalent builtin
Change-Id: I6d782c07166f51587d2f1d06607823268debb5d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7574
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:59:18 +00:00
Vincent Ambo
51deadd983 refactor(tvix/eval): use EvalIO::path_exists for the builtin
Change-Id: I49822ce30137777865e7370ee86666636e277b35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7573
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:53:02 +00:00
Vincent Ambo
0ef3c2fc2b refactor(tvix/eval): use EvalIO::read_to_string in impure builtins
With this change, the behaviour of reading a string from a file path
is controlled by the provided `EvalIO` structure.

This is a huge step towards abstracting away I/O behaviour correctly.

Change-Id: Ifde8e46cd863b16e0301dca45a434ad27560399f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7567
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:37:11 +00:00
Vincent Ambo
bf286a54bc refactor(tvix/eval): add a LightSpan type for lighter span tracking
This type carries the information required for calculating a
span (i.e. the chunk and offset), instead of the span itself. The span
is then only calculated in cases where it is required (when throwing
errors).

This reduces the eval time for
`builtins.length (builtins.attrNames (import <nixpkgs> {}))` by *one
third*!

The data structure in chunks that carries span information reduces
in-memory size by trading off the speed of retrieving span
information. This is because the span information is only actually
required when throwing errors (or emitting warnings).

However, somewhere along the way we grew a dependency on carrying span
information in thunks (for correctly reporting error chains). Hitting
the code paths for span retrieval was expensive, and carrying the
spans in a different way would still be less cache-efficient. This
change is the best tradeoff I could come up with.

Refs: b/229.
Change-Id: I27d4c4b5c5f9be90ac47f2db61941e123a78a77b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7558
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:21:25 +00:00
Adam Joseph
e04b1697e4 feat(tvix/eval): wrap Closure in Rc<> to match cppnix semantics
Change-Id: I595087eff943d38a9fc78a83d37e207bb2ab79bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7443
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 14:50:04 +00:00