Commit graph

903 commits

Author SHA1 Message Date
Vincent Ambo
83b2290e1b test(tvix/eval): add test for infinite recursion detection
Change-Id: Ief20544a44c3542fe40a5c09f81d0f064a346f44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8149
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-13 20:30:59 +00:00
Vincent Ambo
025c67bf4d refactor(tvix/eval): flatten call stack of VM using generators
Warning: This is probably the biggest refactor in tvix-eval history,
so far.

This replaces all instances of trampolines and recursion during
evaluation of the VM loop with generators. A generator is an
asynchronous function that can be suspended to yield a message (in our
case, vm::generators::GeneratorRequest) and receive a
response (vm::generators::GeneratorResponsee).

The `genawaiter` crate provides an interpreter for generators that can
drive their execution and lets us move control flow between the VM and
suspended generators.

To do this, massive changes have occured basically everywhere in the
code. On a high-level:

1. The VM is now organised around a frame stack. A frame is either a
   call frame (execution of Tvix bytecode) or a generator frame (a
   running or suspended generator).

   The VM has an outer loop that pops a frame off the frame stack, and
   then enters an inner loop either driving the execution of the
   bytecode or the execution of a generator.

   Both types of frames have several branches that can result in the
   frame re-enqueuing itself, and enqueuing some other work (in the
   form of a different frame) on top of itself. The VM will eventually
   resume the frame when everything "above" it has been suspended.

   In this way, the VM's new frame stack takes over much of the work
   that was previously achieved by recursion.

2. All methods previously taking a VM have been refactored into async
   functions that instead emit/receive generator messages for
   communication with the VM.

   Notably, this includes *all* builtins.

This has had some other effects:

- Some test have been removed or commented out, either because they
  tested code that was mostly already dead (nix_eq) or because they
  now require generator scaffolding which we do not have in place for
  tests (yet).

- Because generator functions are technically async (though no async
  IO is involved), we lose the ability to use much of the Rust
  standard library e.g. in builtins. This has led to many algorithms
  being unrolled into iterative versions instead of iterator
  combinations, and things like sorting had to be implemented from scratch.

- Many call sites that previously saw a `Result<..., ErrorKind>`
  bubble up now only see the result value, as the error handling is
  encapsulated within the generator loop.

  This reduces number of places inside of builtin implementations
  where error context can be attached to calls that can fail.
  Currently what we gain in this tradeoff is significantly more
  detailed span information (which we still need to bubble up, this
  commit does not change the error display).

  We'll need to do some analysis later of how useful the errors turn
  out to be and potentially introduce some methods for attaching
  context to a generator frame again.

This change is very difficult to do in stages, as it is very much an
"all or nothing" change that affects huge parts of the codebase. I've
tried to isolate changes that can be isolated into the parent CLs of
this one, but this change is still quite difficult to wrap one's mind
and I'm available to discuss it and explain things to any reviewer.

Fixes: b/238, b/237, b/251 and potentially others.
Change-Id: I39244163ff5bbecd169fe7b274df19262b515699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Vincent Ambo
cbb4137dc0 feat(tvix/eval): implement generator-based Nix equality logic
Adds a `Value::neo_nix_eq` method (the `neo_` prefix will be dropped
when we flip over to the generator implementation of the VM) which
implements Nix equality semantics using async, generator-based
comparisons.

Instead of tracking the "kind" of equality that is being compared (see
the pointer-equality doc) through a pair of booleans, I've introduced
an enum that explicitly lists the possible comparisons.

Change-Id: I3354cc1470eeccb3000a5ae24f2418db1a7a2edc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8241
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-03-13 20:30:59 +00:00
Vincent Ambo
cd447e1859 feat(tvix/eval): add generator-related functions to RuntimeObserver
These functions will be used by the changes in the VM to observe the
runtime execution of generator frames, and provide a more linear view
of the execution of the Tvix VM.

Change-Id: I10b1b1933dedc065e7c61d5d6062f0aaeee0097e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8240
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-03-13 20:30:59 +00:00
Vincent Ambo
d00229753d feat(tvix/eval): implement asynchronous list sorting algorithm
In order to implement an asynchronous builtins.sort (required for
moving builtins to generators), we need an `async` sorting algorithm
as our comparators involve invoking a Nix function.

This commit implements a fairly simple, optimised bubble sort as the
sorting algorithm used in our `async fn sort_by`.

There don't seem to be any crates providing async versions of things
like this, and they might actually be pretty hard to implement
generically due to some constraints about how `async` works.

Note that this algorithm is less efficient than the hybrid
"timsort/mergesort/insert sort" used in the Rust standard library. I
tried to write a merge sort implementation, but ran into isuses with
the sort becoming unstable because our comparators can not yield
equality. This is the simplest implementation which I know to be
correct.

Note that as of this commit this is *not* covered by the Tvix test
suite, but it will be as soon as the rest of the generator code lands.

Change-Id: Ia9a604f7dd941d6acc9212c902e0e637ed75bebc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8239
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-03-13 20:30:59 +00:00
Florian Klink
c9c8b10370 feat(tvix/eval): don't warn twice about dead code
We currently send two warnings in case of detecting dead code - W008
inside compile_dead_code, and a more detailed warning in all places that
invoke compile_dead_code:

```
warning[W007]: useless operation on boolean: this expression is always false
     --> /nix/store/qz3gjn95gazab4fkb7s8lm6hz17rdzzy-414z9nnj1wy66ymq6vgb693x9xjz6hf2-nixpkgs-src/pkgs/top-level/perl-packages.nix:12079:15
      |
12079 |     doCheck = false && !stdenv.isDarwin;
      |               ^^^^^^^^^^^^^^^^^^^^^^^^^

warning[W008]: this code will never be executed
     --> /nix/store/qz3gjn95gazab4fkb7s8lm6hz17rdzzy-414z9nnj1wy66ymq6vgb693x9xjz6hf2-nixpkgs-src/pkgs/top-level/perl-packages.nix:12079:24
      |
12079 |     doCheck = false && !stdenv.isDarwin;
      |                        ^^^^^^^^^^^^^^^^
```

The place invoking `compile_dead_code` has more context to why the code
is unused, so it's error message is much more useful.

Stop emitting the less informative warning inside compile_dead_code
(W008), and update the comment that we expect the caller to emit a
warning.

I kept W008 itself still around, in case we end up having places this
will get used again.

Change-Id: I2c5d84fc0cb4035872cd4b71cc3e9e34e120eb37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8024
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:26:49 +00:00
Vincent Ambo
2696839770 feat(tvix/eval): introduce generators module
This module contains the request/response types for generators
requesting actions from the VM.

For most of these, an async helper function is added that will be used
inside of generator functions to make use of these requests/responses
instead of constructing them directly.

Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-07 22:04:59 +00:00
Vincent Ambo
9cebae9b56 refactor(tvix/eval): merge OpCall & OpTailCall
As applies are thunked, there was no situation where OpCall could be
emitted. In practice, all calls were already tail calls.

Change-Id: Id0d441dcdd86f804d7cddd0cc14f589bbfc75e5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8147
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-07 22:04:59 +00:00
Vincent Ambo
bfb787a6c5 refactor(tvix/eval): remove VM argument from suspended native thunks
Because they do not use it, and it can not be passed with the coming
generator refactoring.

Change-Id: I0d96f2357a7ee79cd8a0f401583d4286230d4a6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8146
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-04 15:53:36 +00:00
Vincent Ambo
38fd3cb292 refactor(tvix/eval): insert storeDir "builtin" in eval startup
Instead of using a suspended native thunk, calculate and optionally
insert the storeDir builtin when the VM is constructed.

We already have the IO handle available at this point and can just
check whether a storeDir is present, and insert its absolute value as
a builtin.

Change-Id: If966eee6ff26dc888b6e888e7c46170c0c346b05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8145
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-04 15:53:36 +00:00
Vincent Ambo
e1f082a3ab feat(tvix/eval): add SharedThunkSet
This is a ThunkSet wrapped to be shareable, which will be required
once ThunkSets are embedded in futures.

Change-Id: I5a067b7972ac86e4d354c75ef05c86b2284c1137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8144
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-04 15:18:37 +00:00
Vincent Ambo
dccbda5960 fix(tvix/eval): ThunkSet does not need mutable pointers
Change-Id: Iea248870a0ea5d38cb02ff059c968fbd563570b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8143
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-04 15:18:37 +00:00
Vincent Ambo
8c7d5b4f87 chore(tvix/eval): implement From<OrdMap<..>> for NixAttrs
Again simplifying some code down the line, where bits of code that
construct attribute sets already have the final structure available.

Change-Id: I0bb7a1daa63298122b51be73d35d695a4f73f8b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8140
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-04 15:18:37 +00:00
Vincent Ambo
a11a3e2c59 refactor(tvix/eval): implement From<Span> for LightSpan
This simplifies some code down the line.

Change-Id: I58dd71e796e11479f44516cf24932f8061843d23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8139
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-04 15:18:37 +00:00
Vincent Ambo
344c119370 chore(tvix/eval): fix clippy warnings
Change-Id: I4c02f0104c455ac00a3f299c1fbf75cbb08e8972
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8142
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-03-03 11:01:52 +00:00
Vincent Ambo
7e9a65dcdb refactor(tvix/eval): remove useless map call
Change-Id: Ifb59ef148ea4fab613f2e4efb133c04baafa3a98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8141
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-03-03 10:57:50 +00:00
Vincent Ambo
fb4c197b39 refactor(tvix/eval): enhance debug output for bytecode dumps
This adds addresses of thunk and closure chunks to the debug output
displayed when dumping bytecode.

This makes it possible to see in the dump which thunks are referenced
by constants in other thunks.

Change-Id: I2c98de5227e7cb415666cd3134c947a56979dc80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8137
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-03 10:52:48 +00:00
Aaqa Ishtyaq
faffb2a4cb refactor(tvix/eval): remove redundant clone
This CL removes redundant clone from value which is
going to be dropped without further use.

Change-Id: Ibd2a724853c5cfbf8ca40bf0b3adf0fab89b9be5
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8125
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-16 22:42:14 +00:00
Vincent Ambo
bfe6cace5e docs(tvix/eval): add proposal for VM loop restructuring
Change-Id: Ib991d68724a73886a8343d7f785b5b3aedd637ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8103
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-02-16 22:23:51 +00:00
Vincent Ambo
3c311ffab2 fix(tvix/eval): correctly print lambda address in observer
We want the address that the Rc is pointing to, not the address of the
Rc.

Change-Id: I8eba21677f242bbe4166c74d4aa4269c316076e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8045
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-14 08:02:40 +00:00
Aaqa Ishtyaq
fc61dff037 chore(tvix/eval): use writeln for newline string
This CL address clippy warning which expects to
use `writeln` instead of `write` for strings with
new line.

Change-Id: Ia72a07502c60cfd489ecf1e3833b9d42d44a8b17
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8030
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-13 19:45:58 +00:00
Aaqa Ishtyaq
105069302b chore(tvix/eval): clippy warn is length zero
This CL address clippy warning about finding the zero
length of something using `is_empty()` instead of `len() == 0`.

Change-Id: I2b36c7c7b65b733609fc0dcd33be06f9d772bc9b
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8029
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-13 19:45:58 +00:00
Vincent Ambo
c98c5399b9 fix(tvix/eval): skip runtime completely on compiler errors
This branch was missing, and an assumption elsewhere just executed the
returned (broken) bytecode.

This fixes b/253.

Change-Id: I015023ba921bc08ea03882167f1f560feca25e50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8090
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-13 16:21:47 +00:00
Vincent Ambo
91a366af46 fix(tvix/eval): make fields of eval's Error type public
These should be inspectable by callers.

Change-Id: Ia9ef871aa63958d06066aaea61b2aecbd217369b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8089
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-13 16:21:46 +00:00
Vincent Ambo
720e80c1f9 fix(tvix/eval): fix the default case for path parsing
Plain paths like `foo/bar.nix` are also allowed, so we can not
determine this based on the prefix.

The upstream PR that is referenced in a comment here has a
significantly different interface than we expected, so I'm not
touching that comment yet in this CL before I've had more time to
digest it.

Change-Id: Iea33bbb35de9c00a7d7fedf64d02253c75c1cc9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8032
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-04 12:44:59 +00:00
Florian Klink
d50044d570 chore(tvix/eval): only use Rc with impure feature
We only use Rc in `impl EvalIO for StdIO`, which is only included when
building with the "impure" feature.

Change-Id: Id29d647c899cbfcdda11abfb9fabd5aa7e24299f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8025
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-03 18:58:17 +00:00
Vincent Ambo
f16b0f24e2 refactor(tvix/eval): wrap Builtin type in a Box
This reduces the size of `Builtin` from 88 (!) bytes to 8, and as the
largest variant of `Value`, the size of that type from 96 to 64.

The next largest type is NixList, clocking in at 64 bytes.

This has noticeable performance impact. In an implementation without
disk I/O, evaluating nixpkgs.stdenv looks like this:

Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath'
  Time (mean ± σ):      1.151 s ±  0.003 s    [User: 1.041 s, System: 0.109 s]
  Range (min … max):    1.147 s …  1.155 s    10 runs

After this change, it looks like this:

Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath'
  Time (mean ± σ):      1.046 s ±  0.004 s    [User: 0.954 s, System: 0.092 s]
  Range (min … max):    1.041 s …  1.053 s    10 runs

Change-Id: I5ab7cc02a9a450c0227daf1f1f72966358311ebb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8027
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 18:47:33 +00:00
Vincent Ambo
32698766ef refactor(tvix/eval): statically resolve select from constant attrs
When resolving a select expression (`attrs.name` or `attrs.name or
default`), if the set compiles to a constant attribute set (as is most
notably the case with `builtins`) we can backtrack and replace that
attribute set directly with the compiled value.

For something like `builtins.length`, this will directly emit an
`OpConstant` that leaves the `length` builtin on the stack.

Change-Id: I639654e065a06e8cfcbcacb528c6da7ec9e513ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 18:47:33 +00:00
Vincent Ambo
3419f63575 fix(tvix/eval): ensure all evaluated thunks are correctly memoized
This fixes a very complicated bug (b/246). Evaluation
progresses *much* further after this, leading to several less
complicated bugs likely being uncovered by this

What was the problem?
=====================

Previously, when evaluating a thunk, we had a code path that looked
like this:

    match *thunk {
        ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => {
            let inner_repr = inner_thunk.0.borrow().clone();
            drop(thunk);
            self.0.replace(inner_repr);
        }
        /* ... */
    }

This code path created a copy of the inner `ThunkRepr` of a nested
thunk, and moved that copy into the `ThunkRepr` of the parent.

The effect of this was that the original `ThunkRepr` (unforced!) lived
on in the original thunk, without the memoization of the subsequent
forcing applying to it.

This had the result that Tvix would repeatedly evaluate these thunks
without ever memoizing them, if they occured repeatedly as shared
inner thunks. Most notably, this would *always* occur when
builtins.import was used.

What's the solution?
====================

I have completely rewritten `Thunk::force_trampoline_self` to make all
flows that can occur in it explicit. I have also removed the outer
loop inside of that function, and resorted to more use of trampolining
instead.

The function is now well-commented and it should be possible to read
it from top-to-bottom and get a general sense of what is going on,
though the trampolining itself (which is implemented in the VM) needs
to be at least partially understood for this.

What's the new problem(s)?
==========================

One new (known) problem is that we have to construct `Error` instances
in all error types here, but we do not have spans available in some
thunk-related situations. Due to b/238 we cannot ask the VM for an
arbitrary span from the callsite leading to the force. This means that
there are now code paths where, under certain conditions, causing an
evaluation error during thunk forcing will panic.

To fix this we will need to investigate and fix b/238, and/or add a
span tracking mechanism to thunks themselves.

What other impacts does this have?
==================================

With this commit, eval of nixpkgs mostly succeeds (things like stdenv
evaluate to the same hashes for us and C++ Nix, meaning we now
construct identical derivations without eval breaking).

Due to this we progress much further into nixpkgs, which lets us
uncover more additional bugs. For example, after this commit we can
quickly see that cl/7949 introduces some kind of behavioural issue and
should not be merged as-is (this was not apparent before).

Additionally, tvix-eval is now seemingly very fast. When doing
performance analysis of a nixpkgs eval, we now mostly see the code
path for shelling out to C++ Nix to add things to the store in there.
We still need those code paths, so we can not (yet) do a performance
analysis beyond that.

Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 10:47:18 +00:00
Vincent Ambo
503e6810e7 fix(tvix/eval): unsafeDiscardStringContext is a no-op
... not just a TODO.

Most use-cases of unsafeDiscardStringContext are for cases where a
string is processed in some ways and no longer contains a "physical"
reference, but still has its context attached in C++ Nix.

We don't need to do this. This does diverge in behaviour in use-cases
related to build scheduling, but that whole behaviour will be
different in Tvix.

Change-Id: I4056d4c09f62d44d6bd52b791db03fe5556672b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8016
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-02 14:30:32 +00:00
Vincent Ambo
75d1b72b7c refactor(tvix/eval): import_cache can be a HashMap
... instead of a BTreeMap, as we do not need ordering guarantees here.
HashMaps are noticeably faster here (especially as we've been sorting
essentially random data!).

Change-Id: Ie92d74286df9f763c04c9b226ef1066ee8484c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8014
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 14:25:30 +00:00
Vincent Ambo
98ea614c6e chore(tvix/eval): elaborate on internal types in Value::type_of
This aids in debugging quite substantially.

Change-Id: Ic43232aa6165ae1c3db7ac2701938e1dfeeb418c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8013
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 09:29:43 +00:00
Vincent Ambo
ddf6e6cf47 fix(tvix/eval): allow builtins.toXML to serialise any function
This adds a fake argument name to builtins.toXML which allows toXML to
serialise any value instead of panicking on functions. We do still
have to fix the value itself, eventually, though.

Change-Id: I2e330ecddcd80442b4fac5eced64431ac86123ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 15:31:30 +00:00
Vincent Ambo
7702941dd9 test(tvix/eval): add tests for internal formals dependencies
Formals can depend on each other when using another formal as a
default value.

This test ensures that the compiler's declaration and initialisation
order of formals is consistent with what actually happens in the VM.

Change-Id: Ibdabe262554e8066d67fac1ebc3b5a48ef626e18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7948
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 10:25:02 +00:00
Vincent Ambo
1028aff105 chore(tvix/eval): remove dead comment
Change-Id: Icedb7f272e5067569b8dbf1c2d8b0fdd352b8e12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7936
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-26 23:30:43 +00:00
sterni
8a8325fb9d docs(tvix/eval): builtins.add is not equivalent to +
While it is in the given example, i.e. for integer addition, to claim
that they are equivalent is a bit misleading: builtins.add is less
overloaded than +, i.e. builtins.add "foo" "bar" will fail whereas
"foo" + "bar" performs string concatenation.

Change-Id: Ib52d530d1ab289b367565b286f06a76dd518d4fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7929
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-25 14:30:50 +00:00
Florian Klink
669496f0ba test(tvix/eval): add test for total_fmt_float
Change-Id: If6c478ee3d2e4ecf5ef92289614f86535ad05cb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7927
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-25 11:24:20 +00:00
Vincent Ambo
164005656d refactor(tvix/eval): extract float formatting into a helper
This keeps the actual TotalDisplay implementation readable, as this
float formatting code suddenly made up the majority of its implementation.

Change-Id: I2c0d00e4a691e0b8ffbc72680f680e16feef4bee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7925
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 11:24:19 +00:00
Florian Klink
e0b05c0fa6 feat(tvix/eval): implement builtins.fromTOML
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML
testcase from nix_tests. It uses the `toml` crate, and the serde
integration it brings with it.

Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 07:49:44 +00:00
Florian Klink
1facd889bb feat(tvix/eval): use lexical-core to format float
Apparently our naive implementation of float formatting, which simply
used {:.5}, and trimmed trailing "0" strings not sufficient.

It wrongly trimmed numbers with zeroes but no decimal point, like
`10000` got trimmed to `1`.

Nix uses `std::to_string` on the double, which according to
https://en.cppreference.com/w/cpp/string/basic_string/to_string
is equivalent to `std::sprintf(buf, "%f", value)`.

https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated
like this:

> Precision specifies the exact number of digits to appear after
> the decimal point character. The default precision is 6. In the
> alternative implementation decimal point character is written even if
> no digits follow it. For infinity and not-a-number conversion style
> see notes.

This doesn't seem to be the case though, and Nix uses scientific
notation in some cases.

There's a whole bunch of strategies to determine which is a more compact
notation, and which notation should be used for a given number.
https://github.com/rust-lang/rust/issues/24556 provides some pointers
into various rabbit holes for those interested.

This gist seems to be that currently a different formatting is not
exposed in rust directly, at least not for public consumption.

There is the
[lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate
though, which provides a way to format floats with various strategies
and formats.

Change our implementation of `TotalDisplay` for the `Value::Float` case
to use that. We still need to do some post-processing, because Nix
always adds the sign in scientific notation (and there's no way to
configure lexical-core to do that), and lexical-core in some cases keeps
the trailing zeros.

Even with all that in place, there as a difference in `eval-okay-
fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated
the fixture to a less problematic number.

With this, the testsuite passes again, and does for the upcoming CL
introducing builtins.fromTOML, and enabling the nix testsuite bits for
it, too.

Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-25 07:49:44 +00:00
Vincent Ambo
9ecf668452 chore(tvix/eval): delete "useless parenthesis" warning/optimisation
Two main reasons:

1. Traversing the structure to do this optimisation is
   actually *slower* than not optimising it.

2. There are literally hundreds of thousands of incidences of this in
   nixpkgs, and with some of the weird code there some of
   these (functionally) useless parens are actually required for
   readability reasons.

Change-Id: I1044b1c5f9fe20df4b6085851fc3b191277c65dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7917
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-23 17:59:06 +00:00
Vincent Ambo
d2f1a290a4 fix(tvix/eval): force functors before applying them
call_value in the VM expects the callable to be forced when calling
it, which was not the case for functors.

Change-Id: Id55a2fe32a9573be42aef8669e268df519a989cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7909
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-23 17:27:53 +00:00
Vincent Ambo
5719763fd3 feat(tvix/eval): support builtins implemented in Nix itself
This makes it possible to inject builtins into the builtin set that
are written in Nix code, and which at runtime are represented by a
thunk that will compile them the first time they are used.

Change-Id: Ia632367328f66fb2f26cb64ae464f8f3dc9c6d30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7891
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-22 20:48:53 +00:00
Vincent Ambo
8513a58b37 docs(tvix/eval): update some outdated comments
These don't apply anymore since the "antidote-CL".

Change-Id: I40ee73ef43d44bbfc650a8fe6c2b33263dd06959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7890
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-22 19:39:00 +00:00
Florian Klink
758b94f47c docs(tvix): move most of //tvix/eval README up to //tvix
While moving the CLI out of the evaluator, we forgot to update the
README in //tvix/eval. Move this up to //tvix, so people know where
to start.

Keep the instructions on how to build only `//tvix/eval` in `//tvix/
eval/README.md`.

Change-Id: Ie2755e8b5a0056225dbf3a0ee040f70f7f6a1f27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7887
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 15:08:38 +00:00
Adam Joseph
22b9e6ff09 refactor(tvix/eval): administer antidote for poison
The codebase contains a lot of complexity and odd roundabout
handling for shadowing globals.  I'm pretty sure none of this is
necessary, and all of it disappears if you simply make the globals
part of the ordinary identifier resolution chain, with their own
scope up above the root scope.  Then the ordinary shadowing routines
do the right thing, and no special cases or new terminology are
required.

This commit does that.

Note by tazjin: This commit was originally abandoned when Adam decided
not to take away reviewer bandwidth for this at the time (eval was
still in a much earlier stage). As we've recently done some
significant refactoring of globals initialisation this came up again,
and it seems we can easily cover the use-cases of the poison tracking
in other ways now, so I've rebased, updated and resurrected the CL.

Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib3309a47a7b31fa5bf10466bade0d876b76ae462
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7089
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-21 10:19:15 +00:00
Vincent Ambo
7442558b33 refactor(tvix/eval): keep globals alive through VM struct
This forces users to pass the fully constructed set of globals to the
VM, making it harder to accidentally "lose" the set while weak
references to it still exist.

This doesn't modify any functionality, but is laying the foundation
for simplifying some of the builtins behaviour that has grown more
complex again.

Change-Id: I5120f97861c65dc46d90b8a4e2c92ad32cc53e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7877
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 22:48:13 +00:00
Vincent Ambo
6d03e31060 docs(tvix/eval): add doc comments to VM fields
Change-Id: Ia4857c217de15aec8b61e1abd39e22c50e2d816a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7876
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-20 22:48:13 +00:00
Vincent Ambo
7d0456fa0e feat(tvix/eval): let builtin macro capture external state
This adds a feature to the `#[builtins]` macro which lets users
specify an additional state type to (optionally) thread through to
builtins when constructing them.

This makes it possible for builtins-macro users to pass external state
handles (specifically, in our case, known path tracking) into a set of
builtins.

Change-Id: I3ade20d333fc3ba90a80822cdfa5f87a9cfada75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7840
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-20 15:39:51 +00:00
Vincent Ambo
f12f938166 refactor(tvix/eval): directly return builtin tuples from macro
All invocations of the builtin macro had to previously filter through
the `builtin_tuple` function, but it's more sensible to directly
return these from the macro.

Change-Id: I45600ba84d56c9528d3e92570461c319eea595ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7825
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 15:39:51 +00:00
Vincent Ambo
972c867b36 feat(tvix/eval): add error contexts to annotate error kinds
This makes it possible for users to add additional context to an
error, which will then be rendered as an additional secondary span in
the formatted error output.

We should strive to do this basically anywhere errors are raised that
can occur multiple times, *especially* during type casts. This was
triggered by me debugging a type cast error attached to a fairly
large-ish span (a builtin invocation).

Change-Id: I51be41fabee00cf04de973935daf34fe6424e76f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 15:39:51 +00:00
Vincent Ambo
bf6f6a0b3f refactor(tvix/eval): non-hacky suspended native thunks
Instead of having a representation of suspended native thunks that
involves constructing a fake code chunk, make these thunks a
first-class part of the internal thunk representation.

The previous code was not that simple to understand, and actually
contained a critical bug which could lead to Tvix crashes. This
version fixes the particular instance of that bug, but instead
uncovers another (b/238) which can still lead to Tvix crashes.

Fixes: b/237.
Change-Id: I771d03864084d63953bdbb518fec94487481f839
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7750
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-17 10:38:06 +00:00
Vincent Ambo
559a09c5e6 refactor(tvix/eval): remove Box in new_suspended_native
This is unnecessary, Rc already provides all the boxing we need.

Change-Id: I08cf0939c48da43f04c847526c7e5dae5336d528
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7749
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-17 10:38:06 +00:00
Vincent Ambo
b9646ab40c feat(tvix/eval): add builtins to builtins
This is a somewhat terrifying hack that enables us to support
`builtins.builtins`, by running a "fake compilation" inside of a
suspended native thunk that can resolve the weak pointer to the
globals.

With this implementation, the thunk at `builtins.builtins` actually
resolves to the "real" `builtins` (verified with a new test).

This is kind of ugly, and it's something users shouldn't use, but
bubbling a warning out of this is difficult at the moment due to a
little bit of trickery with how the spans in suspended native thunks
work (they don't) (see b/237, b/238)

Change-Id: I67d0e93246dd5b279c960aeda00402031aa12af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:38:06 +00:00
Vincent Ambo
940251b87f refactor(tvix/value): use proptest strategies from imbl crate
Instead of going through Vec/BTreeMap for generating our internal
types, use the proptest strategies from imbl.

The one thing I couldn't figure out in the previous implementation is
where the ranges/sizes of generated collections came from. The
strategies in proptest use different types (Range, with an unknown
default value, and SizeRange with 0..100). I've opted to specify
0..100 directly, but we can probably make it configurable.

Change-Id: I749bc4c703fe424099240cab822b1642e5216361
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7791
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:20:38 +00:00
Vincent Ambo
67addf3b3d feat(tvix/eval): add error variant for threading through errors
This variant is required for external builtins (which in our case
includes `derivation`) to thread through reasonable error messages.

This has some potential for improvement, but it's an improvement over
the status quo of panicking in the external builtins when no
appropriate error is available.

Change-Id: I7e4bdb0a156c7717092dde30aa4785192182dc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7841
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-16 13:43:14 +00:00
Vincent Ambo
d365b09226 feat(tvix/eval): implement builtins.toXML
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-16 13:43:14 +00:00
Vincent Ambo
1786b4c835 chore(tvix/eval): add other required items to public API
External implementors of builtins must be able to force values, which
necessitates publishing a bunch more items from the crate.

Change-Id: I8f6b8ae88156aae417dbe630a698d123d0c1c8d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7830
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-16 13:43:14 +00:00
Aaqa Ishtyaq
158f4d1d69 fix(tvix/eval): len_without_is_empty clippy warn
This CL addresses clippy warning len_without_is_empty
which expects `.is_empty()` method to be present when
implementing `.len()` method for an item.

Change-Id: I8878db630b9ef5853649a906b764a33299bb5dc8
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7806
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-12 18:06:49 +00:00
Vincent Ambo
09654ffb80 docs(tvix): add build-references / string-context document
This explains my current thinking on string contexts. Thanks to
everyone who gave input so far.

Change-Id: I773219402a79a9d4753b4e7cfbf3a4a751a993a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7807
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-12 11:01:40 +00:00
Vincent Ambo
02d35e4aa6 feat(tvix/eval): implement builtins.toJSON
Implements `Serialize` for `tvix_eval::Value`. Special care is taken
with serialisation of attribute sets, and forcing of thunks.

The tests should cover both cases well.

Change-Id: I9bb135bacf6f87bc6bd0bd88cef0a42308e6c335
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-12 10:42:44 +00:00
Aaqa Ishtyaq
9382afdb0d fix(tvix/eval): address useless_format clippy warn
This CL address clippy warnings related to use of 'format!' macro
to return unmodified 'String'.

Change-Id: I88726e59d8f39f6a455a8c1f48075b52d167e489
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7804
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-10 19:13:45 +00:00
Ryan Lahfa
805219a2fa feat(tvix/eval): implement serde::Deserialize for Value
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>

Change-Id: Ib6f7d1f4f4faac36b44f5f75cccc57bf912cf606
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7626
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-10 09:53:21 +00:00
Vincent Ambo
c011a6130c refactor(tvix/eval): impl Display for ErrorKind
This just shuffles the Display implementations around so that
ErrorKind itself is displayable, which is useful in some situations
where errors under construction need to be type-converted.

Change-Id: I7b633d03d0dc34f345c4f20676e0023ecb1db0c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7802
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-01-10 09:52:52 +00:00
Vincent Ambo
1d59d3ba8f fix(tvix/eval): fix last uses of Vec<Value> -> NixList in builtins
Change-Id: I0d71b82eb7ddc1e457b0996b0668006f55f56751
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7790
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-08 11:11:52 +00:00
Vincent Ambo
a149a1ee06 fix(tvix/eval): fix typo'd function name in tests
Caught by sterni on cl/7783.

Change-Id: I15d57b893ef22538fdd7e809f3b92861dd2bc1af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7789
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-07 14:48:42 +00:00
Vincent Ambo
e0c47f32af refactor(tvix/eval): use builtins macro for placeholders
Change-Id: I30bc475e3e36a163fa169083481cdd4b4d0ca456
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7785
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-06 20:53:02 +00:00
Vincent Ambo
6b6bd30772 refactor(tvix/eval): move mocked builtins.derivation to tests
This placeholder should not live in the main crate anymore as we will
be injecting the real one from outside of eval, but there are still
language tests that depend on a (simple, mockable) version of it.

Change-Id: I68ea169db15cbdbeed320930d3069e21e376c90d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7783
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 20:53:02 +00:00
Vincent Ambo
87c80895cd feat(tvix/eval): skip & warn for useless parenthesis
Change-Id: I567ca0682012b9d09f1217e57a104ac5671f8d82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7771
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 12:23:55 +00:00
Vincent Ambo
aadf71a6ed feat(tvix/eval): warn on empty let-bindings
Change-Id: Ib6ef7ce514abbd3e372dfe9df7137aa36dbda9d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7770
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
61b8a9b2ba refactor(tvix/eval): short-circuit on empty attrs in compiler
This is marginally more efficient and has simpler bytecode.

Change-Id: Iad37c9aeef24583e8f696911bcd83d43639f2e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7769
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 12:23:55 +00:00
Vincent Ambo
4e98730f38 feat(tvix/eval): warn about empty inherits
Change-Id: I82bec6fe2210bcb88c46fd2fdf3e26bd613d1c1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7768
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
e8dcdceb34 fix(tvix/eval): compile but don't emit dead code
This adds a mechanism to the compiler to compile an expression without
emitting any code. This allows for detected dead code to still be
compiled to detect errors & warnings inside of it.

Change-Id: Ie78479173570e9c819d8f32ae683ce34234a4c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7767
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
6a8541e35a feat(tvix/eval): implement initial compiler::optimiser module
This optimiser can rewrite some expressions into more efficient forms,
and warn users about those cases.

As a proof-of-concept, only some simple boolean comparisons are
supported for now.

Change-Id: I7df561118cfbad281fc99523e859bc66e7a1adcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 12:23:55 +00:00
Vincent Ambo
36e5a4cc07 refactor(tvix/eval): take owned ast::Expr in Compiler::compile
This adds a very minimal amount of additional Rc-increments (~1 per
compilation), but makes it a lot easier to add an AST-optimising
compiler pass without incurring a lot of extra cost.

Change-Id: I57208bdfc8882e3ae21c5850e14aa380d3ccea36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7765
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
5926a05f46 feat(tvix/eval): add Evaluation::compile_only method
This would make it possible to implement something like a linter based
on the tvix-eval compiler warnings.

Change-Id: I1feb4e7c4a44be7d1204b0a962ab522fd32b93c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7763
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 12:23:55 +00:00
Vincent Ambo
27fecee1bc fix(tvix/eval): don't increase with_stack_size in scope inherits
There was probably a misunderstanding somewhere about the
with_stack_size being related to how far away it is from the with, but
it is about whether there is a with at all.

This broke a warning (`UselessInherit`), and may actually have let to
more inefficient codegen in some cases.

Change-Id: I08338ea59ae39dad01ca8a4e09d934a936cdea2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7762
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
88432235ae fix(tvix/eval): VM & Builtin* types have to be public
... without them, using the new Builtins API is basically impossible
for library consumers.

Change-Id: Ice0557a2e55e12d812f51bf5a99e6b8c91ad1b91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7755
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:15:23 +00:00
sterni
5946b47ab7 feat(tvix/eval): add builtins.{null,true,false}
Code probably rarely relies on these, but it's not hard to support them.

Change-Id: I8499fec34efaf031f9c013bbd370a13db929a2a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7772
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-01-06 12:14:42 +00:00
sterni
b7090ec874 test(tvix/eval): add test for builtins parity
This will eventually force us to have a base builtins set in common with
C++ Nix, i.e. all 2.3 builtins except the controversial
builtins.valueSize.

Change-Id: I2c767f07d6a14711911658e87da9f18ede57a143
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7747
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 12:00:38 +00:00
Vincent Ambo
34be6466d4 feat(tvix/serde): implement enum deserialisation
Implements externally tagged enum deserialisation. Other serialisation
methods are handled by serde internally using the existing methods.

See the tests for examples.

Change-Id: Ic4a9da3b5a32ddbb5918b1512e70c3ac5ce64f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7721
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 17:21:40 +00:00
Vincent Ambo
0c17718dd1 refactor(tvix/eval): avoid unnecessary pop/push in OpForce
Change-Id: Ic7559eaa43aa0dcc97babb7669770c0f7f959f1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7754
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-04 16:30:03 +00:00
sterni
9b8ba915c8 fix(tvix/eval): ' is allowed in nonfirst position in Nix identifiers
With this is_valid_nix_identifier should line up with the upstream lexer
definition:

    ID          [a-zA-Z\_][a-zA-Z0-9\_\'\-]*

While we're working on this, add a simple test checking the various
formatting rules. Interestingly, it would not be suitable as an identity
test, since you have to write

    { "assert" = null; }

in order to avoid an evaluation error, but C++ Nix is happy to print
this as

    { assert = null; }

– maybe should be considered to be a bug.

Change-Id: I0a4e1ccb5033a80f3767fb8d1c4bba08d303c5d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7744
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 12:36:28 +00:00
Vincent Ambo
3d238c350b refactor(tvix/eval): streamline construction of globals/builtins
Previously the construction of globals (a compiler-only concept) and
builtins (a (now) user-facing API) was intermingled between multiple
different modules, and kind of difficult to understand.

The complexity of this had grown in large part due to the
implementation of `builtins.import`, which required the notorious
"knot-tying" trick using Rc::new_cyclic (see cl/7097) for constructing
the set of globals.

As part of the new `Evaluation` API users should have the ability to
bring their own builtins, and control explicitly whether or not impure
builtins are available (regardless of whether they're compiled in or
not).

To streamline the construction and allow the new API features to work,
this commit restructures things by making these changes:

1. The `tvix_eval::builtins` module is now only responsible for
   exporting sets of builtins. It no longer has any knowledge of
   whether or not certain sets (e.g. only pure, or pure+impure) are
   enabled, and it has no control over which builtins are globally
   available (this is now handled in the compiler).

2. The compiler module is now responsible for both constructing the
   final attribute set of builtins from the set of builtins supplied
   by a user, as well as for populating its globals (that is
   identifiers which are available at the top-level scope).

3. The `Evaluation` API now carries a `builtins` field which is
   populated with the pure builtins by default, and can be extended by
   users.

4. The `import` feature has been moved into the compiler, as a
   special case. In general, builtins no longer have the ability to
   reference the "fix point" of the globals set.

This should not change any functionality, and in fact preserves minor
differences between Tvix/Nix that we already had (such as
`builtins.builtins` not existing).

Change-Id: Icdf5dd50eb81eb9260d89269d6e08b1e67811a2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7738
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 12:28:43 +00:00
Vincent Ambo
49ee3e3b14 chore(tvix/eval): implement std::error::Error for tvix_eval::Error
This makes it easier to interface this error with other crates.

Change-Id: I4947ea6097608f8c0427fb94a819ef748d94ea4b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7711
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-02 22:24:43 +00:00
Vincent Ambo
86361f0f4a refactor(tvix/eval): remove extra Rc<..> around Value::Attrs
The `im::OrdMap` is already small and cheap to copy while sharing
memory, so this is not required anymore.

Only the `KV` variant may have slightly larger content, but in
practice this doesn't seem to make a difference when comparing the two
variants and this one is less complicated.

Change-Id: I64a563b209a2444125653777551373cb2989ca7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7677
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
91465dc78e refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrs
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory
between different iterations of a map.

This slightly speeds up eval, but not significantly. Future work might
include benchmarking whether using a `HashMap` and only ordering in
cases where order is actually required would help.

This switches to a fork of `im` that fixes some bugs with its OrdMap
implementation.

Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
610c44ec1e refactor(tvix/eval): use im::Vector directly where possible
The conversion from im::Vector -> Vec is cheaper for NixList
construction (of course), so where possible we should make use of
that.

This updates most builtins dealing with lists to use Vector directly,
and marks the function constructing NixList from Vec as deprecated so
that we get appropriate warnings in places where it's still in use.

These places are currently inside of JSON serialisation logic which is
in flux right now, so lets leave them as-is until it's stabilised.

Change-Id: I037f12a2800f2576db4d9526bd935efd079163f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7671
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 16:33:14 +00:00
Vincent Ambo
5d73c06b1a refactor(tvix/eval): use im::Vector for NixList representation
This is a persistent, structurally sharing data structure which is
more efficient in some of our use-cases. I have verified the
efficiency improvement using `hyperfine` repeatedly over expressions
on nixpkgs.

Lists are not the most performance-critical structure in Nix (that
would be attribute sets), but we can already see a small (~5-10%)
improvement.

Note that there are a handful of cases where we still go via `Vec`
that need to be fixed, most notable for `builtins.sort` which can not
currently be implemented directly using `im::Vector` because of a
restrictive type bound.

Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-29 12:27:59 +00:00
sterni
56555b211e docs(tvix/eval): sketch in place list/attr set update idea
Change-Id: Ic7debbd8cbd3acdf5f3947288f2aa2964bd163a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7660
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 09:52:17 +00:00
Vincent Ambo
71174f6626 fix(tvix/eval): fix current clippy warnings
It's been a while since the last time, so quite a lot of stuff has
accumulated here.

Change-Id: I0762827c197b30a917ff470fd8ae8f220f6ba247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7597
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-25 18:25:06 +00:00
Adam Joseph
67d508f2ec refactor(tvix/eval): non-recursive thunk forcing
Introduces continuation-passing-based trampolining of thunk forcing to
avoid recursing when forcing deeply nested expressions.

This is required for evaluating large expressions.

This change was extracted out of cl/7362.

Co-authored-by: Vincent Ambo <tazjin@tvl.su>
Co-authored-by: Griffin Smith <grfn@gws.fyi>
Change-Id: Ifc1747e712663684b2fff53095de62b8459a47f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7551
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-25 18:17:46 +00:00
Ryan Lahfa
4cda236c0c feat(tvix/eval): implement From<f64> for Value
Change-Id: I287282a195d6f752260242739332b2357791974a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7625
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-25 15:47:40 +00:00
Vincent Ambo
712f1167ad feat(tvix/eval): display function names in documentation
... if they are known. We currently do not propagate names correctly
for curried functions.

Change-Id: I19d57fb30a5c0000ccdf690b91076f6b2191de23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7596
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-22 19:09:40 +00:00
Vincent Ambo
e306d1d1a1 feat(tvix/eval): add Value::explain method
This value creates a human-readable explanation of a value. This can
be used to implement documentation related functionality.

For some values, the amount of information displayed can be expanded
quite a bit.

Change-Id: Ie8c400feae909e7680af163596f99060262e4241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7592
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-22 19:09:40 +00:00
Vincent Ambo
ea7d63e177 feat(tvix/cli): implement NixCompatIO helper type
This type allows for temporarily compatibility with the C++ Nix store,
specifically (for now) it gives us the store directory used by Nix and
imports files the same way.

Change-Id: I4767794ef2863eba49661315c63c4e17de946d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7587
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-22 19:09:40 +00:00
Vincent Ambo
22ed59fabd chore(tvix/eval): fix a broken comment
Change-Id: I0a6edb51685f94d4712089ae805170da3fb7faae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7609
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-21 23:29:26 +00:00
Vincent Ambo
908cebf35c fix(tvix/builtin-macros): parse multi-line docstrings correctly
Having a multi-line docstring yields multiple doc-attributes in order,
however we were previously discarding all but the first one.

This reduces them into a single string instead, which can then be
displayed as multi-line documentation.

Change-Id: I1f237956cdea2e4c746d3f13744e0373c1c645a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7594
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-21 23:05:22 +00:00
Vincent Ambo
270b1084e8 feat(tvix/eval): use EvalIO::import_path when coercing paths
This "ties the knot" of importing files into a store when referring
to them through path literals, e.g. inside of strings.

I'm not yet sure if this interface is sufficient for
builtins.path (which we haven't implemented at all yet), but it's
enough to wire up eval & store initially.

In the default implementations nothing interesting happens in this
function at all.

Change-Id: Ie01ff4161617d1e743a68dbd1a5e54c1b40c0990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7582
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:59:18 +00:00
Vincent Ambo
edd13573f5 refactor(tvix/eval): use light spans in builtins.import
Change-Id: I05732073155b430575babb6f076bf465aef98857
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7581
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:59:18 +00:00