Commit graph

61 commits

Author SHA1 Message Date
Aspen Smith
d0ab3c8d15 feat(tvix): Switch to jemalloc
Switch tvix to using jemalloc as the default global allocator on
supported (eg, non-msvc) platforms. This gives a pretty reasonable
performance boost basically for free:

int                     time:   [24.361 µs 24.386 µs 24.418 µs]
                        change: [-19.355% -18.859% -18.527%] (p = 0.00 < 0.05)
                        Performance has improved.

merge small attrs       time:   [37.201 µs 37.328 µs 37.442 µs]
                        change: [-24.609% -24.266% -23.982%] (p = 0.00 < 0.05)
                        Performance has improved.

merge large attrs with small attrs
                        time:   [20.030 ms 20.135 ms 20.251 ms]
                        change: [-4.2811% -3.2549% -2.3807%] (p = 0.00 < 0.05)
                        Performance has improved.

hello outpath           time:   [967.91 ms 974.07 ms 983.33 ms]
                        change: [-5.5008% -4.4552% -3.4099%] (p = 0.00 < 0.05)
                        Performance has improved.

Change-Id: I6c6e6e3295ec2fca01ea28dc37bcb201cd811767
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10851
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-06 10:29:33 +00:00
Profpatsch
5b2ba0efa1 refactor(tvix/eval): rewrite xml emitter to be simple-stupid
In order to be compatible with the nix XML generator, it’s easier to
generate the XML directly, instead of going through a library which we
have to bend to do what we need.

Removes dependency on `xml-rs`, which came with a full XML parser that
we didn’t use. Only takes a tiny bit of code for the XML escaping,
somewhat simplified.

I add a little escaping value, to make sure we have the same behaviour
as nix proper.

Interestingly enough, we never need to escape XML attribute names,
because the `builtins.toXML` format encodes user-defined values as
attribute keys only. So we only escape attribute values.

Fixes: https://b.tvl.fyi/issues/399
Change-Id: If4d407d324864b3bb9aa3160e2ec6889f7727127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11697
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
2024-05-22 10:32:55 +00:00
Florian Klink
839c971a0f docs(tvix/eval): remove link to cargo manifest reference
We don't have it in all Cargo.toml files either.

Change-Id: Ifcb333a39603a7e402ab0068e0f0f1b5af769e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11388
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
f8800ba189 chore(tvix): bump rstest to 0.19.0
Change-Id: Ib2f5e84fdb8be1210b3507da67d4fe84f061651e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11387
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-12 22:16:56 +00:00
Padraic-O-Mhuiris
5c3065b43a feat(tvix/eval): implement builtins.hashString
Implements md5, sha1, sha256 and sha512 using the related crates from
the RustCrypto hashes project (https://github.com/RustCrypto/hashes)

Change-Id: I00730dea44ec9ef85309edc27addab0ae88814b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11005
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-02-23 16:04:35 +00:00
Aspen Smith
201173afac fix(tvix): Represent strings as byte arrays
C++ nix uses C-style zero-terminated char pointers to represent strings
internally - however, up to this point, tvix has used Rust `String` and
`str` for string values. Since those are required to be valid utf-8, we
haven't been able to properly represent all the string values that Nix
supports.

To fix that, this change converts the internal representation of the
NixString struct from `Box<str>` to `BString`, from the `bstr` crate -
this is a wrapper around a `Vec<u8>` with extra functions for treating
that byte vector as a "morally string-like" value, which is basically
exactly what we need.

Since this changes a pretty fundamental assumption about a pretty core
type, there are a *lot* of changes in a lot of places to make this work,
but I've tried to keep the general philosophy and intent of most of the
code in most places intact. Most notably, there's nothing that's been
done to make the derivation stuff in //tvix/glue work with non-utf8
strings everywhere, instead opting to just convert to String/str when
passing things into that - there *might* be something to be done there,
but I don't know what the rules should be and I don't want to figure
them out in this change.

To deal with OS-native paths in a way that also works in WASM for
tvixbolt, this also adds a dependency on the "os_str_bytes" crate.

Fixes: b/189
Fixes: b/337
Change-Id: I5e6eb29c62f47dd91af954f5e12bfc3d186f5526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10200
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-01-31 14:51:49 +00:00
Ryan Lahfa
75cc52ddb1 fix(tvix/eval): getContext merges underlying values
Previously, we were assembling very naively an attribute set composed of context we saw.

But it was forgetting that `"${drv}${drv.drvPath}"` would contain 2 contexts with the same key, but
with different values, one with `outputs = [ "out" ];` and `allOutputs = true;`.

Following this reasoning and comparing with what Nix does, we ought to merge underlying values systematically.

Hence, I bring `itertools` to perform a group by on the key and merge everything on the fly, it's not
beautiful but it's the best I could find, notice that I don't use
`group_by` but I talk about group by, that is, because `group_by` is a
`group_by_consecutive`, see
https://github.com/rust-itertools/itertools/issues/374.

Initially, I tried to do it without a `into_grouping_map_by`, it was akin to assemble the final `NixAttrs` directly,
it was less readable and harder to pull out because we don't have a lot of in-place mutable functions on
our data structures.

Change-Id: I9933c9bd88ffe04de50dda14f21879b60d8b8cd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10620
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-17 17:25:24 +00:00
Ryan Lahfa
bc8fb825c7 feat(tvix/eval): move away from test_generator to rstest
`test-generator` has not been updated in the past 2 years.
`rstest` has not been updated in the past 5 months.

This is an improvement in the maintenance state… I guess?
We get also new features, it changes the name of the tests with numbers too.

Change-Id: I5376104c7704f525dba7524da78daa09867cc669
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10623
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 07:34:02 +00:00
Florian Klink
c9c95f4ef3 chore(tvix): bump itertools
Change-Id: I2c798a30d3e69789c7d9120422bfdd67b1560dd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10565
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-06 15:33:12 +00:00
Florian Klink
01a3c87a3a chore(tvix): bump criterion to 0.5
Change-Id: I28904ca23437b4bb745c0eb1f4eb9ae33e09eb5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10244
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 03:59:30 +00:00
Vincent Ambo
f325456cba chore(tvix): bump proptest dependency
This *might* contain a fix for a clippy lint thrown by that crate.

Relates to b/321.

Change-Id: Ia7ebd3e26e0feb8bcc7a6c811b1e583f9016fd9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9966
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 20:28:38 +00:00
Florian Klink
269ab866f1 refactor(tvix/eval/tests): migrate to tempfile
tempdir pulls in remove_dir_all 0.5.3 with
https://rustsec.org/advisories/RUSTSEC-2023-0018.html, and we use
tempfile everywhere else too, so let's just migrate to that.

Change-Id: I735ade7b65e12fc26e3d43ca95fcfa07fcc64642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9565
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-08 22:47:33 +00:00
Florian Klink
432222f098 feat(tvix/store/proto): use Bytes instead of Vec<u8>
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our
bytes field cheaper to clone.

It's a bit annoying to configure due to
https://github.com/hyperium/tonic/issues/908, but the workaround does
get the job done.

Change-Id: I25714600b041bb5432d3adf5859b151e72b12778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-22 09:03:02 +00:00
Florian Klink
6a30eb69e0 chore(tvix/*): bump to smol_str 0.2.0
Change-Id: Ic9ac1b6fecb564eafb41b265bf317cd385fdc170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8560
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-11 18:19:20 +00:00
Vincent Ambo
b3f8d66a6a chore(tvix/eval): prune some dependencies & features
* We no longer need backtrace-on-stack-overflow, as we no longer
  overflow the stack with the recent eval refactorings. This was weird
  voodoo anyways, introduced earlier to debug some cases where stack
  overflows occured.

* default features of genawaiter crate are not needed, as we don't use
  their proc macros

Change-Id: I346fc5a18d7f117ee805909a8be8f535b96be76c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8263
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-13 20:30:59 +00:00
Vincent Ambo
2696839770 feat(tvix/eval): introduce generators module
This module contains the request/response types for generators
requesting actions from the VM.

For most of these, an async helper function is added that will be used
inside of generator functions to make use of these requests/responses
instead of constructing them directly.

Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-07 22:04:59 +00:00
Florian Klink
e0b05c0fa6 feat(tvix/eval): implement builtins.fromTOML
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML
testcase from nix_tests. It uses the `toml` crate, and the serde
integration it brings with it.

Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 07:49:44 +00:00
Florian Klink
1facd889bb feat(tvix/eval): use lexical-core to format float
Apparently our naive implementation of float formatting, which simply
used {:.5}, and trimmed trailing "0" strings not sufficient.

It wrongly trimmed numbers with zeroes but no decimal point, like
`10000` got trimmed to `1`.

Nix uses `std::to_string` on the double, which according to
https://en.cppreference.com/w/cpp/string/basic_string/to_string
is equivalent to `std::sprintf(buf, "%f", value)`.

https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated
like this:

> Precision specifies the exact number of digits to appear after
> the decimal point character. The default precision is 6. In the
> alternative implementation decimal point character is written even if
> no digits follow it. For infinity and not-a-number conversion style
> see notes.

This doesn't seem to be the case though, and Nix uses scientific
notation in some cases.

There's a whole bunch of strategies to determine which is a more compact
notation, and which notation should be used for a given number.
https://github.com/rust-lang/rust/issues/24556 provides some pointers
into various rabbit holes for those interested.

This gist seems to be that currently a different formatting is not
exposed in rust directly, at least not for public consumption.

There is the
[lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate
though, which provides a way to format floats with various strategies
and formats.

Change our implementation of `TotalDisplay` for the `Value::Float` case
to use that. We still need to do some post-processing, because Nix
always adds the sign in scientific notation (and there's no way to
configure lexical-core to do that), and lexical-core in some cases keeps
the trailing zeros.

Even with all that in place, there as a difference in `eval-okay-
fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated
the fixture to a less problematic number.

With this, the testsuite passes again, and does for the upcoming CL
introducing builtins.fromTOML, and enabling the nix testsuite bits for
it, too.

Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-25 07:49:44 +00:00
Vincent Ambo
940251b87f refactor(tvix/value): use proptest strategies from imbl crate
Instead of going through Vec/BTreeMap for generating our internal
types, use the proptest strategies from imbl.

The one thing I couldn't figure out in the previous implementation is
where the ranges/sizes of generated collections came from. The
strategies in proptest use different types (Range, with an unknown
default value, and SizeRange with 0..100). I've opted to specify
0..100 directly, but we can probably make it configurable.

Change-Id: I749bc4c703fe424099240cab822b1642e5216361
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7791
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:20:38 +00:00
Vincent Ambo
d365b09226 feat(tvix/eval): implement builtins.toXML
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-16 13:43:14 +00:00
Ryan Lahfa
805219a2fa feat(tvix/eval): implement serde::Deserialize for Value
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>

Change-Id: Ib6f7d1f4f4faac36b44f5f75cccc57bf912cf606
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7626
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-10 09:53:21 +00:00
Vincent Ambo
91465dc78e refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrs
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory
between different iterations of a map.

This slightly speeds up eval, but not significantly. Future work might
include benchmarking whether using a `HashMap` and only ordering in
cases where order is actually required would help.

This switches to a fork of `im` that fixes some bugs with its OrdMap
implementation.

Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
5d73c06b1a refactor(tvix/eval): use im::Vector for NixList representation
This is a persistent, structurally sharing data structure which is
more efficient in some of our use-cases. I have verified the
efficiency improvement using `hyperfine` repeatedly over expressions
on nixpkgs.

Lists are not the most performance-critical structure in Nix (that
would be attribute sets), but we can already see a small (~5-10%)
improvement.

Note that there are a handful of cases where we still go via `Vec`
that need to be fixed, most notable for `builtins.sort` which can not
currently be implemented directly using `im::Vector` because of a
restrictive type bound.

Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-29 12:27:59 +00:00
Vincent Ambo
9d6ee5b6a6 fix(tvix/eval): use test-generator fork that supports workspaces
This should make no difference in Nix builds, but allows running tests
locally again with `cargo test` for //tvix/eval.

Change-Id: I97d61840143d5c14db61d5862781bf635f9a28e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7590
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 13:23:38 +00:00
Vincent Ambo
257c67f204 chore(tvix): upgrade to clap 4.0
In //tvix/eval:

* criterion bumped to 4.0, which at least depends on clap 3.x instead
  of 2.x, which is less incompatible

In //tvix/cli:

* no changes required

In //tvix/nix_cli:

* some minor changes for compatibility with clap 4.0, no functionality
  changes

Change-Id: If793f64b59fcaa2402d3d483ddbab4092f32df03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7588
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
d9e2bec953 refactor(tvix): split binary (REPL etc.) out from evaluator library
The tvix-eval project is independent from any *uses* of the evaluator,
such as the tvix-repl.

This functionality has been split out into a separate "tvix-cli"
crate. Note that this doesn't have to mean that this CLI crate is the
"final" CLI crate for tvix, the point of this is not "getting the CLI
structure right" but rather "getting the evaluator structure right".

This reshuffling is part of restructuring the way that functionality
like store communication is injected into language evaluation.

Note that at this commit the new CLI crate is not at feature-parity.

Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:09:29 +00:00
Vincent Ambo
d9d627cdf0 refactor(tvix): share a Cargo.lock file between Rust projects
This relates to the (abandoned) cl/7256.

Introduces a Cargo workspace at //tvix that is primarily intended to
be used as a workaround for the annoying Nix+Rust tooling while having
a consistent set of dependencies.

This is driven in part by a desire to adopt crate2nix and get more
granular Nix builds for Tvix's Rust projects, and in part by a need to
split //tvix/eval into something providing the CLI (REPL etc.), and a
library providing eval, without significantly altering the structure
of build targets.

To accomplish this the workspace has been designed to allow projects
to remain independent build targets. I want to avoid lumping all the
projects together - something like //tvix/eval should always be
independent of other parts of tvix.

A helper function in //tvix/default.nix lets downstream naersk
projects construct a sparse root for the project which combines the
workspace's `Cargo.lock` with the project's own `Cargo.toml`.

Note that cargo commands in the workspace itself require the build
dependencies of _all_ projects to be present, which is currently a bit
annoying to accomplish.

This introduces some breakage:

1. It breaks usage of rust-analyser without being in a shell with the
   dependencies of *all* Tvix projects, as it is not capable of
   respecting only the subset of dependencies for a part of the
   workspace.

2. It is no longer possible to run tests using `cargo test`, as the
   test generation crate we use does not work with workspaces:

   https://github.com/frehberg/test-generator/issues/6

   This still works in the Nix build as we construct a Cargo project
   that looks like it's not in a workspace there. Until somebody fixes
   that crate / writes a new macro / does something else with the test
   suite, the way to run the tests is through the Nix build.

Long-term we'll probably want to get rid of cargo completely, it's
just a big wart and most tooling works without it if correctly
configured, but we don't have time for that now.

Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-15 17:26:45 +00:00
Vincent Ambo
f282984c25 fix(tvix/eval): use rnix-parser from crates.io
A few weeks ago, oberblastmeister did a release to crates.io so we can
stop importing it via GitHub.

Change-Id: I9d5fa5cd281685779c71b12fed45ed201a1db17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7532
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-06 18:29:50 +00:00
Adam Joseph
56c00df710 feat(tvix/eval): use backtrace-on-stack-overflow crate
The backtrace-on-stack-overflow create provides best-effort stack
traces when a stack overflow happens.  Since it's running on the
(usually tiny) signal alternate stack this isn't easy.

This is guarded by a new `backtrace_overflow` feature flag and never
enabled (even if that feature is selected) for release builds.  This
is strictly for debugging; there's crazy unsafe voodoo in there.

   https://lib.rs/crates/backtrace-on-stack-overflow

Example output:

```
Stack Overflow:
   0: backtrace_on_stack_overflow::handle_sigsegv
             at /home/amjoseph/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-on-stack-overflow-0.2.0/src/lib.rs:93:40
   1: <unknown>
   2: __rust_probestack
   3: tvix_eval::vm::VM::run_op
             at src/vm.rs:399
   4: tvix_eval::vm::VM::run
             at src/vm.rs:388:23
   5: tvix_eval::vm::VM::enter_frame
             at src/vm.rs:360:22
   6: tvix_eval::value::thunk::Thunk::force
             at src/value/thunk.rs:116:25
   7: tvix_eval::vm::VM::run_op
             at src/vm.rs:801:37
   8: tvix_eval::vm::VM::run
             at src/vm.rs:388:23
   9: tvix_eval::vm::VM::enter_frame
             at src/vm.rs:360:22
  10: tvix_eval::value::thunk::Thunk::force
             at src/value/thunk.rs:116:25
...
```

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I1d8a2017f836be7bf91a2223e7adacb86fa1dbb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7354
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-26 19:41:43 +00:00
Adam Joseph
11874d3a71 chore(tvix/eval): postpone use of "dep:" for a bit
The "dep:" syntax in Cargo.toml is very new; crate2nix master has
support for it, but they have not yet made a release with this
update, and therefore the crate2nix in nixpkgs does not yet support
it.

Could we avoid using "dep:" for a few weeks to give crate2nix a
chance to release so I can bump the version in nixpkgs?  I've opened
an issue asking crate2nix to make a release:

  https://github.com/kolloch/crate2nix/issues/264

I propose that if they haven't acted within a month we stop waiting
and revert this at that time.

Change-Id: I999a72429db667bedf4b2cdba27cb63b3f3d9657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7350
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-24 20:11:46 +00:00
Griffin Smith
645d0c06e5 feat(tvix/eval): Add a proc-macro for defining builtins
Add a single new proc macro to a new proc-macro crate,
`tvix-eval-proc-macros` for defining an inline module containing nix
builtins, and automatically generating a function within that module
which returns a list of those builtins as `tvix_eval::value::Builtin`.

Change-Id: Ie4afae438914d2af93d15637151a49b4c68aa352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7198
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2022-11-08 13:42:37 +00:00
sterni
e3637136d1 test(tvix/eval): enable nix_tests by default
Since cl/7036 we have a mechanism for dealing with the nix_tests we do
not pass yet.

Change-Id: I246c52963ae7f2500253f4035a77d7006dd35307
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7049
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 02:28:38 +00:00
Vincent Ambo
0abc66ad91 feat(tvix/eval): add an AST pretty-printing module
This implements serde::Serialize for the rnix AST through a wrapper
type, and exposes a function for serialising the AST into
a (pretty-printed JSON) string representation.

This can be used to debug issues with the AST, and to display an AST
reprsentation in tools like tvixbolt.

Serialize is implemented manually because we don't own any of the
structs and the way to traverse them is not easily derived
automatically, and this is quite verbose. We might be able to condense
it a little bit, but at the same time it's also fairly straightforward.

Change-Id: I922df43cfc25636f3c8baee7944c75ade516055c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6943
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 12:26:56 +00:00
Griffin Smith
03a3189a3d feat(tvix/eval): Initial impl of builtins.match
Implement an *initial* version of builtins.match, using the rust `regex`
crate for regular expressions. The rust regex crate definitely has
different semantics than nix's regular expressions - but we'd like to
see how far we can get before the incompatibility starts to matter.

This consciously leaves out any sort of memo for compiled regular
expressions (which upstream nix also has) for the sake of expediency -
in the future we should implement that so we don't have to compile the
same regular expression multiple times.

Change-Id: I5b718635831ec83397940e417a9047c4342b6fa1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6989
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 20:35:22 +00:00
Griffin Smith
5eb89be682 feat(tvix/eval): Implement builtins.fromJSON
Using `serde_json` for parsing JSON here, plus an `impl FromJSON for
Value`. The latter is primarily to stay "dependency light" for now -
likely going with an actual serde `Deserialize` impl in the future is
going to be way better as it allows saving significantly on intermediary
allocations.

Change-Id: I152a0448ff7c87cf7ebaac927c38912b99de1c18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6920
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 20:35:22 +00:00
Vincent Ambo
c46025d520 fix(tvix/eval): support string identifiers in inherits
This updates rnix-parser to a version where inherits provide an
iterator over `ast::Attr` instead of `ast::Ident`, which mirrors the
behaviour of Nix (inherits can have (statically known) strings as
their identifiers).

This actually required some fairly significant code reshuffling in the
compiler, as there was an implicit assumption in many places that we
would have an `ast::Ident` node available when dealing with variable
access (which is then explicitly only not true in this case).

Change-Id: I12f1e786c0030c85107b1aa409bd49adb5465546
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6747
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-22 23:07:14 +00:00
Griffin Smith
9e16d70809 feat(tvix/eval): Sketch out impure builtins
Sketch out a new set of "impure" builtins, which supplement the existing
set of "pure" builtins but are gated behind a feature flag, which allows
them to be omitted by crates depending on tvix-eval that only want pure
evaluation, such as tvixbolt.

Change-Id: I2736017b5c9b4776bbba8758e108ec84887abd66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6655
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-22 01:36:44 +00:00
Griffin Smith
e720545e5b refactor(tvix/eval): use Clap for arg+env parsing
Refactor the environment variable and argument parsing for the tvix repl
to use Clap instead of doing things ad-hoc, and thread through options
obtained from environment variables via explicit arguments rather than
obtaining them from the environment as they're needed. This makes adding
more flags more sustainable, and also makes the binary fully
self-documenting, including supported env vars, via `--help`.

Change-Id: Ib1f6a0cd20056e8c9196760ff755fa5729667760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6653
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-18 22:08:43 +00:00
Vincent Ambo
b3cc4c8c2c chore(tvix/eval): bump rnix-parser to latest HEAD
This bumps rnix-parser to a commit that should be unaffected by the
Nix >= 2.4 bug that prevents it from cloning repositories with filters.

Change-Id: Ie01da95245ec6740fa889eb710819e512202f665
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6634
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-18 16:24:40 +00:00
Griffin Smith
84def7ef19 chore(tvix/eval): Add deps on proptest+test_strategy
It's proptest time!

Add deps on the proptest and test_strategy crates, gated behind a
new (default-enabled) "arbitrary" feature flag so that they don't affect
dependencies of things like tvixbolt that depend on tvix.

These are going in dependencies, not dev-dependencies, so that we can
impl Arbitrary for stuff outside of test modules (which will be
important for integration suites which want to run proptests)

Change-Id: I1613bd3ea9a835e22986ad4e59700e8736007963
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6624
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-17 19:02:59 +00:00
Griffin Smith
67e796b2e1 test(tvix/eval): Add the start of a nix oracle test suite
Add the start of a test suite that compares tvix eval results against
nix, using the string repr of the value as the comparison. This shells
out to a nix-instantiate binary, which is configurable as an environment
variable, to eval - there's some extra machinery there to setup a new
nix store as a tempdir to allow running this test inside the nix build
for tvix-eval itself.

Currently this has a macro that'll allow writing lots and lots of
hardcoded tests, but going forward I'm also going to be looking into
adding proptest-based generation of expressions to compare.

Change-Id: I9f4895fab1e668ed2b7dfd6f92f8c80de1bbb16b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6307
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-17 17:23:55 +00:00
Vincent Ambo
dea6d9b73b chore(tvix/eval): add dependency on codemap-diagnostic
This is a crate for source-span based error reporting. Since all of
our spans are already codemap spans, it is a good starting point.

We have to figure out quite a bit of logic for neat error printing;
later on if we want fancier presentation we might want to look at one
of the other libraries in this space like miette.

Change-Id: I4e28886af1ed199b7112d9dbf063c9f29b612bf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6531
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-13 10:54:14 +00:00
Vincent Ambo
8ee4d6d5db feat(tvix/eval): implement DisassemblingObserver for compiler
This type implements an observer that is called whenever the compiler
emits a chunk (after the toplevel, thunks, or lambdas) and prints the
output of the disassembler to its internal writer.

This replaces half of the uses of the `disassembler` feature, which
has been removed from the Cargo configuration.

Note that at this commit runtime tracing is not yet implemented as an
observer.

Change-Id: I7894ca1ba445761aba4ad51d98e4a7b6445f1aea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6449
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:10:10 +00:00
Vincent Ambo
8c64ebe074 chore(tvix/eval): gate REPL-only dependencies behind repl feature
With this change, it becomes possible to compile tvix-eval to
webassembly if the `repl` feature is disabled.

Change-Id: Icc0a059964cd0bea2054110c682d50fc5c87ec01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6446
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-09 10:48:10 +00:00
Vincent Ambo
18f8fecba4 chore(tvix/eval): bump all dependencies
Updated the rnix hash manually, and ran `cargo update` for the rest.

Change-Id: I457262625d648e25d745efa4d33ae44cb8f21326
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6375
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:08:56 +00:00
Vincent Ambo
39e960daa6 chore(tvix/eval): add codemap dependency
This will be used to track source spans when emitting bytecode.

The codemap is a data structure which tracks *all* the source files
visited by an evaluation, and makes it possible to represent locations
across all of the files using a simple span (i.e. pair of offsets).

When reporting errors, this even contains enough information to
reconstruct the rnix AST to create fancier reporting in certain cases
if desired.

Change-Id: I4ae98620b9b150fb5a389bd7f1e12670e3192c62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6374
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:08:56 +00:00
Vincent Ambo
727845645d chore(tvix/eval): add release-with-debug cargo profile
This lets us create a release build with debug info, for use with e.g.
perf + hotspot

Change-Id: I03897de36c872d318abf1332ca0c1aeabe344ec6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6362
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Griffin Smith
3270817b90 test(tvix/eval): Add attr merge benchmarks
Add a quick couple of benchmarks for merging attribute sets, large and
small.

Change-Id: I26940a9cf4e0d30e3d9eb07a7b8c366ca4072ca3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6286
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
2022-09-03 06:56:02 +00:00
Vincent Ambo
3bf487b98b chore(tvix/eval): bump rnix-parser to latest master
In this commit, the string interpolation parsing is identical to
nixpkgs which makes some of the upstream Nix tests for
interpolation-related weirdness pass.

Change-Id: I3a295cfdc404c32228a54846e6efd3c0dcee5842
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6233
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-01 17:41:22 +00:00
Vincent Ambo
813fb98470 refactor(tvix/eval): Upgrade to latest rnix-parser
Since the latest published version of rnix-parser on crates.io, the
crate has undergone major changes which are only available in the git
repository at the moment. This commit updates the compiler to this
newer version of rnix.

Most notably, the entire AST provided by rnix is now wrapped in the
AST type system. As a result of this traversal is much nicer in many
places, especially for things like nested attribute selection.

There are a handful of smaller features missing for full feature
parity with the previous version, especially handling of path
literals, but PRs for these already exist in rnix-parser.

Change-Id: Icde6d393067976549492b7d89c4cc49e5e575fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6231
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-01 17:41:22 +00:00