This trait is going to be used to abstract filesystem interactions in
Tvix.
For now, it only contains a `read_to_string` method that closely
mirrors `std::fs::read_to_string`.
As a first step, to see how this works in practice, we will thread
through only this function to the various relevant parts.
Two implementations are provided in tvix-eval itself: A dummy
implementation (which just returns ErrorKind::NotImplemented for all
operations), and a std implementation which delegates to `std`
functions.
Change-Id: Ied3e3bf4bd0e874dd84e166190e3873a0f923ddb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7565
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This type carries the information required for calculating a
span (i.e. the chunk and offset), instead of the span itself. The span
is then only calculated in cases where it is required (when throwing
errors).
This reduces the eval time for
`builtins.length (builtins.attrNames (import <nixpkgs> {}))` by *one
third*!
The data structure in chunks that carries span information reduces
in-memory size by trading off the speed of retrieving span
information. This is because the span information is only actually
required when throwing errors (or emitting warnings).
However, somewhere along the way we grew a dependency on carrying span
information in thunks (for correctly reporting error chains). Hitting
the code paths for span retrieval was expensive, and carrying the
spans in a different way would still be less cache-efficient. This
change is the best tradeoff I could come up with.
Refs: b/229.
Change-Id: I27d4c4b5c5f9be90ac47f2db61941e123a78a77b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7558
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Having thunks which, when forced, execute native Rust code rather
than interpreted opcodes lets us avoid having to bundle
`src/libexpr/primops/derivation.nix` like cppnix does by implementing
it in Rust instead.
Change-Id: If91d77a6736234321eee87ba4b4777eed5a3fe1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7450
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Instead of finding locals by doing 2x O(n) walks over the compiler's
locals list, use a secondary name-based index for resolving locals by
name.
Previously, almost 60% (!!) of eval time on some expressions over
nixpkgs was spent in `Local::has_name`. This function doesn't even
exist anymore now, and eval speed about doubles as a result.
Note that this doesn't exactly make the locals code easier to read,
but I'm also not sure what we can simplify in there in general.
This fixes b/227.
Change-Id: I29ce5eb9452b02d3b358c673e1f5cf8082e2fef9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7560
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
With this change, it should be possible to have both existing
use-cases (CLI & Tvixbolt) use the same API.
Change-Id: I2195264f08cc892177b559a28660dc5f98e48e41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7545
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This is required for passing through NIX_PATH from the CLI.
Change-Id: If129df79ef9c3ffab31408adb85679909276c4f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7544
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This should make no difference in Nix builds, but allows running tests
locally again with `cargo test` for //tvix/eval.
Change-Id: I97d61840143d5c14db61d5862781bf635f9a28e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7590
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In //tvix/eval:
* criterion bumped to 4.0, which at least depends on clap 3.x instead
of 2.x, which is less incompatible
In //tvix/cli:
* no changes required
In //tvix/nix_cli:
* some minor changes for compatibility with clap 4.0, no functionality
changes
Change-Id: If793f64b59fcaa2402d3d483ddbab4092f32df03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7588
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
The tvix-eval project is independent from any *uses* of the evaluator,
such as the tvix-repl.
This functionality has been split out into a separate "tvix-cli"
crate. Note that this doesn't have to mean that this CLI crate is the
"final" CLI crate for tvix, the point of this is not "getting the CLI
structure right" but rather "getting the evaluator structure right".
This reshuffling is part of restructuring the way that functionality
like store communication is injected into language evaluation.
Note that at this commit the new CLI crate is not at feature-parity.
Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This removes internal uses of the previous crate::eval module, which
is being removed.
Change-Id: I5fb3c53460a9c5381853d0258f9ed074ab23c630
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7543
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
A step towards something more like how I imagine the future public API
for tvix-eval. Please note that this is definitely not the final
version yet, but it's better than the previous API that either exposed
a side-effecting blackbox, or a very low-level "interface".
The basic idea is that an evaluation of some Nix code is requested by
a caller with various parameters, but not all callers are interested
in all of these parameters.
There are also some bits of information that are returned from an
evaluation that are not necessarily relevant to all callers.
To support this somewhat ergonomically, the API is built around an
`Evaluation` struct that is configured by the caller with the various
parameters and then "executed".
Change-Id: I71826f3897126898adc2873d31c44d3eaf5c2be0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7542
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Introduces granular dependency builds using crate2nix, bootstrapped
off the generated configuration from the newly introduced
workspace (see cl/7533).
This commit checks in the generated Cargo.nix file which can be
regenerated with a parameterless invocation of `crate2nix generate` in
`//tvix`. I tried generating this in IFD, but it turned out to be
harder than what seemed worthwhile for now.
In this setup, the various build targets for Rust projects end up
being attributes of the imported `Cargo.nix` file at the `tvix.crates`
attribute. These still lack configuration, however, which has been
fixed in the various `default.nix` files of individual projects.
Note that we (temporarily) lose the ability to build tvix-eval's
benchmarks in CI. I haven't figured out what magic incantation summons
them from the void again ...
The `eval-okay-readDir` tests from both test suites have been disabled
because they fail for unknown reasons when run in this new derivation.
Somebody will have to debug it!
Change-Id: I2014614ccb9c8951aedbd71df7966ca191a13695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7538
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This relates to the (abandoned) cl/7256.
Introduces a Cargo workspace at //tvix that is primarily intended to
be used as a workaround for the annoying Nix+Rust tooling while having
a consistent set of dependencies.
This is driven in part by a desire to adopt crate2nix and get more
granular Nix builds for Tvix's Rust projects, and in part by a need to
split //tvix/eval into something providing the CLI (REPL etc.), and a
library providing eval, without significantly altering the structure
of build targets.
To accomplish this the workspace has been designed to allow projects
to remain independent build targets. I want to avoid lumping all the
projects together - something like //tvix/eval should always be
independent of other parts of tvix.
A helper function in //tvix/default.nix lets downstream naersk
projects construct a sparse root for the project which combines the
workspace's `Cargo.lock` with the project's own `Cargo.toml`.
Note that cargo commands in the workspace itself require the build
dependencies of _all_ projects to be present, which is currently a bit
annoying to accomplish.
This introduces some breakage:
1. It breaks usage of rust-analyser without being in a shell with the
dependencies of *all* Tvix projects, as it is not capable of
respecting only the subset of dependencies for a part of the
workspace.
2. It is no longer possible to run tests using `cargo test`, as the
test generation crate we use does not work with workspaces:
https://github.com/frehberg/test-generator/issues/6
This still works in the Nix build as we construct a Cargo project
that looks like it's not in a workspace there. Until somebody fixes
that crate / writes a new macro / does something else with the test
suite, the way to run the tests is through the Nix build.
Long-term we'll probably want to get rid of cargo completely, it's
just a big wart and most tooling works without it if correctly
configured, but we don't have time for that now.
Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
A few weeks ago, oberblastmeister did a release to crates.io so we can
stop importing it via GitHub.
Change-Id: I9d5fa5cd281685779c71b12fed45ed201a1db17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7532
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Fixes b/212. Based on feedback in https://cl.tvl.fyi/c/depot/+/7492, all
uses of `NixAttrs::from_map` have been removed. Only `from_iter` and
`from_kv` remain.
Change-Id: I52e25f73018c2aa1843197427516b7a852503e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7500
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: IslandUsurper <lyle@menteeth.us>
Allows for the removal of some BTreeMap usage when constructing NixAttrs
by allowing any iterator over 2-tuples to build a NixAttrs. Some
instances of BTreeMap didn't have anything to do with making NixAttrs,
and some were just the best tool for the job, so they are left using the
old `from_map` interface.
Change-Id: I668ea600b0d93eae700a6b1861ac84502c968d78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7492
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Maybe counter-intuitively the inner elements of a list or the
attribute values of an attribute set will be forced despite
pointer equality (but only one layer deep).
Change-Id: I485d96452fb56f5fb342d39039c9137725b33d3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7371
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
This came up in the Nix Language channel today and I thought it
warranted a test case.
We did actually implement this correctly.
Change-Id: I4b37c92d06eb6e3a7f59ea3d10af38f2b0a93d53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7493
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Rust doesn't do tail-call elimination (still!) so the best we can
hope for here is to inline non-recursive invocations.
Change-Id: I78949967e48b006fcbf31786d8f6281cd122f36f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7360
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Before this, tvix was spending most of its time furiously re-parsing
and re-compiling nixpkgs, each time hoping to get a different result...
Change-Id: I1c0cfbf9af622c276275b1f2fb8d4e976f1b5533
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7361
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
With this change, the test introduced by cl/7370 passes.
Change-Id: Ie7d2f02a59d61151f14ebd328e6cfa5892cacfb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7375
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: Adam Joseph <adam@westernsemico.com>
This passes all the function/thunk-pointer-equality tests in
cl/7369.
Change-Id: Ib47535ba2fc77a4f1c2cc2fd23d3a879e21d8b4c
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7358
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The backtrace-on-stack-overflow create provides best-effort stack
traces when a stack overflow happens. Since it's running on the
(usually tiny) signal alternate stack this isn't easy.
This is guarded by a new `backtrace_overflow` feature flag and never
enabled (even if that feature is selected) for release builds. This
is strictly for debugging; there's crazy unsafe voodoo in there.
https://lib.rs/crates/backtrace-on-stack-overflow
Example output:
```
Stack Overflow:
0: backtrace_on_stack_overflow::handle_sigsegv
at /home/amjoseph/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-on-stack-overflow-0.2.0/src/lib.rs:93:40
1: <unknown>
2: __rust_probestack
3: tvix_eval::vm::VM::run_op
at src/vm.rs:399
4: tvix_eval::vm::VM::run
at src/vm.rs:388:23
5: tvix_eval::vm::VM::enter_frame
at src/vm.rs:360:22
6: tvix_eval::value::thunk::Thunk::force
at src/value/thunk.rs:116:25
7: tvix_eval::vm::VM::run_op
at src/vm.rs:801:37
8: tvix_eval::vm::VM::run
at src/vm.rs:388:23
9: tvix_eval::vm::VM::enter_frame
at src/vm.rs:360:22
10: tvix_eval::value::thunk::Thunk::force
at src/value/thunk.rs:116:25
...
```
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I1d8a2017f836be7bf91a2223e7adacb86fa1dbb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7354
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
See cl/7368
Change-Id: I97630994c3d65f4d16414a0da236ce000a5b6d33
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7374
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
See cl/7372; Nix equality semantics require the ability to track
pointer equality of upvalue-sets.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I82ba517499cf370189a80355e4e46a5caaab7153
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7373
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This test case checks two things:
* A sanity check that "pointer equality for functions" means not
just the lambda, but also the upvalues.
* To be pointer-equal, it is not enough for the upvalues to be
normal-form equal (i.e. `nix_eq()`-equal); the upvalues must be
*pointer*-equal. The second part of the test case checks for
this.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I4e59327a6f199b8212e97197b212e3c3934bb3f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7372
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
The "dep:" syntax in Cargo.toml is very new; crate2nix master has
support for it, but they have not yet made a release with this
update, and therefore the crate2nix in nixpkgs does not yet support
it.
Could we avoid using "dep:" for a few weeks to give crate2nix a
chance to release so I can bump the version in nixpkgs? I've opened
an issue asking crate2nix to make a release:
https://github.com/kolloch/crate2nix/issues/264
I propose that if they haven't acted within a month we stop waiting
and revert this at that time.
Change-Id: I999a72429db667bedf4b2cdba27cb63b3f3d9657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7350
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
When we start unrecursivifying (sp?) things, Rust's borrow checker
is going to be a headache; its magic only works when you use the CPU
stack as your call stack.
Fixing the borrow checker issues usually involves adding lots of
`clone()`s. Right now `NixList` is the only variant of `Value` that
isn't cheap to clone() -- all the others are either a wrapper around
Rc or else are of bounded size.
Note that this requires dropping the `DerefMut for NixList` instance
and using `Vec<Value>` instead in those situations.
Change-Id: I5a47df66855342aa2064f8f3cb7934ff422d26bd
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7359
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
When comparing Nix values for equality, an issue can occur where
recursive values contain thunks to themselves which causes borrow
errors when forcing them for comparison later down the line.
To work around this we clone the values for now. There might be some
optimisations possible like checking for thunk equality directly and
short-circuiting on that (we have to check what Nix does).
Change-Id: I7e75c992ea68f100058f52b4b46168da7d671994
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7314
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
When passing multiple arguments, every intermediate callable needs to
be forced as this is expected by the VM's call_value function.
Also adds a debug assertion for this which makes it easier to spot
exactly what went wrong.
Change-Id: I3aa519cb6cdaab713bd18282bef901c4cd77c535
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7312
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This function covers builtins.genericClosure, seemingly including
weird behaviour around the order in which the work set is processed.
For some reason, in C++ Nix the test expectation is written in XML
which we do not yet support, so I have created a new expectation file
using `nix-instantiate --eval --strict` on the file (yes, using C++
Nix).
Change-Id: Id90e7117d120dc66d963a51083c4d8e8f2d9f181
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7311
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This implementation closely follows the original implementation in
Nix, including the use of an equality-based "set" structure to track
keys that have already been processed.
Note that this test does not yet enable the `notyetpassing` test for
builtins.genericClosure because (for as of yet unknown reasons) this
test compares against XML output (however, evaluating the test case
actually does work).
This takes us one step closer to nixpkgs eval.
This commit was written somewhere in the North Sea.
Co-Authored-By: Griffin Smith <root@gws.fyi>
Change-Id: I450a866e6f2888b27c2fe7c7f77ce0f79bfe3e6c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7310
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Both //tvix/eval and //tvix/nix_cli have need to for rust tooling available
in $PATH.
Move this one level up, so it's accessible in all subdirectories.
Change-Id: I0763bbe9cefdc962f3a8f86c51e8f67cde8b4b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7248
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This detects if the second argument of a division is a zero (either as integer
or as float). If so, an error message is displayed.
This fixes b/219.
Change-Id: I50203d14a71482bc757832a2c8dee08eb7d35c49
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7258
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Add a new `documentation: Option<&'static str>` field to Builtin, and
populate it in the `#[builtins]` macro with the docstring of the builtin
function, if any.
Change-Id: Ic68fdf9b314d15a780731974234e2ae43f6a44b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7205
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Refactor the arguments of a Builtin to be a vec of a new BuiltinArgument
struct, which contains the old strictness boolean and also a static
`name` str - this is automatically determined via the ident for the
corresponding function argument in the proc-macro case, and passed in in
the cases where we're still manually calling Builtin::new.
Currently this name is unused, but in the future this can be used as
part of a documentation system for builtins.
Change-Id: Ib9dadb15b69bf8c9ea1983a4f4f197294a2394a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7204
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Some new top-level re-exports (specifically VM, Builtin, and ErrorKind)
were added to lib.rs in tvix/eval to allow the builtin-macros tests to
work - we should be clear which of these are part of the public
interface (I think it's reasonable for ErrorKind to be) and which
aren't (specifically I'm not sure VM and Builtin necessarily should be,
at least yet).
Change-Id: I3bbeaa63cdda9227224cd3bc298a9bb8da4deb7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7203
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Similar to what we did with pure builtins, define the impure builtins
within a module at the top-level using the new #[builtins] attribute
macro
Change-Id: Ie5d5135d00bb65e651531df6eadba642cd4eb08e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7202
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Break out all pure builtin functions to top-level functions defined
within the `pure_builtins` module in `builtins/mod.rs`.
Change-Id: I9a10660446d557b1a86da4c45a463e9a1a9b4f2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7201
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Mostly as a proof-of-concept of the new proc-macros for defining
builtins, define a single builtin (the first in the list, `abort`) at
the top-level of a child module within builtins/mod.rs, and add it to
the list of builtins returned from `pure_builtins`.
If this works nicely, we can start breaking out the rest of the builtins
into the top-level too, in addition to introducing additional sets of
builtins (to differentiate between pure and impure builtins).
Change-Id: I5bdd57c57fecf8d63c9fed4fc6b1460f533b20f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7199
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Add a single new proc macro to a new proc-macro crate,
`tvix-eval-proc-macros` for defining an inline module containing nix
builtins, and automatically generating a function within that module
which returns a list of those builtins as `tvix_eval::value::Builtin`.
Change-Id: Ie4afae438914d2af93d15637151a49b4c68aa352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7198
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
This commit adds a markdown document which explains how the
thread-local VM infrastructure works, in case it is useful in the
future.
Change-Id: Id10e32a9e3c5fa38a15d4bec9800f7234c59234a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7193
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
This is more generally useful than just inside the VM, until it is
stabilised in Rust itself.
Change-Id: Id9aa3d5b533ff38e3d2c6b85ad484394fdd05dcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7186
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
This fixes a mistake I made in d978b556e6.
Change-Id: I88db697105a7149e9785f6aface03bff68566d2b
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7085
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Scope_depth and with_stack_depth were being reset to zero for nested
function abstractions. Fortunately nothing depends on them being
computed correctly in these cases, but it sure was confusing.
Change-Id: I59980b6a5aff043f60079f97211220b0086eb97d
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7091
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
It is very confusing that this opcode is called DataLocalIdx, but it
carries a StackIdx rather than a LocalIdx. It seems like this
really ought to be called DataStackIdx, but maybe I've
misunderstood; if so please explain it to me.
Change-Id: I91f6ffa759412beef0b91d3c19ec0d873fe51b99
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7088
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This implements builtins.split, and passes eval-okay-regex-split.nix
(which is moved out of notyetpassing).
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ieb0975da2058966c697ee0e2f5b3f26ccabfae57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7143
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
We have to be careful implementing `builtins.groupBy`, since the
list may contain thunks, and tvix's to_xxx() functions do not work
on thunks.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I182b6fc2d4296f864ed16744ef70b153e8e6978a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7039
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The impl Display for NixAttrs needs to wrap double quotes around any
keys which are not valid Nix identifiers. This commit does that,
and adds a test (which fails prior to this commit and passes after
this commit).
Change-Id: Ie31ce91e8637cb27073f23f115db81feefdc6424
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7084
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The variable name `local_idx` is used here for a StackIdx, which invites
confusion.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I2e22db90acdc0d29586ee5b72ea18d42d93badcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7086
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
If self.depth > other.depth then self is deeper than other, so self
is *below* other, not above it. Let's just inline the function.
Change-Id: I8dda3d90cbc86c8a6fa01bc4a5e506a2e403bd20
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7090
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
It isn't possible to implement PartialEq properly for Value, because
any sensible implementation needs to force() thunks, which cannot be
done without a `&mut VM`.
The existing derive(PartialEq) has false negatives, which caused the
bug which cl/7142 fixed. Fortunately that bug was easy to find, but
a silent false negative deep within the bowels of nixpkgs could be a
real nightmare to hunt down.
Let's just remove the PartialEq impl for Value, and the other
derive(PartialEq)'s that depend on it.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Iacd3726fefc7fc1edadcd7e9b586e04cf8466775
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7144
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I think we should bring this into $PATH too.
Change-Id: Ie31ac558355b7c4ed9dcd3dd60e1b03f141d1178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7166
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The current implementation of nix_eq will force one level of thunks
and then switch to the (non-forcing) rust Eq::eq() method. This
gives incorrect results for lists-of-thunks.
This commit changes nix_eq() to be recursive.
A regression test (which fails prior to this commit) is included.
This fix also causes nix_tests/eval-okay-fromjson.nix to pass, so it
is moved out of notyetpassing.
Change-Id: I655fd7a5294208a7b39df8e2c3c12a8b9768292f
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7142
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We're getting close to the finish line, folks.
I went through the list of builtins and there are only 33 that
remain unimplemented. I've marked them, and indicated which are
ready to be implemented vs which are waiting for other things.
We can delete this column from the table once everything is
implemented.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Idfaef93283536288b12e59aef5c3e1cd139bd133
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7140
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
I believe that the currentTime, findFile, hashFile, pathExists,
readDir, path (unless ?sha256), and readFile builtins are impure.
This commit marks them as such in docs/builtins.md.
Change-Id: Ib1b59fe643dde73cb2b00050b4ef9d3401ad22eb
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7139
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is a bit tricky because the comparator can throw errors, so we
need to propagate them out if they exist and try to avoid sorting
forever by returning a reasonable ordering in this case (as
short-circuiting is not available).
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: Icae1d30f43ec1ae64b2ba51e73ee467605686792
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7072
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Lists are compared lexicographically in C++ nix as of [0], and our
updated nix test suites depend on this. This implements comparison of
list values in `Value::nix_cmp` using a very similar algorithm to what
C++ does - similarly to there, this requires passing in the VM so we can
force thunks in the list elements as we go.
[0]: 09471d2680#
Change-Id: I5d8bb07f90647a1fec83f775243e21af856afbb1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7070
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
CL/7034 looks great, except that for a length-N target string it
will perform N deep copies of each of the from and to-lists. Let's
use references instead of clones.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Icd341213a9f0e728f9c8453cec6d23af5e1dea91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7095
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I assumed that AttrsRep::KV represented attrsets with a single
attribute as a Key-Value pair. That is not the case. Let's warn
other people about this.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ie3d2765fcc1ab705c153ab94ffe77bbd6d4ab39e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7093
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Rustc uses wasm32-unknown-unknown, which is rejected by config.sub,
for wasm-in-the-browser environments. Rustc should be using
wasm32-unknown-none, which config.sub accepts. Hopefully the rustc
people will change their triple before stabilising this triple. In
the meantime, we fix it here in order to unbreak tvixbolt.
https://doc.rust-lang.org/beta/nightly-rustc/rustc_target/spec/wasm32_unknown_unknown/index.html
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I941fd8d6f3db4e249901772fd79321ad88cd9cc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7107
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This commit contains two search-and-replace renames which are broken
out from I04131501029772f30e28da8281d864427685097f in order to
reduce the noise in that CL:
- `is_thunk -> is_suspended_thunk`, since there are now
OpThunkClosure and OpThunkSuspended
- `compile_lambda_or_thunk` -> `compile_lambda_or_suspension`
Change-Id: I7cc5bbb75ef6605e3428c7be27e812f41a10c127
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7037
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
CL/6867 added support for builtins.import, which required a cyclic
reference import->globals->builtins->import. This was implemented
using a RefCell, which makes it possible to mutate the builtins
during evaluation. The commit message for CL/6867 expressed a
desire to eliminate this possibility:
This opens up a potentially dangerous footgun in which we could
mutate the builtins at runtime leading to different compiler
invocations seeing different builtins, so it'd be nice to have
some kind of "finalised" status for them or some such, but I'm not
sure how to represent that atm.
This CL replaces the RefCell with Rc::new_cyclic(), making the
globals/builtins immutable once again. At VM runtime (once opcodes
start executing) everything is the same as before this CL, except
that the Rc<RefCell<>> introduced by CL/6867 is turned into an
rc::Weak<>.
The function passed to Rc::new_cyclic works very similarly to
overlays in nixpkgs: a function takes its own result as an argument.
However instead of laziness "breaking the cycle", Rust's
Rc::new_cyclic() instead uses an rc::Weak. This is done to prevent
memory leaks rather than divergence.
This CL also resolves the following TODO from CL/6867:
// TODO: encapsulate this import weirdness in builtins
The main disadvantage of this CL is the fact that the VM now must
ensure that it holds a strong reference to the globals while a
program is executing; failure to do so will cause a panic when the
weak reference in the builtins is upgrade()d.
In theory it should be possible to create strong reference cycles
the same way Rc::new_cyclic() creates weak cycles, but these cycles
would cause a permanent memory leak -- without either an rc::Weak or
RefCell there is no way to break the cycle. At some point we will
have to implement some form of cycle collection; whatever library we
choose for that purpose is likely to provide an "immutable strong
reference cycle" primitive similar to Rc::new_cyclic(), and we
should be able to simply drop it in.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I34bb5821628eb97e426bdb880b02e2097402adb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7097
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds a comment noting that StackIdx is an offset relative to
the base of the current CallFrame, whereas UpvalueIdx is an absolute
index into the upvalues array.
It also removes the confusing mention of StackIdx in the descriptive
comment for LocalIdx. They index into totally different structures;
one exists at runtime and the other exists at compile time.
Change-Id: Ib932b1b0679734c15001e8c5c95a08293fa016b4
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7017
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds a function NixList::force_elements() which forces each
element of the list shallowly. This behavior is needed for
`builtins.replaceStrings`, and probably a few other builtins as
well.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I3f0681acbbfe50e781b5f07b6a441647f5e6f8da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7094
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Unfortunately we have to mangle test case filenames into rust-valid
symbols, since test-generator doesn't use `r#"..."` (deliberately?).
This means that when a test fails, there's nothing on the console
you can copy-and-paste in order to view/edit the code of the failing
test case.
This commit (partially) fixes it by including the unmangled name in
the panic!() string. However failures due to panic!()s inside the
vm (including deliberate panics due to panic!()-debugging) still
won't display an unmangled filename.
Maybe we should reconsider the use of test-generator?
Change-Id: I2208a859ffab1264f17f48fd303ff5e19675967e
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7092
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Rather than implementing all of the interesting semantics of value
comparison with a macro bound to the VM, implement the bulk of the logic
with a method on Value itself that returns an Ordering, and then use the
macro to implement the comparison against that Ordering. This has no
functional change, but paves the way to implementing lexicographic
comparison of list values, which is supported in the latest version of
upstream nix.
Change-Id: I8af1a020b41577021af5939f5edc160c407d4a9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7069
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
I played around a little bit with doing this in-place, but ended up
going with this perhaps slightly clone-heavy approach for now because
ideally most clones on Value are cheap - but later we should benchmark
alternate approaches that get to reuse allocations better if necessary
or possible.
Change-Id: If998eb2056cedefdf2fb480b0568ac8329ccfc44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7068
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The last bump in langVersion (5->6) in C++ Nix was due to making lists
comparable (commit `09471d2680292af48b2788108de56a8da755d661`), which
we support in Tvix with cl/7070.
Change-Id: Id3beed5150b8fb6e0a46a4d1b7e3942022a65346
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7074
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This commit implements builtins.currentSystem, by capturing the
cargo environment variable `TARGET` and exposing it to rustc as
`TVIX_CURRENT_SYSTEM` so it can be inserted into the source code
using `env!()`.
The resulting value needs to be massaged a bit, since it is an "LLVM
triple". The current code should work for all the platforms for
which cppnix works (thanks qyliss for generating the list!). It
does *not* reject all of the triples that cppnix's configure.ac
rejects -- it is much more forgiving. We can tighten this up in a
future commit.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I947f504b2af5a7fee8cf0cb301421d2fc9174ce1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6986
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Since we push arguments onto a stack when calling multi-argument
functions, we actually were ending up calling `call_with` with the
arguments in the *reverse order* - we patched around this by passing the
arguments in the reverse order for `foldl'`, but it makes more sense to
have them just be the order that the function would be called with in
user surface code instead.
Change-Id: Ifddb98f46970ac89872383709c3ce758dc965c65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7067
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
When compiling a lambda, take the name of the outer slot (if
available) and store it as the name on the lambda.
These names are then shown in the observer, and nowhere else (so far).
It is of course common for these things to thread through many
different context levels (e.g. `f = a: b: c: ...`), in this setup only
the outermost closure or thunk gains the name, but it's better than
nothing.
Change-Id: I681ba74e624f2b9e7a147144a27acf364fe6ccc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7065
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
There are some rare scope cases with deferred access where this
doesn't behave correctly otherwise.
Change-Id: I6c774f5e62c1cb50b598026c54727017a52cd22d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7064
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The warning needs to consider whether it is occuring inside of a
thunk, i.e. the dynamic ancestry chain of lambda contexts must be
inspected and not just the current scope.
Change-Id: I5cf5482d67a8bbb9f03b0ecee7a62f58754f8e59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7063
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
This check is now actually simply equivalent to checking whether the
target has been initialised or not.
Change-Id: I30660d11073ba313358f3a64234a90ed81abf74c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7062
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
These are builtins which can be basically implemented as the identity
function from the perspective of pure evaluation, and which help us
get closer to evaluating nixpkgs.
For now, builtins added here will be "usable" and just emit a warning
about not being implemented yet.
Change-Id: I0fce94677f01c98c0392aeefb7ab353c7dc7ec82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7060
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Checking the computed depth and stack slot against the computed depth
and stack slot is equivalent to just checking the indices into the
locals vector against each other (i.e. "is the slot we're compiling
into the slot we're accessing?")
Change-Id: Ie85a68df073e3b2e3d9aba7fe8634c48eada81fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7059
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
When capturing an upvalue, return a detailed TvixBug error that
contains metadata about what exactly was missing.
This particular thing helps with debugging some scope issues we still
seem to have.
Change-Id: I1089a1df4b3bbc63411a4907c3641a5dc3fad984
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7058
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Using the same method as in Thunk::deep_force, detect cycles when
printing values by maintaining a set of already seen thunks.
With this, display of infinite values matches that of Nix:
> nix-instantiate --eval --strict -E 'let as = { x = 123; y = as; }; in as'
{ x = 123; y = { x = 123; y = <CYCLE>; }; }
> tvix-eval -E 'let as = { x = 123; y = as; }; in as'
=> { x = 123; y = { x = 123; y = <CYCLE>; }; } :: set
Change-Id: I007b918d5131d82c28884e46e46ff365ef691aa8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7056
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This can be raised in cases where some panics lurk (e.g. indexed
accesses).
Change-Id: I93f4d60568277e9c5635aa52f378645626d68c5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7057
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Note that this test (ironically) fails if the observer is used (e.g.
with --trace-runtime), but that's a separate issue.
Change-Id: I952eaeac8b5a7acce9c66cd4744ec570280748e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7055
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This is done via a new `deepForce` function on Value. Since values can
be cyclical (for example, see the test-case), we need to do some extra
work to avoid RefCell borrow errors if we ever hit a graph cycle:
While deep-forcing values, we keep a set of thunks that we have
already seen and avoid doing any work on the same thunk twice. The set
is encapsulated in a separate type to stop potentially invalid
pointers from leaking out.
Finally, since deep_force is conceptually similar to
`VM::force_for_output` (but more suited to usage in eval since it
doesn't clone the values) this removes the latter, replacing it with
the former.
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: Iefddefcf09fae3b6a4d161a5873febcff54b9157
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7000
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
When forcing thunks in `force_with_output`, the call stack of the VM
is actually empty (as the calls are synthetic and no longer part of
the evaluation of the top-level expression).
This means that Tvix crashed when constructing error spans for the
`fallible` macro, as the assumption of there being an enclosing span
was violated.
To work around this, we instead pass the span for the whole top-level
expression to force_for_output and set this as the span for the
enclosing error chain. Existing output logic will already avoid
printing the entire expression as an error span.
This fixes b/213.
Change-Id: I93978e0deaf5bcb0f47a6fa95b3f5bebef5bad4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7052
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Failures to resolve a nix search path lookup in angle brackets can be
caught using tryEval (if it reaches the runtime). Resolving relative
paths (either to the current directory or the current user's home) can
never be caught, even if they happen inside a thunk at runtime (which is
currently the case for home-relative paths).
Change-Id: I7f73221df66d82a381dd4063358906257826995a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7025
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
cl/7036 moved these paths around, but neglected to update the relative
paths they contain. Without these updated, they will never start
passing!
Change-Id: Ib16468611af59729883e501be8486f43d850fd58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7046
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
eval-okay-eq.nix is actually an ancient test case that used the ATerm
syntax for the test result. It was disabled for a while, since the
behavior got reverted for a bit, but works on any reasonably modern
Nix implementation. This change matches my [C++ Nix PR].
[C++ Nix PR]: https://github.com/NixOS/nix/pull/7196
Change-Id: I602fd7c83a0bc104ab502c8b6a74e4591272be1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7045
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The language test suite actually doesn't require flakes and the
new features are mostly sensible (added builtins) as well as some
tests for regressions the C++ implementation experienced.
The path interpolation test is not included in this update because there
is no way to construct an location-independent .exp file for it (the C++
repo also doesn't have one). We may still want to implement that feature
eventually (in case rnix adds support for it).
The C++ Nix revision used is ac0fb38e8a5a25a84fa17704bd31b453211263eb.
Change-Id: I75f1e780ddeeee6f6b1f28cf3c66c288dca2c20c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7043
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Since cl/7036 we have a mechanism for dealing with the nix_tests we do
not pass yet.
Change-Id: I246c52963ae7f2500253f4035a77d7006dd35307
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7049
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reimplement the test discovery of the lang tests script in Nix which
allows for a more flexible skipping logic that can e.g. react to the C++
Nix version used. This allows us to run the test suite against both
C++ Nix 2.3 and the latest C++ Nix version 2.11. The latter is mainly
useful, so we can implement newer Nix features and still verify them
against the C++ implementation.
Change-Id: I30c802844133b86b5e49f5e4f4fefacdb6215e0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7042
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
It is helpful to be able to use the test suite as a regression test:
make a change to the compiler/vm, re-run the tests, and if there are
any failures you know it's your fault.
Right now we can't do that, because the expected-to-fail tests are
mixed in with the expected-to-pass tests. So we can't use them as a
regression test.
Change-Id: Ied606882b9835a7effd7e75bfcf3e5f827e0a2c8
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7036
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit deduplicates the Thunk-like functionality from Closure
and unifies it with Thunk.
Specifically, we now have one and only one way of breaking reference
cycles in the Value-graph: Thunk. No other variant contains a
RefCell. This should make it easier to reason about the behavior of
the VM. InnerClosure and UpvaluesCarrier are no longer necessary.
This refactoring allowed an improvement in code generation:
`Rc<RefCell<>>`s are now created only for closures which do not have
self-references or deferred upvalues, instead of for all closures.
OpClosure has been split into two separate opcodes:
- OpClosure creates non-recursive closures with no deferred
upvalues. The VM will not create an `Rc<RefCell<>>` when executing
this instruction.
- OpThunkClosure is used for closures with self-references or
deferred upvalues. The VM will create a Thunk when executing this
opcode, but the Thunk will start out already in the
`ThunkRepr::Evaluated` state, rather than in the
`ThunkRepr::Suspeneded` state.
To avoid confusion, OpThunk has been renamed OpThunkSuspended.
Thanks to @sterni for suggesting that all this could be done without
adding an additional variant to ThunkRepr. This does however mean
that there will be mutating accesses to `ThunkRepr::Evaluated`,
which was not previously the case. The field `is_finalised:bool`
has been added to `Closure` to ensure that these mutating accesses
are performed only on finalised Closures. Both the check and the
field are present only if `#[cfg(debug_assertions)]`.
Change-Id: I04131501029772f30e28da8281d864427685097f
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7019
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
In `a++b`, the previous implementation would move `b` (i.e. memcpy
its elements) twice. Let's do that only once.
We sure do call NixList.clone() a whole lot. At some point in the
future we probably want to do a SmolStr-type split for NixList into
a two-variant enum where one side is an Rc<Vec<Value>> for lists
longer than a certain length.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I32154d18785a1f663454a8b9d4afd3e78bffdf9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7040
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This commit adds a comment for the benefit of non-Lua-users
explaining what an upvalue is. It also adds a comment explaining
what `with_stack` does, including a brief reminder of nix's
slightly-unusual-but-well-justified handling of this construct.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: If6d0e133292451af906e1c728e34010f99be8c7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7007
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Without this change it was possible to cause situations (see the new
test) in which a `with`-namespace was forced prematurely.
Change-Id: I879ea7763b43edc693feace2c73c890d426fafd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7031
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Now that we're tracking formals on Lambda this ends up being quite easy;
we just pull them off of the Lambda for the argument closure and use
them to construct the result attribute set.
Change-Id: I811cb61ec34c6bef123a4043000b18c0e4ea0125
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7003
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Validate "closed formals" (formal parameters without an ellipsis) via a
new ValidateClosedFormals op, which checks the arguments (in an attr set
at the top of the stack) against the formal parameters on the Lambda in
the current frame, and returns a new UnexpectedArgument error (including
the span of the formals themselves!!) if any arguments aren't allowed
Change-Id: Idcc47a59167a83be1832a6229f137d84e426c56c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7002
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
In preparation for both implementing the `functionArgs` builtin and
adding support for validating closed formals, record information about
the formal arguments to a function *on the Lambda itself*. This may seem
a little odd for the purposes of just closed formal checking, but is
something we have to have anyway for builtins.functionArgs so I figured
I'd do it this way to kill both birds with one stone.
Change-Id: Ie3770a607bf352a1eb395c79ca29bb25d5978cd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Since we already have infra for forcing arguments to builtins, this ends
up being almost *too* simple - we just return the second argument!
Change-Id: I070d3d0b551c4dcdac095f67b31e22e0de90cbd7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6999
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This resolves a TODO.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: If4d2124648ac88094e547e1ad7f1b446feb26182
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7010
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The previous serialisation format kind of lost the information about
what AST node we're dealing with (e.g. `1234` would serialise to an
AST with a literal `1234`).
That's great for pretty-printing the _code_, but we explicitly want to
serialise how rnix-parser parses something.
To that end, literals are now instead serialised into a structure like
all the other ones (`kind: literal` and appropriate value fields).
Change-Id: I586c95d7db41820b8ec43565ba4016ed3834d1b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7030
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Home relative paths depend on the environment to be resolved. We have
elected to do everything that depends on the environment, e.g. resolving
SPATH expressions using NIX_PATH, at runtime, so tvix evaluation would
continue to behave correctly even if we separated the compilation and
execution phases more, e.g. via serializing bytecode. Then the value of
HOME, NIX_PATH etc. could reasonably change in the time until execution,
yielding wrong results if the resolution results were cached in the
bytecode.
We also take the opportunity to fix the broken path concatenation
previously found in the compiler, fixing b/205.
Another thing we could consider is emitting a warning for home relative
path literals, as they are by nature relatively fragile.
One sideeffect of this change is that home path resolution errors
become catchable which is not the case in C++ Nix. This will need to be
fixed up in a subsequent change.
Change-Id: I30bd69b575667c49170a9fdea23a020565d0f9ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7024
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
To assert that OpFindFile is only emitted for specially compiled SPATH
expressions, as well as make sure it doesn't accidentally operate on
“ordinary values”, introduce an UnresolvedPath internal value. If
OpFindFile sees a non-UnresolvedPath value, it'll crash.
Note that this change is not done purely for OpFindFile: We may want to
compile SPATH expressions as function calls to __findFile (like C++ Nix
does) in the future, so the UnresolvedPath value would definitely need
to be an ordinary string again then. Rather, this change is done in
preparation for resolving home dir relative paths at runtime (since they
depend on the environment) for which we'll need a similar mechanism to
OpFindFile.
Change-Id: I6acf287f35197cd9e13377079f972b9d36e5b22e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7023
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This implements serde::Serialize for the rnix AST through a wrapper
type, and exposes a function for serialising the AST into
a (pretty-printed JSON) string representation.
This can be used to debug issues with the AST, and to display an AST
reprsentation in tools like tvixbolt.
Serialize is implemented manually because we don't own any of the
structs and the way to traverse them is not easily derived
automatically, and this is quite verbose. We might be able to condense
it a little bit, but at the same time it's also fairly straightforward.
Change-Id: I922df43cfc25636f3c8baee7944c75ade516055c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6943
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Lambda has a quite large and variable-sized runtime representation,
unlike Rc<Lambda>. It would be easy to accidentally call clone() on
this and create input-dependent performance regressions.
Nothing in the codebase is currently using Lambda.clone(). Let's
remove the derived instance. If it's really needed it is very easy
to add it back in, but whoever does that will have to look at the
struct they're adding Clone to, which will hopefully prompt them to
think about whether or not that's really what they want to do.
Change-Id: I7806a741862ab4402229839ce3f4ea75aafd6d12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7029
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I believe this variant is left over from a previous implementation.
If not, please let me know.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I02a3bf2f63794d09e96a5a92a034c0ad3d1ff221
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7027
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Implement an *initial* version of builtins.match, using the rust `regex`
crate for regular expressions. The rust regex crate definitely has
different semantics than nix's regular expressions - but we'd like to
see how far we can get before the incompatibility starts to matter.
This consciously leaves out any sort of memo for compiled regular
expressions (which upstream nix also has) for the sake of expediency -
in the future we should implement that so we don't have to compile the
same regular expression multiple times.
Change-Id: I5b718635831ec83397940e417a9047c4342b6fa1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6989
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Using `serde_json` for parsing JSON here, plus an `impl FromJSON for
Value`. The latter is primarily to stay "dependency light" for now -
likely going with an actual serde `Deserialize` impl in the future is
going to be way better as it allows saving significantly on intermediary
allocations.
Change-Id: I152a0448ff7c87cf7ebaac927c38912b99de1c18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6920
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Working on https://github.com/NixOS/nix/pull/7158, I discovered that C++
Nix actually is strict in the accumulator, just not in the first value.
This seems due to the fact that in the C++ evaluator, function calls
don't seem to be thunked unconditionally and foldl' just elects not to
wrap it in a thunk (don't quote me on this summary, even though it seems
to line up with the code for primop_foldlStrict and testable behavior).
It doesn't seem worth it to risk breaking the odd Nix expression just to
be strict in one more value per invocation of foldl' (i.e. the initial
accumulator value `nul`), so let's match the existing C++ Nix behavior
here.
Change-Id: If59e62271a90d97cb440f0ca72a58ec7840d1690
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7022
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This commit causes the test eval-okay-builtins.nix to pass.
It also adds tests/tvix_tests/eval-okay-dirof.nix which has better
coverage than the nix tests for this builtin.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I71d96b48680696fd6e4fea3a9861742b35cfaa66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6987
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit implements builtins.toPath. Like OP_ADD, it currently
does not handle string contexts.
This commit allows the
tests::nix_eval_okay_src_tests_nix_tests_eval_okay_pathexists_nix
test to pass.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Iadd4f7605f8f297adbd0dba187b8481c21370b6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6996
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Maybe I misunderstood this part of the code, but the use of `unsafe`
appears unnecessary here? In any event it is the one and only
`unsafe` in the codebase.
Hopefully getting to "no `unsafe` anywhere" is worth the extra
never-taken branch caused by unwrap() instead of unwrap_unchecked()?
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I33fbd5aad9d8307ea82c24b6220412783e1973c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7011
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The case branch in vm.rs for OpResolveWithOrUpvalue is
unreachable/deadcode.
I believe this opcode is unnecessary, since it should always be
statically detectable (at parse-time) whether a reference is to an
upvalue (i.e. enclosing binding); otherwise, and only then, is
with-resolution applicable.
Perhaps I've misunderstood how with-resolution works. If so, please
explain it to me and -1/-2 this CL.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I4a90b9eb6cb3396df92a6a943d42ecc301871ba0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7009
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This makes it easier to compare currently implemented ones with the full
list.
Change-Id: Ibaffd99d05afa15fc9ab644fd101afa24fc7a1b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7008
Tested-by: BuildkiteCI
Autosubmit: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
When I first read the `README.md` I didn't know what this `mg build`
was. Some grepping around turned up `/tools/magrathea`. Apparently
this thing "helps me build planets". Awesome! Let's tell other
people where to find it. It seems that this tool is (currently)
specific to depot, so people who arrive at depot because of tvix
probably won't know where to find `mg`.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Icb8c51087fd41b696794516abcfee24a4b3b4a14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6948
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
