Commit graph

749 commits

Author SHA1 Message Date
Adam Joseph
d8841376e7 fix(tvix_eval): {stack,local}_idx confusion
The variable name `local_idx` is used here for a StackIdx, which invites
confusion.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I2e22db90acdc0d29586ee5b72ea18d42d93badcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7086
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-04 01:39:24 +00:00
Adam Joseph
f93f138c6c fix(tvix/eval): inline mis-named Local::above()
If self.depth > other.depth then self is deeper than other, so self
is *below* other, not above it.  Let's just inline the function.

Change-Id: I8dda3d90cbc86c8a6fa01bc4a5e506a2e403bd20
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7090
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-04 01:33:22 +00:00
Adam Joseph
0649474206 fix(tvix/eval): remove impl PartialEq for Value
It isn't possible to implement PartialEq properly for Value, because
any sensible implementation needs to force() thunks, which cannot be
done without a `&mut VM`.

The existing derive(PartialEq) has false negatives, which caused the
bug which cl/7142 fixed.  Fortunately that bug was easy to find, but
a silent false negative deep within the bowels of nixpkgs could be a
real nightmare to hunt down.

Let's just remove the PartialEq impl for Value, and the other
derive(PartialEq)'s that depend on it.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Iacd3726fefc7fc1edadcd7e9b586e04cf8466775
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7144
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-04 00:30:13 +00:00
Florian Klink
73fb474752 feat(tvix/eval): add rust-analyzer to dev-env
I think we should bring this into $PATH too.

Change-Id: Ie31ac558355b7c4ed9dcd3dd60e1b03f141d1178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7166
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-03 23:29:04 +00:00
Florian Klink
64afe59bbe chore(tvix/proto/castore.proto): drop Blob and Chunk
In ripple, this is used for the internal on-disk format, but it's not
suitable for remote consumption.

Change-Id: I327361a2254566ac9216e23eaed36dba8fdd283b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7127
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2022-11-02 11:48:35 +00:00
Florian Klink
5392d73816 feat(tvix/proto): add castore.proto
This is
0d4906cbabb183caa96e763671810fb39bd0c935:ripple/fossil/src/store.proto,
from https://src.unfathomable.blue.

It is not pulled in as a subtree, as some of the commits in there break
index-pack:

```
remote: error: object 2f487c3bf7cd8efd64f1d217edac732db98ff1c0: badDateOverflow: invalid author/committer line - date causes integer overflow
remote: fatal: fsck error in packed object
error: remote unpack failed: index-pack abnormal exit
```

Co-authored-by: edef <edef@unfathomable.blue>
Change-Id: I3369044090a3192e2322775a335887c37536a942
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7126
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2022-11-02 11:48:35 +00:00
Adam Joseph
9ada456260 fix(tvix/eval): nix_eq() must recurse
The current implementation of nix_eq will force one level of thunks
and then switch to the (non-forcing) rust Eq::eq() method.  This
gives incorrect results for lists-of-thunks.

This commit changes nix_eq() to be recursive.

A regression test (which fails prior to this commit) is included.

This fix also causes nix_tests/eval-okay-fromjson.nix to pass, so it
is moved out of notyetpassing.

Change-Id: I655fd7a5294208a7b39df8e2c3c12a8b9768292f
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7142
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-31 12:36:15 +00:00
Adam Joseph
98a981a737 docs(tvix/eval): builtins.md: note implementation status
We're getting close to the finish line, folks.

I went through the list of builtins and there are only 33 that
remain unimplemented.  I've marked them, and indicated which are
ready to be implemented vs which are waiting for other things.

We can delete this column from the table once everything is
implemented.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Idfaef93283536288b12e59aef5c3e1cd139bd133
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7140
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-31 12:36:15 +00:00
Adam Joseph
cc3060ed7c docs(tvix/eval): builtins.md: mark impure
I believe that the currentTime, findFile, hashFile, pathExists,
readDir, path (unless ?sha256), and readFile builtins are impure.
This commit marks them as such in docs/builtins.md.

Change-Id: Ib1b59fe643dde73cb2b00050b4ef9d3401ad22eb
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7139
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-30 21:08:41 +00:00
sterni
313a24e4de test(tvix/eval): builtins.sort must preserve order of equal elements
Change-Id: I59a0756940d1e5360a2ab4e886cf0bc9af7b8901
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7133
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-29 14:06:28 +00:00
Griffin Smith
3412ae4956 feat(tvix/eval): Implement builtins.sort
This is a bit tricky because the comparator can throw errors, so we
need to propagate them out if they exist and try to avoid sorting
forever by returning a reasonable ordering in this case (as
short-circuiting is not available).

Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: Icae1d30f43ec1ae64b2ba51e73ee467605686792
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7072
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-10-29 14:04:12 +00:00
Griffin Smith
d0a836b0e1 feat(tvix/eval): Implement comparison for lists
Lists are compared lexicographically in C++ nix as of [0], and our
updated nix test suites depend on this. This implements comparison of
list values in `Value::nix_cmp` using a very similar algorithm to what
C++ does - similarly to there, this requires passing in the VM so we can
force thunks in the list elements as we go.

[0]: 09471d2680#

Change-Id: I5d8bb07f90647a1fec83f775243e21af856afbb1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7070
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-29 10:45:51 +00:00
Adam Joseph
b8a7dba709 feat(tvix/eval): builtins.replaceStrings: don't clone() N times
CL/7034 looks great, except that for a length-N target string it
will perform N deep copies of each of the from and to-lists.  Let's
use references instead of clones.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Icd341213a9f0e728f9c8453cec6d23af5e1dea91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7095
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-28 10:26:37 +00:00
James Landrein
ccab9c06a2 feat(tvix/eval): add builtins.replaceStrings
Change-Id: I93dcdaeb101364ee2273bcaeb19acb57cf6b9e7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7034
Autosubmit: j4m3s <james.landrein@gmail.com>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-28 10:21:35 +00:00
Adam Joseph
3aec678679 docs(tvix/eval): warn that AttrsRep::KV is not for Key-Value pairs
I assumed that AttrsRep::KV represented attrsets with a single
attribute as a Key-Value pair.  That is not the case.  Let's warn
other people about this.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ie3d2765fcc1ab705c153ab94ffe77bbd6d4ab39e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7093
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-28 10:16:34 +00:00
Adam Joseph
a79bbad03b docs(tvix/eval): add "intern literals" to future optimisations
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I460230863de853ca5248733bc977d4780b216f36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7096
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-10-28 02:57:23 +00:00
Adam Joseph
2502c0abef fix(tvix/eval): correct wasm32-unknown-unknown to wasm32-none
Rustc uses wasm32-unknown-unknown, which is rejected by config.sub,
for wasm-in-the-browser environments.  Rustc should be using
wasm32-unknown-none, which config.sub accepts.  Hopefully the rustc
people will change their triple before stabilising this triple.  In
the meantime, we fix it here in order to unbreak tvixbolt.

https://doc.rust-lang.org/beta/nightly-rustc/rustc_target/spec/wasm32_unknown_unknown/index.html

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I941fd8d6f3db4e249901772fd79321ad88cd9cc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7107
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-27 23:23:22 +00:00
Adam Joseph
4d83fd84b4 refactor(tvix/eval): search-and-replace changes
This commit contains two search-and-replace renames which are broken
out from I04131501029772f30e28da8281d864427685097f in order to
reduce the noise in that CL:

- `is_thunk -> is_suspended_thunk`, since there are now
  OpThunkClosure and OpThunkSuspended

- `compile_lambda_or_thunk` -> `compile_lambda_or_suspension`

Change-Id: I7cc5bbb75ef6605e3428c7be27e812f41a10c127
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7037
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-27 22:00:39 +00:00
Adam Joseph
8616f13a71 feat(tvix/eval): builtins.import without RefCell
CL/6867 added support for builtins.import, which required a cyclic
reference import->globals->builtins->import.  This was implemented
using a RefCell, which makes it possible to mutate the builtins
during evaluation.  The commit message for CL/6867 expressed a
desire to eliminate this possibility:

  This opens up a potentially dangerous footgun in which we could
  mutate the builtins at runtime leading to different compiler
  invocations seeing different builtins, so it'd be nice to have
  some kind of "finalised" status for them or some such, but I'm not
  sure how to represent that atm.

This CL replaces the RefCell with Rc::new_cyclic(), making the
globals/builtins immutable once again.  At VM runtime (once opcodes
start executing) everything is the same as before this CL, except
that the Rc<RefCell<>> introduced by CL/6867 is turned into an
rc::Weak<>.

The function passed to Rc::new_cyclic works very similarly to
overlays in nixpkgs: a function takes its own result as an argument.
However instead of laziness "breaking the cycle", Rust's
Rc::new_cyclic() instead uses an rc::Weak.  This is done to prevent
memory leaks rather than divergence.

This CL also resolves the following TODO from CL/6867:

  // TODO: encapsulate this import weirdness in builtins

The main disadvantage of this CL is the fact that the VM now must
ensure that it holds a strong reference to the globals while a
program is executing; failure to do so will cause a panic when the
weak reference in the builtins is upgrade()d.

In theory it should be possible to create strong reference cycles
the same way Rc::new_cyclic() creates weak cycles, but these cycles
would cause a permanent memory leak -- without either an rc::Weak or
RefCell there is no way to break the cycle.  At some point we will
have to implement some form of cycle collection; whatever library we
choose for that purpose is likely to provide an "immutable strong
reference cycle" primitive similar to Rc::new_cyclic(), and we
should be able to simply drop it in.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I34bb5821628eb97e426bdb880b02e2097402adb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7097
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-27 21:36:01 +00:00
Adam Joseph
4384418877 docs(tvix/eval): StackIdx, LocalIdx UpvalueIdx
This adds a comment noting that StackIdx is an offset relative to
the base of the current CallFrame, whereas UpvalueIdx is an absolute
index into the upvalues array.

It also removes the confusing mention of StackIdx in the descriptive
comment for LocalIdx.  They index into totally different structures;
one exists at runtime and the other exists at compile time.

Change-Id: Ib932b1b0679734c15001e8c5c95a08293fa016b4
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7017
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-26 14:27:37 +00:00
Adam Joseph
499a443032 feat(tvix/eval): add NixList::force_elements()
This adds a function NixList::force_elements() which forces each
element of the list shallowly.  This behavior is needed for
`builtins.replaceStrings`, and probably a few other builtins as
well.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I3f0681acbbfe50e781b5f07b6a441647f5e6f8da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7094
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-26 12:32:54 +00:00
Adam Joseph
dc3543e0ca feat(tvix/eval): include filename of failing test when failing
Unfortunately we have to mangle test case filenames into rust-valid
symbols, since test-generator doesn't use `r#"..."` (deliberately?).
This means that when a test fails, there's nothing on the console
you can copy-and-paste in order to view/edit the code of the failing
test case.

This commit (partially) fixes it by including the unmangled name in
the panic!() string.  However failures due to panic!()s inside the
vm (including deliberate panics due to panic!()-debugging) still
won't display an unmangled filename.

Maybe we should reconsider the use of test-generator?

Change-Id: I2208a859ffab1264f17f48fd303ff5e19675967e
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7092
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-26 10:20:01 +00:00
James Landrein
8425c2016c feat(tvix/eval): add builtins.{floor,ceil}
Change-Id: I4e6c4f96f6f5097a5c637eb3dbbd7bb8b34b7d52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7032
Autosubmit: j4m3s <james.landrein@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-25 08:46:37 +00:00
Griffin Smith
5bd0e723c1 refactor(tvix/eval): Implement value comparison with a method
Rather than implementing all of the interesting semantics of value
comparison with a macro bound to the VM, implement the bulk of the logic
with a method on Value itself that returns an Ordering, and then use the
macro to implement the comparison against that Ordering. This has no
functional change, but paves the way to implementing lexicographic
comparison of list values, which is supported in the latest version of
upstream nix.

Change-Id: I8af1a020b41577021af5939f5edc160c407d4a9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7069
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-24 13:44:02 +00:00
Griffin Smith
7b3bda9e08 feat(tvix/eval): Implement builtins.mapAttrs
I played around a little bit with doing this in-place, but ended up
going with this perhaps slightly clone-heavy approach for now because
ideally most clones on Value are cheap - but later we should benchmark
alternate approaches that get to reuse allocations better if necessary
or possible.

Change-Id: If998eb2056cedefdf2fb480b0568ac8329ccfc44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7068
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-24 13:44:02 +00:00
Vincent Ambo
8497b83569 feat(tvix/eval): add builtins.langVersion
The last bump in langVersion (5->6) in C++ Nix was due to making lists
comparable (commit `09471d2680292af48b2788108de56a8da755d661`), which
we support in Tvix with cl/7070.

Change-Id: Id3beed5150b8fb6e0a46a4d1b7e3942022a65346
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7074
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-10-24 13:27:57 +00:00
Adam Joseph
f7ba502005 feat(tvix/eval): implement builtins.currentSystem
This commit implements builtins.currentSystem, by capturing the
cargo environment variable `TARGET` and exposing it to rustc as
`TVIX_CURRENT_SYSTEM` so it can be inserted into the source code
using `env!()`.

The resulting value needs to be massaged a bit, since it is an "LLVM
triple".  The current code should work for all the platforms for
which cppnix works (thanks qyliss for generating the list!).  It
does *not* reject all of the triples that cppnix's configure.ac
rejects -- it is much more forgiving.  We can tighten this up in a
future commit.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I947f504b2af5a7fee8cf0cb301421d2fc9174ce1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6986
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-24 12:20:01 +00:00
Griffin Smith
e2f0967d3f feat(nix/eval): Implement builtins.groupBy
Change-Id: I3e0aa017a7100cbeb86d2e5747471b36affcc102
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7038
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-24 08:13:59 +00:00
Griffin Smith
89aee6443c fix(tvix/eval): Use natural arg order for call_with
Since we push arguments onto a stack when calling multi-argument
functions, we actually were ending up calling `call_with` with the
arguments in the *reverse order* - we patched around this by passing the
arguments in the reverse order for `foldl'`, but it makes more sense to
have them just be the order that the function would be called with in
user surface code instead.

Change-Id: Ifddb98f46970ac89872383709c3ce758dc965c65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7067
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-23 17:39:55 +00:00
Vincent Ambo
d4569cb504 feat(tvix/eval): initial attempt at setting lambda names
When compiling a lambda, take the name of the outer slot (if
available) and store it as the name on the lambda.

These names are then shown in the observer, and nowhere else (so far).

It is of course common for these things to thread through many
different context levels (e.g. `f = a: b: c: ...`), in this setup only
the outermost closure or thunk gains the name, but it's better than
nothing.

Change-Id: I681ba74e624f2b9e7a147144a27acf364fe6ccc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7065
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-23 15:58:53 +00:00
Vincent Ambo
1050a1d650 test(tvix/eval): add a test case for nested sibling access
Change-Id: I3898284bc4871e5483ead0c27e0f2832d66f2b29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7061
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
5612ec0f41 fix(tvix/eval): thunk let-expression
There are some rare scope cases with deferred access where this
doesn't behave correctly otherwise.

Change-Id: I6c774f5e62c1cb50b598026c54727017a52cd22d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7064
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-10-23 15:50:35 +00:00
Vincent Ambo
5b305fab40 fix(tvix/eval): fix condition for useless inherit warning
The warning needs to consider whether it is occuring inside of a
thunk, i.e. the dynamic ancestry chain of lambda contexts must be
inspected and not just the current scope.

Change-Id: I5cf5482d67a8bbb9f03b0ecee7a62f58754f8e59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7063
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
9b52e5b9c2 refactor(tvix/eval): simplify check for deferring upvalue resolution
This check is now actually simply equivalent to checking whether the
target has been initialised or not.

Change-Id: I30660d11073ba313358f3a64234a90ed81abf74c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7062
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
1f4420cb4a feat(tvix/eval): add mechanism for placeholder builtins
These are builtins which can be basically implemented as the identity
function from the perspective of pure evaluation, and which help us
get closer to evaluating nixpkgs.

For now, builtins added here will be "usable" and just emit a warning
about not being implemented yet.

Change-Id: I0fce94677f01c98c0392aeefb7ab353c7dc7ec82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7060
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
546fcf51cd refactor(tvix/eval): simplify self-reference check
Checking the computed depth and stack slot against the computed depth
and stack slot is equivalent to just checking the indices into the
locals vector against each other (i.e. "is the slot we're compiling
into the slot we're accessing?")

Change-Id: Ie85a68df073e3b2e3d9aba7fe8634c48eada81fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7059
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
c9bf7c4cf1 chore(tvix/eval): return detailed TvixBug if an upvalue is missing
When capturing an upvalue, return a detailed TvixBug error that
contains metadata about what exactly was missing.

This particular thing helps with debugging some scope issues we still
seem to have.

Change-Id: I1089a1df4b3bbc63411a4907c3641a5dc3fad984
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7058
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
60f24c3c53 fix(tvix/eval): detect cycles when printing infinite values
Using the same method as in Thunk::deep_force, detect cycles when
printing values by maintaining a set of already seen thunks.

With this, display of infinite values matches that of Nix:

    > nix-instantiate --eval --strict -E 'let as = { x = 123; y = as; }; in as'
    { x = 123; y = { x = 123; y = <CYCLE>; }; }

    > tvix-eval -E 'let as = { x = 123; y = as; }; in as'
    => { x = 123; y = { x = 123; y = <CYCLE>; }; } :: set

Change-Id: I007b918d5131d82c28884e46e46ff365ef691aa8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7056
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
4ff06ba67d feat(tvix/eval): add TvixBug error kind
This can be raised in cases where some panics lurk (e.g. indexed
accesses).

Change-Id: I93f4d60568277e9c5635aa52f378645626d68c5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7057
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-22 18:30:12 +00:00
Vincent Ambo
359444360b test(tvix/eval): add a test for observing an infinite attribute set
Note that this test (ironically) fails if the observer is used (e.g.
with --trace-runtime), but that's a separate issue.

Change-Id: I952eaeac8b5a7acce9c66cd4744ec570280748e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7055
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-22 18:20:48 +00:00
Griffin Smith
d4fa3152e9 feat(tvix/eval): Implement builtins.deepSeq
This is done via a new `deepForce` function on Value. Since values can
be cyclical (for example, see the test-case), we need to do some extra
work to avoid RefCell borrow errors if we ever hit a graph cycle:

While deep-forcing values, we keep a set of thunks that we have
already seen and avoid doing any work on the same thunk twice. The set
is encapsulated in a separate type to stop potentially invalid
pointers from leaking out.

Finally, since deep_force is conceptually similar to
`VM::force_for_output` (but more suited to usage in eval since it
doesn't clone the values) this removes the latter, replacing it with
the former.

Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: Iefddefcf09fae3b6a4d161a5873febcff54b9157
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7000
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-22 18:11:55 +00:00
Vincent Ambo
8724d2fff8 fix(tvix/eval): use top-level span for force_with_output
When forcing thunks in `force_with_output`, the call stack of the VM
is actually empty (as the calls are synthetic and no longer part of
the evaluation of the top-level expression).

This means that Tvix crashed when constructing error spans for the
`fallible` macro, as the assumption of there being an enclosing span
was violated.

To work around this, we instead pass the span for the whole top-level
expression to force_for_output and set this as the span for the
enclosing error chain. Existing output logic will already avoid
printing the entire expression as an error span.

This fixes b/213.

Change-Id: I93978e0deaf5bcb0f47a6fa95b3f5bebef5bad4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7052
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-22 17:01:41 +00:00
sterni
64bb501de1 fix(tvix): distinguish search- and relative path resolution errors
Failures to resolve a nix search path lookup in angle brackets can be
caught using tryEval (if it reaches the runtime). Resolving relative
paths (either to the current directory or the current user's home) can
never be caught, even if they happen inside a thunk at runtime (which is
currently the case for home-relative paths).

Change-Id: I7f73221df66d82a381dd4063358906257826995a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7025
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-21 00:11:29 +00:00
sterni
b410655a5e test(tvix/verify-lang-tests): also check notyetpassing tests
Adding these to the C++ Nix CI ensures that it is possible for the
tests in that directory to pass at all. Mainly this would catch
situations like fixed in a previous CL when moving the tests around
would break them so that they wouldn't even pass in C++ Nix.

For this to work, we need to track skips by basename to be
directory-independent (assuming that every skipped test name is unique
is hopefully okay).

Change-Id: If6cb63ebdef0fc19b082b6a04e79ada2e47c658e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7048
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 18:21:25 +00:00
sterni
bbf1e32737 fix(tvix/eval): restore .exp.xml files and skip in test suite
Change-Id: Iebda5e0d99925a0a8c1d6ae1d7a35397d127bf31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7047
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-20 08:40:01 +00:00
sterni
d9dfe65910 fix(tvix/eval): fix path in notyetpassing tests
cl/7036 moved these paths around, but neglected to update the relative
paths they contain. Without these updated, they will never start
passing!

Change-Id: Ib16468611af59729883e501be8486f43d850fd58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7046
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 02:35:42 +00:00
sterni
60363039ef test(tvix/eval): eval-okay-eq.nix can actually be tested
eval-okay-eq.nix is actually an ancient test case that used the ATerm
syntax for the test result. It was disabled for a while, since the
behavior got reverted for a bit, but works on any reasonably modern
Nix implementation. This change matches my [C++ Nix PR].

[C++ Nix PR]: https://github.com/NixOS/nix/pull/7196

Change-Id: I602fd7c83a0bc104ab502c8b6a74e4591272be1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7045
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 02:35:42 +00:00
sterni
0d9eabcaac fix(tvix/eval): eval-okay-pathexists test can be executed
Change-Id: Ibdcaa165024584370ce9578e67985a3526e44f77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7044
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 02:33:40 +00:00
sterni
8e188d6235 test(tvix/eval): update C++ Nix test suite to current master
The language test suite actually doesn't require flakes and the
new features are mostly sensible (added builtins) as well as some
tests for regressions the C++ implementation experienced.

The path interpolation test is not included in this update because there
is no way to construct an location-independent .exp file for it (the C++
repo also doesn't have one). We may still want to implement that feature
eventually (in case rnix adds support for it).

The C++ Nix revision used is ac0fb38e8a5a25a84fa17704bd31b453211263eb.

Change-Id: I75f1e780ddeeee6f6b1f28cf3c66c288dca2c20c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7043
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 02:33:40 +00:00
sterni
e3637136d1 test(tvix/eval): enable nix_tests by default
Since cl/7036 we have a mechanism for dealing with the nix_tests we do
not pass yet.

Change-Id: I246c52963ae7f2500253f4035a77d7006dd35307
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7049
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 02:28:38 +00:00
sterni
292a751388 test(tvix): nix-planned test verification using C++ Nix 2.3 and 2.11
Reimplement the test discovery of the lang tests script in Nix which
allows for a more flexible skipping logic that can e.g. react to the C++
Nix version used. This allows us to run the test suite against both
C++ Nix 2.3 and the latest C++ Nix version 2.11. The latter is mainly
useful, so we can implement newer Nix features and still verify them
against the C++ implementation.

Change-Id: I30c802844133b86b5e49f5e4f4fefacdb6215e0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7042
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 02:28:38 +00:00
Adam Joseph
f991ff9294 feat(tvix/eval): expect not-yet-passing tests to fail
It is helpful to be able to use the test suite as a regression test:
make a change to the compiler/vm, re-run the tests, and if there are
any failures you know it's your fault.

Right now we can't do that, because the expected-to-fail tests are
mixed in with the expected-to-pass tests.  So we can't use them as a
regression test.

Change-Id: Ied606882b9835a7effd7e75bfcf3e5f827e0a2c8
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7036
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-19 15:01:05 +00:00
Adam Joseph
d978b556e6 feat(tvix/eval): deduplicate overlap between Closure and Thunk
This commit deduplicates the Thunk-like functionality from Closure
and unifies it with Thunk.

Specifically, we now have one and only one way of breaking reference
cycles in the Value-graph: Thunk.  No other variant contains a
RefCell.  This should make it easier to reason about the behavior of
the VM.  InnerClosure and UpvaluesCarrier are no longer necessary.

This refactoring allowed an improvement in code generation:
`Rc<RefCell<>>`s are now created only for closures which do not have
self-references or deferred upvalues, instead of for all closures.
OpClosure has been split into two separate opcodes:

- OpClosure creates non-recursive closures with no deferred
  upvalues.  The VM will not create an `Rc<RefCell<>>` when executing
  this instruction.

- OpThunkClosure is used for closures with self-references or
  deferred upvalues.  The VM will create a Thunk when executing this
  opcode, but the Thunk will start out already in the
  `ThunkRepr::Evaluated` state, rather than in the
  `ThunkRepr::Suspeneded` state.

To avoid confusion, OpThunk has been renamed OpThunkSuspended.

Thanks to @sterni for suggesting that all this could be done without
adding an additional variant to ThunkRepr.  This does however mean
that there will be mutating accesses to `ThunkRepr::Evaluated`,
which was not previously the case.  The field `is_finalised:bool`
has been added to `Closure` to ensure that these mutating accesses
are performed only on finalised Closures.  Both the check and the
field are present only if `#[cfg(debug_assertions)]`.

Change-Id: I04131501029772f30e28da8281d864427685097f
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7019
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-19 10:38:54 +00:00
Adam Joseph
c91d86ee5c feat(tvix/eval): NixList::concat(): avoid an unnecessary move
In `a++b`, the previous implementation would move `b` (i.e. memcpy
its elements) twice.  Let's do that only once.

We sure do call NixList.clone() a whole lot.  At some point in the
future we probably want to do a SmolStr-type split for NixList into
a two-variant enum where one side is an Rc<Vec<Value>> for lists
longer than a certain length.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I32154d18785a1f663454a8b9d4afd3e78bffdf9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7040
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-19 10:37:30 +00:00
Adam Joseph
4b01e594d5 docs(tvix/eval): upvalues.rs: define "upvalue", comment with_stack
This commit adds a comment for the benefit of non-Lua-users
explaining what an upvalue is.  It also adds a comment explaining
what `with_stack` does, including a brief reminder of nix's
slightly-unusual-but-well-justified handling of this construct.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: If6d0e133292451af906e1c728e34010f99be8c7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7007
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-18 09:38:16 +00:00
Vincent Ambo
13a5e7dd5b fix(tvix/eval): wrap dynamic resolution in an extra thunk
Without this change it was possible to cause situations (see the new
test) in which a `with`-namespace was forced prematurely.

Change-Id: I879ea7763b43edc693feace2c73c890d426fafd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7031
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-10-18 09:15:13 +00:00
Griffin Smith
0063e7e913 feat(nix/eval): Implement builtins.functionArgs
Now that we're tracking formals on Lambda this ends up being quite easy;
we just pull them off of the Lambda for the argument closure and use
them to construct the result attribute set.

Change-Id: I811cb61ec34c6bef123a4043000b18c0e4ea0125
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7003
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-17 11:29:49 +00:00
Griffin Smith
2a3d498104 feat(tvix/eval): Validate closed formals
Validate "closed formals" (formal parameters without an ellipsis) via a
new ValidateClosedFormals op, which checks the arguments (in an attr set
at the top of the stack) against the formal parameters on the Lambda in
the current frame, and returns a new UnexpectedArgument error (including
the span of the formals themselves!!) if any arguments aren't allowed

Change-Id: Idcc47a59167a83be1832a6229f137d84e426c56c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7002
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-17 11:29:49 +00:00
Griffin Smith
e63d14419f feat(tvix/eval): Record formals on lambda
In preparation for both implementing the `functionArgs` builtin and
adding support for validating closed formals, record information about
the formal arguments to a function *on the Lambda itself*. This may seem
a little odd for the purposes of just closed formal checking, but is
something we have to have anyway for builtins.functionArgs so I figured
I'd do it this way to kill both birds with one stone.

Change-Id: Ie3770a607bf352a1eb395c79ca29bb25d5978cd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-17 11:29:49 +00:00
Griffin Smith
89dbcbbb3d feat(tvix/eval): Implement builtins.seq
Since we already have infra for forcing arguments to builtins, this ends
up being almost *too* simple - we just return the second argument!

Change-Id: I070d3d0b551c4dcdac095f67b31e22e0de90cbd7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6999
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-17 11:04:17 +00:00
Adam Joseph
6e30fbbf7b docs(tvix/eval) comments for various fields
Change-Id: I8dcddf2b419761e475e71215c199eef2f7dc61dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7028
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-17 09:51:18 +00:00
Adam Joseph
f05a1d27d9 refactor(tvix/eval): unify compile_lambda() with thunk()
This resolves a TODO.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: If4d2124648ac88094e547e1ad7f1b446feb26182
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7010
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 23:42:30 +00:00
Vincent Ambo
8e9bfc1ca7 fix(tvix/eval): more faithfully serialise ast::Literal
The previous serialisation format kind of lost the information about
what AST node we're dealing with (e.g. `1234` would serialise to an
AST with a literal `1234`).

That's great for pretty-printing the _code_, but we explicitly want to
serialise how rnix-parser parses something.

To that end, literals are now instead serialised into a structure like
all the other ones (`kind: literal` and appropriate value fields).

Change-Id: I586c95d7db41820b8ec43565ba4016ed3834d1b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7030
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-10-16 23:35:59 +00:00
James Landrein
5ee2258692 feat(tvix/eval): implement builtins.partition
Change-Id: I8b591f3057c68c1542046fc5a771973f2238c9df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7020
Autosubmit: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-16 22:51:53 +00:00
sterni
4dcb8f38c2 fix(tvix/eval): resolve home relative paths at runtime
Home relative paths depend on the environment to be resolved. We have
elected to do everything that depends on the environment, e.g. resolving
SPATH expressions using NIX_PATH, at runtime, so tvix evaluation would
continue to behave correctly even if we separated the compilation and
execution phases more, e.g. via serializing bytecode. Then the value of
HOME, NIX_PATH etc. could reasonably change in the time until execution,
yielding wrong results if the resolution results were cached in the
bytecode.

We also take the opportunity to fix the broken path concatenation
previously found in the compiler, fixing b/205.

Another thing we could consider is emitting a warning for home relative
path literals, as they are by nature relatively fragile.

One sideeffect of this change is that home path resolution errors
become catchable which is not the case in C++ Nix. This will need to be
fixed up in a subsequent change.

Change-Id: I30bd69b575667c49170a9fdea23a020565d0f9ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7024
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2022-10-16 19:11:23 +00:00
sterni
0624d78af0 refactor(tvix/eval): make OpFindFile use internal UnresolvedPath
To assert that OpFindFile is only emitted for specially compiled SPATH
expressions, as well as make sure it doesn't accidentally operate on
“ordinary values”, introduce an UnresolvedPath internal value. If
OpFindFile sees a non-UnresolvedPath value, it'll crash.

Note that this change is not done purely for OpFindFile: We may want to
compile SPATH expressions as function calls to __findFile (like C++ Nix
does) in the future, so the UnresolvedPath value would definitely need
to be an ordinary string again then. Rather, this change is done in
preparation for resolving home dir relative paths at runtime (since they
depend on the environment) for which we'll need a similar mechanism to
OpFindFile.

Change-Id: I6acf287f35197cd9e13377079f972b9d36e5b22e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7023
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 19:11:23 +00:00
Vincent Ambo
0abc66ad91 feat(tvix/eval): add an AST pretty-printing module
This implements serde::Serialize for the rnix AST through a wrapper
type, and exposes a function for serialising the AST into
a (pretty-printed JSON) string representation.

This can be used to debug issues with the AST, and to display an AST
reprsentation in tools like tvixbolt.

Serialize is implemented manually because we don't own any of the
structs and the way to traverse them is not easily derived
automatically, and this is quite verbose. We might be able to condense
it a little bit, but at the same time it's also fairly straightforward.

Change-Id: I922df43cfc25636f3c8baee7944c75ade516055c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6943
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 12:26:56 +00:00
Adam Joseph
d9c497520c feat(tvix/eval): remove Clone instance from Chunk and Lambda
Lambda has a quite large and variable-sized runtime representation,
unlike Rc<Lambda>.  It would be easy to accidentally call clone() on
this and create input-dependent performance regressions.

Nothing in the codebase is currently using Lambda.clone().  Let's
remove the derived instance.  If it's really needed it is very easy
to add it back in, but whoever does that will have to look at the
struct they're adding Clone to, which will hopefully prompt them to
think about whether or not that's really what they want to do.

Change-Id: I7806a741862ab4402229839ce3f4ea75aafd6d12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7029
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-16 09:23:03 +00:00
sterni
db70c672cf fix(tvix/eval): don't coerce variable name to string
Change-Id: I8aa878dee009901feb453c489ce37c12fa3a31a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7026
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-16 09:20:02 +00:00
Adam Joseph
96dcc51297 refactor(tvix/eval) remove Value::DynamicUpvalueMissing
I believe this variant is left over from a previous implementation.
If not, please let me know.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I02a3bf2f63794d09e96a5a92a034c0ad3d1ff221
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7027
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 09:17:02 +00:00
Griffin Smith
3f45f6191d feat(tvix/eval): Implement builtins.intersectAttrs
Change-Id: Iaba9bcfa19f283cd0c1931be2f211e2528a1a940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6998
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2022-10-15 20:35:22 +00:00
Griffin Smith
03a3189a3d feat(tvix/eval): Initial impl of builtins.match
Implement an *initial* version of builtins.match, using the rust `regex`
crate for regular expressions. The rust regex crate definitely has
different semantics than nix's regular expressions - but we'd like to
see how far we can get before the incompatibility starts to matter.

This consciously leaves out any sort of memo for compiled regular
expressions (which upstream nix also has) for the sake of expediency -
in the future we should implement that so we don't have to compile the
same regular expression multiple times.

Change-Id: I5b718635831ec83397940e417a9047c4342b6fa1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6989
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 20:35:22 +00:00
Griffin Smith
5eb89be682 feat(tvix/eval): Implement builtins.fromJSON
Using `serde_json` for parsing JSON here, plus an `impl FromJSON for
Value`. The latter is primarily to stay "dependency light" for now -
likely going with an actual serde `Deserialize` impl in the future is
going to be way better as it allows saving significantly on intermediary
allocations.

Change-Id: I152a0448ff7c87cf7ebaac927c38912b99de1c18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6920
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 20:35:22 +00:00
Profpatsch
277c69cbe5 chore(tvix): fix vscode rust-analyzer recommendation
They renamed the thing.

Change-Id: If8b9a05581c684793dd42ce4880901c0be3c4b98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7021
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 16:54:28 +00:00
sterni
f3c27f1717 fix(tvix/eval): bring foldl' strictness in line with C++ Nix
Working on https://github.com/NixOS/nix/pull/7158, I discovered that C++
Nix actually is strict in the accumulator, just not in the first value.
This seems due to the fact that in the C++ evaluator, function calls
don't seem to be thunked unconditionally and foldl' just elects not to
wrap it in a thunk (don't quote me on this summary, even though it seems
to line up with the code for primop_foldlStrict and testable behavior).

It doesn't seem worth it to risk breaking the odd Nix expression just to
be strict in one more value per invocation of foldl' (i.e. the initial
accumulator value `nul`), so let's match the existing C++ Nix behavior
here.

Change-Id: If59e62271a90d97cb440f0ca72a58ec7840d1690
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7022
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 14:12:23 +00:00
Adam Joseph
caf9cbf711 feat(tvix/eval): implement builtins.dirOf
This commit causes the test eval-okay-builtins.nix to pass.

It also adds tests/tvix_tests/eval-okay-dirof.nix which has better
coverage than the nix tests for this builtin.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I71d96b48680696fd6e4fea3a9861742b35cfaa66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6987
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-15 12:02:05 +00:00
Adam Joseph
d6088005ef docs(tvix/eval) vm: explain VM::{frames,stack,with_stack}
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I94ba31ae25c1ff744f929a722c76a0c33cc361ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7016
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-14 09:35:28 +00:00
Adam Joseph
ad71fdaa75 feat(tvix/eval): implement builtins.toPath
This commit implements builtins.toPath.  Like OP_ADD, it currently
does not handle string contexts.

This commit allows the
tests::nix_eval_okay_src_tests_nix_tests_eval_okay_pathexists_nix
test to pass.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Iadd4f7605f8f297adbd0dba187b8481c21370b6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6996
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-14 09:23:29 +00:00
Adam Joseph
f868e730be feat(tvix/eval): eliminate the only unsafe in the codebase
Maybe I misunderstood this part of the code, but the use of `unsafe`
appears unnecessary here?  In any event it is the one and only
`unsafe` in the codebase.

Hopefully getting to "no `unsafe` anywhere" is worth the extra
never-taken branch caused by unwrap() instead of unwrap_unchecked()?

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I33fbd5aad9d8307ea82c24b6220412783e1973c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7011
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-14 09:18:45 +00:00
Adam Joseph
91ad5b825b refactor(tvix/eval): remove OpResolveWithOrUpvalue
The case branch in vm.rs for OpResolveWithOrUpvalue is
unreachable/deadcode.

I believe this opcode is unnecessary, since it should always be
statically detectable (at parse-time) whether a reference is to an
upvalue (i.e. enclosing binding); otherwise, and only then, is
with-resolution applicable.

Perhaps I've misunderstood how with-resolution works.  If so, please
explain it to me and -1/-2 this CL.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I4a90b9eb6cb3396df92a6a943d42ecc301871ba0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7009
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-14 09:18:38 +00:00
James Landrein
cbbe283b6f refactor(tvix/eval): order builtins alphabetically
This makes it easier to compare currently implemented ones with the full
list.

Change-Id: Ibaffd99d05afa15fc9ab644fd101afa24fc7a1b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7008
Tested-by: BuildkiteCI
Autosubmit: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-14 08:53:21 +00:00
Adam Joseph
d4bc58b38d docs(tvix/eval): explain where to find the site-specific mg tool
When I first read the `README.md` I didn't know what this `mg build`
was.  Some grepping around turned up `/tools/magrathea`.  Apparently
this thing "helps me build planets".  Awesome!  Let's tell other
people where to find it.  It seems that this tool is (currently)
specific to depot, so people who arrive at depot because of tvix
probably won't know where to find `mg`.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Icb8c51087fd41b696794516abcfee24a4b3b4a14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6948
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
2022-10-13 18:41:02 +00:00
Adam Joseph
a545f21cc0 feat(tvix/eval): implement builtins.baseNameOf
This commit implements builtins.baseNameOf and adds a test case
eval-okay-basenameof.nix to the test suite.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib8bbafba2ac9ca0e1d3dc5e844167f94890d9fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6997
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-13 17:43:17 +00:00
Adam Joseph
a98cb9d56f fix(tvix/eval): parseDrvName should not coerce, and xfail test
builtins.parseDrvName should not coerce its argument to a string.
This commit fixes that oversight in my previous commit, and adds an
xfail test to cover this condition.

Thanks to @sterni for noticing this.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I76bc78f1a82e1e08fe5c787c563a221d55de2639
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6991
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-10-13 17:42:22 +00:00
Adam Joseph
01bc04b1d2 feat(tvix/eval): enable the XFAIL tests
This commit adds support for running the "expected failure" tests in
both the nix and tvix test suites.

I have disabled the eval-fail-blackhole.nix test because it gets
stuck running forever.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Iba75ce6c8f2becab3c834fcfdd9f4fdc5a4bdb9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6990
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-13 17:42:04 +00:00
Adam Joseph
706186eb5d test(tvix/eval): update for cppnix PR #7149
cppnix has merged #7149, so let's update our copy of their tests:

  https://github.com/NixOS/nix/pull/7149

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I0be3bf9da937abd24102e1997daa2087ecfafd98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6992
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-13 17:40:34 +00:00
Vincent Ambo
0f24120a6c fix(tvix/eval): fix Compiler::new on wasm
This path normalisation business causes runtime panics on WebAssembly
because those operations are unsupported.

Maybe this shouldn't be happening in the compiler anyways, not sure,
but for now this commit adds a workaround based on the target to
disable the normalisation if we're compiling for wasm.

Change-Id: I908a84fbdffc3401f8d443e2c73ec673e9f397ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7004
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-13 16:29:49 +00:00
Adam Joseph
78d3d9150b fix(tvix/eval): src/compiler: ensure root_dir is absolute
Cppnix immediately absolutizes pathnames at parse time; if you write
`./foo`, it is immediately converted to `$(pwd)/foo` and manipulated
as an absolute path at all times.

To avoid having to introduce filesystem access operations in the
implementation of otherwise-pure builtins, let's guarantee that the
`root_dir` of the VM is always an absolute path.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I7cbbae2cba4b2716ff3f5ff7c9ce0ad529358c8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6995
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-13 09:48:32 +00:00
Adam Joseph
25336fc47b refactor(tvix/eval): factor out all calls to canon_path
Right now we're pretending that the Rust library path_clean does the
same thing that cppnix's canonPath() does.  This is not true.  It's
close enough for the test suite, but may come back to bite us.

Let's create our own canon_path() function and call that in all the
places where we intend to match the behavior of cppnix's
canonPath().  That way when we fix this we can fix it once, in one
place.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ia6f9577f62f49ef352ff9cfa5efdf37c32d31b11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6993
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-13 09:07:47 +00:00
Adam Joseph
534a2f95f8 fix(tvix/eval): nix_search_path.rs: use /etc instead of /bin
On Debian machines (and most FHS distros) /bin is now a symlink to
/usr/bin, which causes this test to fail.  Let's use /etc as a
root-relative test case, since it is less likely to be a symlink.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I1c94de0d2a242394182442fe1c895dc17eb04a4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6994
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-13 09:07:30 +00:00
Adam Joseph
e83609a061 feat(tvix/eval): allow to disable warnings
The nix_tests test suite produces lots of warnings.  We can't fix
these, since they are kept in sync with upstream, so there's little
point in cluttering up the console with them every time the tests
are run.

Let's add a clap flag "warnings" and TVIX_WARNINGS environment
variable.  The default is "true".  The test runner overrides this
default and mutes the warnings.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I4b065f96fe15838afcca6970491a54e248ae4df7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6985
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-13 02:49:59 +00:00
Adam Joseph
224ec708f7 test(tvix/eval): add eval-okay-parsedrvname.{exp,nix}
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I59254a0ac513b3d2a6b2184554358bb6eba2ba99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6949
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-12 09:50:21 +00:00
Adam Joseph
8111f2de1e feat(tvix/eval): builtins: implement parseDrvName
This commit passes nix_eval_okay_src_tests_nix_tests_eval_okay_versions_nix.

See also: https://github.com/NixOS/nix/pull/7149

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I24605c2a0cd0da434f37f6c518f20693bfa1b799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6913
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-12 09:50:10 +00:00
Adam Joseph
32ac7d6c6d refactor(tvix/eval) s/NixPath/NixSearchPath/
Since NixString is the Rust type for nix strings, people might
mistake NixPath for the Rust type of nix paths, which it is not.
Let's call it NixSearchPath instead.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib2ea155c4b27cb90d6180a04ea7b951d86607373
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6927
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-12 08:09:49 +00:00
Adam Joseph
f2bf65a40b fix(tvix/eval): remove overlapping instances
I broke the build.

I didn't understand that before hitting "submit" you need to re-test
your changes on latest HEAD (and that CI doesn't do that for you); I
failed to re-test cl/6912 following the merge of cl/6907.

This commit fixes the build by removing the overlapping instances.

Change-Id: I2a720d2c60cc7103b350f78102a8998f93bac828
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6965
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-12 07:25:51 +00:00
Adam Joseph
4e94a2696b feat(tvix/eval): From<Into<NixString>> for Value
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I27615baf4104ab96600708b12a780599a3714bae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6912
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-12 04:23:07 +00:00
Adam Joseph
2e0a4aaa84 feat(tvix/eval): From<Utf8Error> for ErrorKind
Change-Id: I074d9e862fbdd4e78e443cbaf0e77c7ffe825a10
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6911
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-12 04:22:44 +00:00
sterni
fcd5e53703 fix(tvix/eval/builtins): force acc not list element in foldl'
When investigating discrepancies between foldl' in tvix and C++ Nix,
I discovered that C++ Nix's foldl' doesn't seem to be strict at all.
Since this seemed wrong, I looked into Haskell's foldl' implementation
which doesn't force the list elements (`val` in our code), but the
accumulation value (`res` in our code). You can look at the code here:
https://hackage.haskell.org/package/base-4.17.0.0/docs/src/GHC.List.html#foldl%27

This actually makes a lot of sense: If `res` is not forced after each
application of `op`, we'll end up thunks nested as deeply as the list is
long, potentially taking up a lot of space. This can be limited by
forcing the `res` thunk before applying `op` again (and creating a new
thunk).

I've also PR-ed an equivalent change for C++ Nix at
https://github.com/NixOS/nix/pull/7158. Since this is not merged nor
backported to our Nix 2.3 fork, I've not copied the eval fail test yet,
since it wouldn't when checking our tests against C++ Nix in depot.

Change-Id: I34edf6fc3031fc1485c3e714f2280b4fba8f004b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6947
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-11 15:53:29 +00:00
Griffin Smith
1181bd78fc feat(tvix/eval): Allow directly evaluating an expr via main
This *maybe* should do something to check that we don't pass both a file
and an expr, but for now this is useful enough to cut corners (plus
we're probably due for a CLI revamp eventually anyway).

Change-Id: Id44357074150b336b6215ba596cc52d01d037dbd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6941
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-11 00:33:49 +00:00
Griffin Smith
4283f0139a feat(tvix/eval): Implement builtins.removeAttrs
Change-Id: I28910991a0108436a42ac7bf3458f9180a44154e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6928
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-11 00:33:39 +00:00