Commit graph

434 commits

Author SHA1 Message Date
Vincent Ambo
bb34665abd refactor(tvix/eval): extract attribute set inherit into helper
This will be re-used between the code paths for
recursive/non-recursive sets, and it might even be possible to unify
it with the logic for compiling `let inherit ...`.

Change-Id: I960a061048ac583a6e932e11ff6e642d9fc3093e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6464
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-11 12:04:27 +00:00
Vincent Ambo
f86327beac feat(tvix/eval): implement "formals" function parameters
The comment explains how this works fairly well.

Note that this does not yet have the ability to check "closed
formals", i.e. without an ellipsis Tvix will *NOT* fail if unexpected
attribute set keys are provided.

Change-Id: I0d2b77e893243093d2789baa57f876d35d0a32ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6463
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-11 10:32:10 +00:00
Vincent Ambo
8643620eb1 fix(tvix/eval): always add history entries in REPL
... even if the code is broken.

Change-Id: I5898bceaebf201b97e8988c94c90e7fafff82529
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6462
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-10 21:57:13 +00:00
Vincent Ambo
4e06e5d2ba fix(tvix/eval): reintroduce 'InvalidAttribuetName' error variant
As pointed out by sterni in cl/6205, this is actually possible in
syntactically valid expressions like

  { ${12 + 13} = 12; }

Change-Id: Id8a1e3aceb551f288f9050c4eea563eb6572f1a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6461
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-10 21:57:13 +00:00
Vincent Ambo
06909f1821 fix(tvix/eval): fix doc comment syntax where applicable
As pointed out by grfn on cl/6091

Change-Id: I28308577b7cf99dffb4a4fd3cc8783eb9ab4d0d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6460
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-10 21:57:13 +00:00
Vincent Ambo
83dd706a3a feat(tvix/eval): conditionally use tracing/disassembling observers
Gates the observes behind the envvars `TVIX_DUMP_BYTECODE` and
`TVIX_TRACE_RUNTIME`.

(hi grfn, yes, we should probably introduce CLI flags soon)

Change-Id: I4fa194a84b04593d609b96b44471c3644fb30296
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6459
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-10 21:57:13 +00:00
Vincent Ambo
6bbe7589c5 feat(tvix/eval): optimise tail calls in emitted chunks
When the last instruction in a chunk is OpCall, make it an OpTailCall instead.

Change-Id: I2c80a06ee85e4abf545887b1a79b6d8b5e6123e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6458
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-10 21:57:13 +00:00
Vincent Ambo
2e018a50a7 feat(tvix/eval): implement OpTailCall
If the last operation within a chunk is a function call, the call can
be executed in the same call frame without increasing the depth of the
call stack.

To enable this, a new OpTailCall instruction (similar to OpCall) is
introduced, but not yet emitted by the compiler.

Change-Id: I9ffbd7da6d2d6a8ec7a724646435dc6ee89712f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6457
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-10 21:57:13 +00:00
Vincent Ambo
6deaa0d6ce fix(tvix/eval): force value passed to builtins.toString
This introduces a macro to do the forcing, but this solution isn't
very nice and also does not work in all cases yet.

Change-Id: Icd18862ec47edb82c0efc3af5835a6cb6126f629
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6456
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:14:56 +00:00
Vincent Ambo
0aeca64777 chore(tvix/eval): clean up a stale comment
Change-Id: If1b02fe1c78398387ea98490e5b099f1ff1b4164
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6455
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:14:56 +00:00
Vincent Ambo
e2a942e4b2 chore(tvix/eval): clean up remains of previous disassembler impl
Change-Id: Ib402ea23a58dc52ed0c5a97178cb5d0e53d69300
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6454
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:14:56 +00:00
Vincent Ambo
e03a729fa3 feat(tvix/eval): implement TracingObserver for runtime tracing
This produces similar output to the previous tracing feature, but can
redirect the output somewhere else.

Change-Id: I9493c260f480904f3932cb74809b622c24d7be96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6453
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:14:56 +00:00
Vincent Ambo
14ff889d60 feat(tvix/eval): implement runtime tracing methods for Observer
These methods make it possible to trace the runtime execution of the
VM through an observer.

Change-Id: I90e26853ba2fe44748613e7f761ed5c1c5fc9ff7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6452
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:10:10 +00:00
Vincent Ambo
cbf2d2d292 refactor(tvix/eval): move disassemble_op to the Chunk structure
Change-Id: Ic6710c609ed647bfa47d673aaf22c4da96c0f319
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6451
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:10:10 +00:00
Vincent Ambo
3cf5c40209 chore(tvix/eval): export some symbols from the crate
These are required for tvixbolt to work. This interface is definitely
not stable yet, though.

Change-Id: I4076498e8f42311de74ee4f33c93a3ee0c5f8d3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6450
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:10:10 +00:00
Vincent Ambo
8ee4d6d5db feat(tvix/eval): implement DisassemblingObserver for compiler
This type implements an observer that is called whenever the compiler
emits a chunk (after the toplevel, thunks, or lambdas) and prints the
output of the disassembler to its internal writer.

This replaces half of the uses of the `disassembler` feature, which
has been removed from the Cargo configuration.

Note that at this commit runtime tracing is not yet implemented as an
observer.

Change-Id: I7894ca1ba445761aba4ad51d98e4a7b6445f1aea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6449
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:10:10 +00:00
Vincent Ambo
7ae45342df feat(tvix/eval): implement (compilation) observer trait
This trait will enable library users of tvix-eval to observe internal
happenings of the compilation and runtime processes.

The initial methods of the observer will be called whenever the
compiler emits a chunk.

Change-Id: I668f6c2cfe3d6f4c1a1612c0f293831011768437
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6448
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:10:10 +00:00
Vincent Ambo
1fe6cfe5a2 refactor(tvix/eval): index into Chunk with ConstantIdx/CodeIdx
This is a step towards hiding the internal fields of thunk, and making
the interface of the type more predictable.

Part of the preparation for implementing observers.

Change-Id: I1a88a96419c72eb9e2332b56a2dd94afa47e6f88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6447
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-09 21:10:10 +00:00
Vincent Ambo
8fdb67847c chore(tvix/eval): debug_assert that all jumps are patched
Suggestion from sterni in cl/6282

Change-Id: I1adbdda9ff74f55a2f72892ffa524808b305f403
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6445
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-09 10:48:10 +00:00
Vincent Ambo
abdfa7459e feat(tvix/eval): thunk binary operations and select expressions
With this, most cases of `fix` in attribute sets will work correctly.
A simple test exercising both has been added.

Change-Id: I70fd431177bb6e48ecb33a87518b050c4c3d1c09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6437
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
9f379ef6df fix(tvix/eval): hold thunk borrow as shortly as possible
At the point where control flow exits Thunk::force (which may be due
to recursing), it is vital that there is no longer a borrow to the
inner thunk representation, otherwise this can cause accidental
infinite recursion (which will be detected, but cause failures on
valid code).

Change-Id: I2846f3142830ae3110a4f5d2299e9d7928634504
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6436
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
48d5f4fd57 feat(tvix/eval): print lambda memory adresses in disassembler
This makes it easier to track exactly which lambda is which when
inspecting e.g. the concrete representation of a thunk.

At runtime all lambdas live in an Rc. To make this print the right
address, the construction of these Rcs had to be moved up right to the
point where the lambda is first emitted (and disassembled).

Change-Id: I6070e6c8ac55f0bd697966c4e7c5565c20d19106
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6435
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
d3421c1cb9 fix(tvix/eval): ensure disassembler prints continous lines correctly
There can be different spans on the same line, so the previous
implementation would duplicate line numbers unnecessarily.

Change-Id: I8d8db77177aee0d834a6ec3584641e1bd5f31c3e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6434
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
fc1c50498e feat(tvix/eval): thunk function applications
Change-Id: I18065ed234ec104ac74d0e1c2d0937c2d78ca7db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6433
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
e129ce15d7 feat(tvix/eval): thunk creation of lists
Change-Id: I84b68c5d002ec613d278315bbf49e9839f0fe8e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6432
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
678533bfc4 test(tvix/eval): add test for stack slot accounting edge-case
This was fixed by some of the previous commits around scopes. It's
somewhat similar to a few other tests, but I had this one failing
earlier and everything else succeeding, so it is useful to keep it
around for sure.

Change-Id: Ie6cf372b5c805daf992cd87aeb3dfe91542c381c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6431
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
09eaa0d4ae fix(tvix/eval): address current clippy & grfn lints
Change-Id: I65c6feb9f817b5b367d37204a1f57acfe4100d97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6430
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
fe047885d7 fix(tvix/eval): consider local depth when deciding to defer
Deferred local upvalues can *only* occur at the same depth as the
thing that is closing over them, but there are various situations with
scope nesting where the actual stack indexes of the local and the
closer look like a deferred value is being accessed.

To fix this, simply compare the depth as well.

Change-Id: Ice77424cc87ab0a2c4f01379e68d4399a917b12b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6429
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 20:17:26 +00:00
Vincent Ambo
360c805efc chore(tvix/eval): remove accidentally duplicated test
This is the same as `eval-okay-attrs-simple-inherit`.

Change-Id: I23878accc6cd62c16ec96601239838a385d31306
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6428
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 19:54:02 +00:00
Vincent Ambo
0af1df4be2 refactor(tvix/eval): clean up logic in Compiler::end_scope
The condition here was extremely hard to read prior to this change.

As the locals vector is now guaranteed to never be empty (there is
always at least a phantom for the current chunk's root expression),
the logic here can be simplified to just dropping tailing locals
entries while their depth matches that of the scope being closed.

Change-Id: I24973e23bc2ad25e62ece64ab4d8624e6e274c16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6427
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 19:54:02 +00:00
Vincent Ambo
23f248b530 fix(tvix/eval): set up root stack slot in closures & thunks
Similar to setting up a phantom slot when compiling the root value of
a file, closures and thunks need to have a phantom stack slot for the
root of the expression yielded by their thunk to make all accounting
work correctly.

The tricky thing here is that closures & thunks *escape* their inner
lambda context (that's the point!), so the functions emitting them
need to know both the *inner* slot (to resolve everything correctly
while compiling the slot) and the *outer* slot (to correctly emit
instructions for closing over upvalues).

Change-Id: I62ac58e2f639c4b9e09cc702bdbfd2373e985d7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6426
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 19:54:02 +00:00
Vincent Ambo
4e24bd56b4 fix(tvix/eval): only pop initialised locals when closing scopes
This avoids emitting OpPop instructions for locals that only existed
virtually (as uninitialised phantoms).

Change-Id: I8105afcca80c3f7b7ef93ce5e2f0d08a93f4ad27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6425
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 19:54:02 +00:00
Vincent Ambo
ecbd7c6ca1 fix(tvix/eval): ensure that root stack slot actually exists
Instead of using a sentinel LocalIdx which potentially points to a
value in the locals stack that does not actually exist, set up an
initial uninitialised phantom value representing the result of the
root expression.

Change-Id: I82ea774daab83168020a3850bed57d35ab25c7df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6424
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 19:54:02 +00:00
Vincent Ambo
f8575fcd74 fix(tvix/eval): compare *stack* slots when deciding whether to defer
When deciding whether an upvalue needs to have a deferred resolution
step, the *stack* indexes should be compared - not the locals indexes.

The results are almost always the same, but there are tricky
situations where this can cause errors.

It's difficult to reproduce these errors in isolation, as they depend
on other scope behaviour, so this is one in a series of commits to
address the combination of issues which will gain some tests at the
end.

Change-Id: Iaa400b8d9500af58f493ab10e4f95022f3b5dd21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6423
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 19:54:02 +00:00
Vincent Ambo
e9b6ecb0ee fix(tvix/eval): propagate scope depth when nesting scopes
Change-Id: Id441646db550f6195c2e247a0afbb5c9d91da8a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6422
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 13:36:26 +00:00
Vincent Ambo
9973ddfcba refactor(tvix/eval): refactor locals to use an enum for phantoms
Instead of using sentinel values and an additional bool, this tracks
the identifier of a local as an enum that is either a statically known
name, or a phantom.

To make this work correctly some more locals related logic has been
encapsulated in the `scope` module, which is a good thing (that's the
goal).

Phantom values are now not initialised by default, but the only
current call site of phantoms (`with` expression compilation) performs
the initialisation right away.

This commit changes no actual functionality right now, but paves the
way for fixing an issue related to `let` bodies.

Change-Id: I679f93a59a4daeacfe40f4012263cfb7bc05034e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6421
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 13:36:26 +00:00
Vincent Ambo
7bc6e5984d refactor(tvix/eval): always pass slot to compiler methods
The slot is now always known (at the root of the file it is simply
stack slot 0 once the scope drops back down to 0), so it does not need
to be wrapped in an `Option` and accessed in cumbersome ways anymore.

Change-Id: I46bf67a4cf5cb96e4874dffd0e3fb07c551d44f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6420
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 13:36:26 +00:00
Vincent Ambo
a303ea3ff5 refactor(tvix/eval): implement much clearer disassembler output
With this change the runtime trace contains much more exact
information about the context of the computation (entering/exiting
calls etc.)

This is in large part due to moving the tracer to be a field on the VM
itself, which enables consistent ordering of traces across the
execution, and tracing an execution with its *input* instead
of *output* stack.

Change-Id: Ibe525e6e7d869756501e52bef1a441619ce7332c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6419
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 13:36:26 +00:00
Vincent Ambo
60ff8d046c chore(tvix/eval): print value when panicking on internals
This makes it much easier to figure out what happened while debugging
this sort of thing.

Change-Id: I2e0e8096709adc647d63c04f213c547c415e5f44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6418
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 13:36:26 +00:00
Vincent Ambo
b83723a1ed fix(tvix/eval): force argument of builtins.length
Change-Id: Iaf94dfc7cb058f5c1c311066d92e01512e09cf71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6417
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 12:53:20 +00:00
Vincent Ambo
0c0ae50f02 fix(tvix/eval): don't panic when printing a black hole
This could occur when the disassembler is enabled and tracing the
runtime while a thunk is being evaluated, as it would not be possible
for the *tracer* to borrow the thunk at this exact moment.

However, we know that if the borrowing fails here we are dealing with
a not-fully evaluated thunk (blackhole), which should just print the
internal representation.

Change-Id: I4bdb4f17818d55795368e3d28842048f488f0a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6416
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 12:53:20 +00:00
Vincent Ambo
2246a31e72 refactor(tvix/eval): return call frame result from VM::call
Previously, "calling" (setting up the VM run loop for executing a call
frame) and "running" (running this loop to completion) were separate
operations.

This was basically an attempt to avoid nesting `VM::run` invocations.
However, doing things this way introduced some tricky bugs for exiting
out of the call frames of thunks vs. builtins & closures.

For now, we unify the two operations and always return the value to
the caller directly. For now this makes calls a little less effective,
but it gives us a chance to nail down some other strange behaviours
and then re-optimise this afterwards.

To make sure we tackle this again further down I've added it to the
list of known possible optimisations.

Change-Id: I96828ab6a628136e0bac1bf03555faa4e6b74ece
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6415
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 12:53:20 +00:00
Vincent Ambo
cc526a2c87 feat(tvix/eval): thread codemap through to disassembler
If the disassembler feature is enabled, make sure that an Rc of the
codemap is available through the chunk.

Change-Id: I700f27ab665a704f73457b19bd2d7efc93828a16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6414
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 12:53:08 +00:00
Vincent Ambo
6b3c3c9826 refactor(tvix/eval): add macros for generating Value casters
The casting methods of `Value` are pretty verbose, and actually
incorrect before this commit as they did not account for inner thunk
values.

To address this, we first attempt to make them correct by introducing
a standard macro to generate them and traverse the inner thunk(s) if
necessary.

This is likely to be a performance hit as it will now involve more
cloning of values. We can do multiple things to alleviate this, but
should do some measurements first.

Change-Id: If315d6e2afe7b69db727df535bc6cbfb89a691aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6412
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 08:45:15 +00:00
Vincent Ambo
0d7ad5e6d1 refactor(tvix/eval): pass a VM reference to builtins
This makes it possible for builtins to force values on their own,
without the VM having to apply a strictness mask to the arguments
first.

Change-Id: Ib49a94e56ca2a8d515c39647381ab55a727766e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6411
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 07:59:15 +00:00
Vincent Ambo
5ee89bcf5c fix(tvix/eval): inherit scope poisoning data in nested contexts
Scope poisoning must be inherited across lambda context boundaries,
e.g. if an outer scope has a poisoned `null`, any lambdas defined on
the same level must reference that poisoned identifier correctly.

Change-Id: I1aac64e1c048a6f3bacadb6d78ed295fa439e8b4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6410
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 07:59:15 +00:00
Vincent Ambo
0a13d267f0 fix(tvix/eval): thread thunk forcing errors through correctly
With this, if an error occurs while forcing a thunk (which is very
likely) it is threaded through to the top by wrapping it in the
ErrorKind::ThunkForce variant.

We could use this to generate "stacktrace-like" error output if we
wanted, or simply jump through and discard everything except the
innermost error.

Change-Id: I3c1c8708c2f73ae062815adf490ce935b1979da8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6409
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 07:59:15 +00:00
Vincent Ambo
377ba19d75 feat(tvix/eval): ensure all errors always carry a span
Previously error spans were optional because the information about
code spans was not available at runtime. Now that this information has
been added, the error type will always carry a span.

This change is very invasive all throughout the codebase. This is due
to the fact that many functions that are called *by* the VM expected
to return `EvalResult`, but this no longer works as the span
information is not available to those functions - only to the VM
itself.

To work around this the majority of these functions have been changed
to return `Result<T, ErrorKind>` instead and an accompanying macro in
the VM constructs the "real" error.

Note that this implementatino currently has a bug where errors
occuring within thunks will yield the location at which the thunk was
forced, not the location at which the error occured within the code.
This will be fixed soon, but the commit is large enough as is.

Change-Id: Ib1ecb81a4d09d464a95ea7ea9e589f3bd08d5202
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6408
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-08 07:59:15 +00:00
William Carroll
197fe37dae feat(tvix/eval): Support builtins.length
Get the length of a list

Change-Id: I41d91e96d833269541a1b3c23b7cc879f96d1e5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6407
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 20:56:02 +00:00
William Carroll
15e2bc54f1 feat(tvix/eval): Support builtins.{mul,div}
Multiplication and division :)

Change-Id: I79e76f3b55b7e9b8d1eb09b3d9741140b4d75742
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6406
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 20:56:02 +00:00
William Carroll
b3097de75d feat(tvix/eval): Support builtins.{add,sub}
First pass at supporting `builtins` for tvix. The following tests appear to be
WAI:

```shell
$ cd tvix/eval
$ cargo build
$ cargo test
```

Change-Id: I27cce23d503b17a886d1109e285e8b4be4264977
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6405
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 20:56:02 +00:00
Vincent Ambo
55d21a1389 refactor(tvix/eval): store spans instead of nodes in Warning/Error
Another step towards being able to report accurate errors. The codemap
spans contain strictly more accessible information, as they now retain
information about which input file something came from.

This required some shuffling around in the compiler to thread all the
right information to the right places.

Change-Id: I18ccfb20f07b0c33e1c4f51ca00cd09f7b2d19c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6404
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:26 +00:00
Vincent Ambo
8033300900 feat(tvix/eval): track source spans for builtin access
As of this commit, the source spans of all emitted bytecode are fully
tracked.

Change-Id: I4c83deee0fc3f5e6fd6acad5a39047aec693b388
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6403
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
6cbd580ba5 feat(tvix/eval): track source spans for OpForce instructions
These source spans will always point to the *value* that is being
forced, not the instruction that caused the force to be emitted. This
makes sense so that errors during forcing point at the value and not
the surrounding expression.

Change-Id: I4694414a3281a0de878f71634105b92148ec61f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6402
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
1ea88fcb65 feat(tvix/eval): track source spans for scopes
Change-Id: Iaedb0742940e4c2b9f4210579ed28e6deb32d5c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6401
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
c269b44f41 feat(tvix/eval): track source spans for upvalues
With this change, the upvalue data instructions used by finalisers for
thunks and closures track the source span of the first identifier that
created the upvalue (if the same value is closed over multiple times
the upvalue will be reused, hence only the first one).

To do this the upvalue struct used by the compiler's scope now carries
an identifier node, which had to be threaded through quite a few
places.

Change-Id: I15a5fcb4c8abbd48544a2325f297a5ad14ec06ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6400
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
3efed26940 refactor(tvix/eval): split out Upvalue struct & UpvalueKind enum
This separation makes it possible to annotate the upvalue itself with
the span that created it, which (due to upvalue reuse) is only the
first one for an instance of the given UpvalueKind.

Change-Id: I9a991da6a3e8d71a92f981314bed900bcf434d44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6399
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
b8874f8d35 feat(tvix/eval): track source spans for thunks
Change-Id: I73f13c914d84e052a43b4eddb09db1acd8a7076a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6398
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
ec8ff79aee feat(tvix/eval): track source spans for function calls
Change-Id: If1bb463026414597acd561dc2032549fdb70f986
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6397
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
e590b585e7 feat(tvix/eval): track source spans for lambdas
Change-Id: I2f420fc915b6bfc5b197e9d0c128b539c4bc7467
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6396
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
20996601ea feat(tvix/eval): track source spans for with expressions
Change-Id: I1d58ce548b5b47e967928e85bb64acf5ed69ecc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6395
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 20:04:17 +00:00
Vincent Ambo
9f5f85a1c1 feat(tvix/eval): track source spans for identifier access
Change-Id: I8e6ec0a84430d6e417fc7fd3e722a588913e4d18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6394
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 19:23:22 +00:00
Vincent Ambo
f37e4a0b9c feat(tvix/eval): track source spans for let bindings
Change-Id: I9457917277a7fdd8bdbe227a567b28969312d06e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6393
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 19:23:22 +00:00
Vincent Ambo
7f687f0e99 feat(tvix/eval): track source spans for if expressions
These are again a bit tricky in terms of emitted errors. The main
error is that the condition is not a boolean, which means that the
jump inspecting the condition must derive from the condition itself to
return an error at the correct position.

For other parts of the expression, it is simply the node itself.

Change-Id: I72411630e5d57dfc199f4c3c48afe443fe966322
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6392
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
baa012d5a3 feat(tvix/eval): track source spans for assert
Change-Id: I5415f63ddde388847a261da4ce9a8d8235657535
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6391
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
805c1870ee feat(tvix/eval): track source spans for or operator
This one is tricky, specifically the span used for the final jump. I
decided that it makes sense to use the attrpath node, as the final
jump is the one that jumps *over* the default value, so the effect of
this is more closely related to the selector than the default.

It might be more correct to pass through the `or` token itself and
point to this for the jumps, but it depends a bit on what shape of
errors we could end up producing from this.

Change-Id: I29fbc97ba6b9e14e1a0e5f3a7759ddc299dd9c0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6390
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
2b1468dde3 feat(tvix/eval): track source spans for attribute selects
Change-Id: Ifa8b0e7905f9d2746f83d6503ef0e8d42ce20f9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6389
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
d1e0c6f680 feat(tvix/eval): track source spans for attrsets
Change-Id: I0fcb07146b5a38c67bc46ed3f67533b056858390
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6388
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
eb169c51a7 feat(tvix/eval): track source spans for lists
Change-Id: Icfd057ee29180cac17264bb3299c67eae0051aee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6387
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
d8f494b50b feat(tvix/eval): track source spans for literal identifiers
Change-Id: I7ca08c6c0124f653e55fcc86413a847f0a4c50c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6386
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
2ad89a00fb feat(tvix/eval): track source spans for ? operator
Change-Id: Idb842fb20fab14150455a81ea6c947e75b314d8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6385
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
c417a084cc feat(tvix/eval): track source spans for binary operators
Change-Id: I43e433ff3094232b244d38335d31e56a79ca70c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6384
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
176e729556 feat(tvix/eval): track source spans for unary operators
Change-Id: I79034a4aa04aea66ec598e33e6eab35e1e19c0d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6383
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
f14b368514 feat(tvix/eval): track source spans for strings
Change-Id: I29e6c7f9e25d1b2e6bafa602c463eb9ccee2131b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6382
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
3c4eed9b76 feat(tvix/eval): track source spans for paths
Change-Id: I42fbd0bb6c2a8feb520e262a25f59ff27dcd035c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6381
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
eaf7af18be feat(tvix/eval): track source spans for literals
Change-Id: Icfe77f85c4f65b6bf28b8752c2795419e8e396ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6380
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
72adcdf965 feat(tvix/eval): add methods for emitting code with tracked spans
These are not actually used yet; this is in preparation for a
multi-commit chain for emitting all the right spans in the right
locations.

Change-Id: Ie99d6add2696c1cc0acb9ab928917a10237159de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6379
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:23:22 +00:00
Vincent Ambo
c5a8b93eaf chore(tvix/eval): thread a codemap::File reference to the compiler
This instantiates a codemap outside of the compiler and passes a
reference to the file currently under compilation to it. Note that the
"file" might just be a REPL line.

Change-Id: I131ae1ddb6d718e1374750da9ba0b99608c6058d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6378
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:10:40 +00:00
Vincent Ambo
b9566da5c9 refactor(tvix/eval): add and use Compiler::push_op method
This is currently just a wrapper around Chunk::push_op, but will gain
the span resolution logic in a moment.

Change-Id: I862bf9ecff0932f8da6708401ea044b9442c5d5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6377
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:08:56 +00:00
Vincent Ambo
6f6bbc12f8 feat(tvix/eval): add data structures for tracking spans in chunks
This adds a new vector to the chunk data structure which tracks spans
into a codemap. The compiler will emit this information to the chunk
when adding instructions.

The internal representation of the spans is slightly optimised to
avoid storing duplicate spans, as there are cases where many
instructions might be derived from the same span.

Change-Id: I336f8c912e7eb50ea02ed71e6164f651ca3ca790
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6376
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 19:08:56 +00:00
Vincent Ambo
f8919dbcd6 fix(tvix/eval): address current clippy lints
Note that I've allowed `needless_lifetimes` for the attribute set
iterator, as I find the type easier to understand with these
annotations present.

Change-Id: I33abb17837ee4813076cdb9a87f54bac4a37044e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6373
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 19:08:56 +00:00
Vincent Ambo
afcfa78364 feat(tvix/eval): semi-strictly evaluate output values of the VM
This essentially makes the VM behave like `nix-instantiate --eval
--strict`, i.e. data structures are traversed strictly and thunks are
forced. Thunks embedded in closures are not forced.

This allows us to re-enable tests that were disabled because they
needed to output nested thunk contents, but is overall a behaviour
that must be configurable later on, as it is not cmopatible with e.g.
an evaluation of nixpkgs.

Change-Id: I5303a5c8e4322feab1384fdb7712fecb950afca5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6372
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 19:08:56 +00:00
Vincent Ambo
6c6f9ae661 feat(tvix/eval): implement NixList::iter
This does not require a custom iterator type (for now?)

Change-Id: I5beb194bd8629571bd4040c69c977c27149807fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6371
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 19:08:56 +00:00
Vincent Ambo
fea7f90a7f fix(tvix/eval): thread Display & PartialEq through to thunk values
If a thunk is already evaluated, there are cases where due to the
memoisation implementation something might observe a value wrapped in
a thunk.

In these cases, the implementation of `Display` and `PartialEq` must
delegate to the underlying value.

Note that there are a handful of other cases like these which we need
to cover.

It is a little tricky to write integration tests for these directly,
especially as some of the open-upvalue optimisations coming down the
pipe will reduce the number of observable thunks.

One test that covers a part of this behaviour is currently
disabled (needs some more machinery), but it's being brought back in
the next commits.

Change-Id: Iaa8cd338c12236af844bbc99d8cec2205f0d0095
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6370
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-07 19:08:56 +00:00
Vincent Ambo
38d3db5fb8 feat(tvix/eval): implement NixAttrs::iter()
Implementing iteration over NixAttrs requires a custom iterator type
in order to encapsulate the different representations. The BTreeMap
for example has its own iterator type which needs to be encapsulated.

This is mostly boilerplate code, but for a change some simple unit
tests have been added in.

Change-Id: Ie13b063241d461b810876f95f53878388e918ef2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6367
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
7d5dca7ba3 chore(tvix/eval): provide 'static references to "name"/"value"
These static strings show up a bunch when dealing with the internals
of attribute sets, and having them available as static references is
required.

Due to the way const expressions are evaluated, taking a reference to
the existing NixString::NAME / NixString::VALUE items does not work
and the references themselves need to be const-evaluated.

Change-Id: If6e75847af978118a3b266fe6a3242321722434d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6366
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
80713f207e refactor(tvix/eval): encapsulate all thunk-forcing logic in module
The VM previously took care of repeatedly forcing a thunk until it
reached an evaluated state. This logic is now encapsulated inside of
the `Thunk::force` implementation.

In addition, force no longer returns a reference to the value by
default, leaving it up to callers to decide whether they want to
borrow the value or not (a helper is provided for this).

Change-Id: I2aa7da922058ad1c57fbf8bfc7785aab7971c02b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6365
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
23a5caabec feat(tvix/eval): construct attribute sets lazily
This thunks the construction of attribute sets. Because Tvix does not
currently have a "strict output" mode, a test had to be disabled that
now displays a thunk representation.

The test will be re-enabled once that is available.

Change-Id: I360332be64cd5c154f9caea21828f6f1b37a265c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6363
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
79a2ba5175 fix(tvix/eval): avoid recomputing width in disassemble_op constantly
As noticed by sterni in cl/6195

Change-Id: Ie9c1e80e2e709284fa8412334af9188d999f64dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6361
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
a8b2ba07df fix(tvix/eval): correctly resolve identifiers in inherit
This should not have grown a second implementation of the identifier
resolution logic, but it somehow did.

This implementation ended up being incorrect because it did not
account for upvalues inside of thunks.

Change-Id: Ieb1364d8fe43c96aaf4b125fd4b8a522aedff167
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6360
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
17dfb92a9f chore(tvix/eval): implement Debug for compiler::scope::Scope
Change-Id: I112b0119bd0511f26bb72f7e73d867d1b7144a36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6359
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
9420a3b53d feat(tvix/eval): insert strictness points for attribute set keys
All attribute set *key* related operations strictly evaluate all key
fragments, including during construction of an attribute set.

Change-Id: I3519e5e9b0886c2cdc8615ea7dcb5f7be0c59b3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6358
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
5b4f3811e6 feat(tvix/eval): insert strictness points for unary/binary operators
The arguments of all unary/binary operators that are built in to Nix
are forced when encountered. This emits the necessary OpForce operations.

Change-Id: I691fcdbebfe7586cfe217c68d44b10b1192f82d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6357
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
6a42d9bddf chore(tvix/eval): thread slot value through all compiler methods
With this change any compilation of an expression is aware of its own
stack slot if it is leaving identifiers on the stack.

Change-Id: I0c9f148ae06b078a46b25180c4961686d5f2e166
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6356
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
50c33989dc chore(tvix/eval): thread slot value through Compiler:compile_attr
This threads the current local slot through the `compile_attr`
function and all of its callers. At the moment this does not improve
any user-facing behaviour, just internally changes the way in which
some correct expressions would fail to run.

Eventually this slot will need to reach everywhere ...

Change-Id: Iba73123dd1ced421093d8fc18ebeeffc16efacf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6355
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
372bf4a0b7 test(tvix/eval): test very simple late-bound identifier access
This is the simplest kind of thunk that can be created (and so far the
only one the compiler knows how to create), in which an identifier
inside a `let` encounters a value that is bound *after* it is
initialised.

Change-Id: I6ea4408a3baef1e7d5137365d70804283f2dbf8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6354
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
786e12a738 feat(tvix/eval): always emit OpForce as the last instruction
Change-Id: Id70c987f654dc5d9b47db74e395281309762b468
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6353
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
9a783e50a4 feat(tvix/eval): implement OpForce in VM
This operation forces the evaluation of a thunk.

There is some potential here for making an implementation that avoids
some copies, but the thunk machinery is tricky to get right so the
first priority is to make sure it is correct by keeping the
implementation simple.

Change-Id: Ib381455b02f42ded717faff63f55afed4c8fb7e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6352
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-07 15:25:59 +00:00
Vincent Ambo
a8c13a5ce1 fix(tvix/eval): allocate Thunk::upvalues with known capacity
The capacity (i.e. number of builtins) is known from the lambda, so we
can size it correctly right away.

Change-Id: Iab0b5a3f47d450fa9866c091ebbbed935b934907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6351
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 16:54:42 +00:00
Vincent Ambo
6c895d4b28 feat(tvix/eval): implement OpThunk for runtime thunk construction
Implements an operation very similar to `OpClosure` which populates a
thunk's upvalues and leaves it on the stack.

Change-Id: I753b4dfeeaae6919316c7028ec361aaa13d87646
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6350
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 16:54:42 +00:00
Vincent Ambo
6c4b9bdce9 refactor(tvix/eval): extract VM::populate_upvalues function
This function is reusable between thunks & closures.

Change-Id: I44d5f9897b087a385c8e75027d2ff39c48a096f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6349
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 16:36:46 +00:00
Vincent Ambo
ce498052e6 refactor(tvix/eval): Carry lambda & upvalues directly in CallFrame
CallFrame has to work for both thunks & closures (as a thunk is
basically a "weird 0-argument closure").

We opt to store the common, relevant fields directly in the frame to
avoid having to dereference through the nested structures
constantly (which would be especially annoying in the case of thunks).

Change-Id: I47781597b84ec5cd55502dba1713e92cf2592af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6348
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
25c62dd0ef refactor(tvix/eval): introduce UpvalueCarrier trait
This trait abstracts over the commonalities of upvalue handling
between closures and thunks.

It allows the VM to simplify the code used for setting up upvalues,
without duplicating between the two different types.

Note that this does not yet refactor the VM code to optimally make use
of this.

Change-Id: If8de5181f26ae1fa00d554f1ae6ea473ee4b6070
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6347
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
8033a7abae refactor(tvix/eval): simplify thunk representations
For now, do not distinguish between closing and non-closing thunks, it
will make the initial implementation easier. See Knuth etc.

Change-Id: I0bd51e0f89f2c77e90bac63b507e5027b649e3d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6346
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
28a9847c65 feat(tvix/eval): emit thunks for recursive local scope resolution
When resolving a value on the same level that is known but not yet
defined, emit a thunk.

Consider for example:

    let
      #       v--- requires a thunk
      a = 1 * b;
      b = 10;
    in a

Change-Id: I922cb50973ebe05e335a7bc7cb851960cf34733b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6345
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
d1798444be feat(tvix/eval): Add Compiler::thunk method for emitting thunks
The logic in this method is *very* similar to `compile_lambda`. It is
intended to be called around any expression that should be
thunked (such as function applications, attribute set values, etc.).

Change-Id: Idfbb2daa9f4b735095378fb9c39a2fd07c8cff91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6344
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
1f37275cfb feat(tvix/eval): introduce Value::Thunk variant
Introduces the representation of runtime thunks, that is lazily
evaluated values. Their representation is very similar to closures.

Change-Id: I24d1ab7947c070ae72ca6260a7bbe6198bc8c7c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6343
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
49296cebe3 refactor(tvix/eval): extract compiler's upvalue logic into helper
This exact same logic is reused for thunk creation.

Change-Id: I731db9cc659a1f2ca87db55d58d6ff632f417812
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6342
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
70b27a6080 test(tvix/eval): add a test case for nested, deferred upvalues
This test case was previously broken by the bug introduced by
confusing local and stack indexes.

Change-Id: Ibef299dad266c6105deac1da5dde112fe9f640b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6341
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
24811d7655 fix(tvix/eval): distinguish statically between StackIdx and LocalIdx
Previously the functions in the scope module returned usize values,
which - sometimes from the same function - were either indexes into
the runtime stack *or* indexes into the compiler's local stack.

This is extremely confusing because it requires the caller to be aware
of the difference, and it actually caused subtle bugs.

To avoid this, there is now a new LocalIdx wrapper type which is used
by the scope module to return indexes into the compiler's stack, as
well as helpers for accounting for the differences between these
indexes and the runtime indexes.

Change-Id: I58f0b50ad94b28a304e3372fd9731b6590b3fdb8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6340
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
68aad6d4cf feat(tvix/eval): emit OpFinalise when local scopes are complete
With this change, the runtime can correctly capture deferred upvalues.

Change-Id: I1e43b7b1ac2553b1812424adfc8bd08ef77bf1ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6339
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
5c4e102ac8 feat(tvix/eval): track whether locals needs to be finalised
When encountering a deferred local upvalue, the compiler will now mark
the corresponding local as needing a finaliser which makes it possible
to emit the OpFinalise instruction for this stack slot a little bit
down the line.

Change-Id: I3962066f10fc6c6e1472722b8bdb415a811e0740
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6338
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
025a9a4a0a feat(tvix/eval): implement OpFinalise instruction
This instruction finalises the initialisation of deferred upvalues in
closures (and soon, thunks).

The compiler does not yet emit this instruction, some more accounting
is needed for that.

Change-Id: Ic4181b26e19779e206f51e17388559400da5f93a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6337
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
1bfe32f412 feat(tvix/eval): set up deferred upvalues at runtime
This will leave a sentinel value in the upvalue slot in which the
actual value is to be captured after resolution once a scope is fully
set up.

Change-Id: I12b37b0dc8d32603b03e675c3bd039468e70b354
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6336
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 14:58:52 +00:00
Vincent Ambo
6c1948a71a feat(tvix/eval): detect deferred upvalue capturing
Uses the threaded through slot offset to determine whether
initialisation of a captured local upvalue must be defered to a later
point where all values of a scope are available.

This adds a new data representation to the opcode for this situation,
but the equivalent runtime handling is not yet implemented. This is in
part because there is more compiler machinery needed to find the
resolution point.

Change-Id: Ifd0c393f76abfe6e2d91483faf0f58947ab1dedc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6329
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-06 14:58:52 +00:00
Vincent Ambo
203d9f2e3e chore(tvix/eval): thread current slot index through compiler
While compiling local bindings, we now know the stack slot of the
value currently being compiled.

This will let us determine whether an upvalue can be captured directly
or whether it needs to wait for a synchronisation point at which the
upvalue can be instantiated.

This machinery lets us avoid unnecessary work at runtime when
instantiating closures that actually do not need complicated recursive
resolution.

This change itself introduces no new functionality, but since the
threading is noisy it is split out as a separate change.

Change-Id: I847c677ee8f6725fda1d2efd689b6a58bdccb779
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6328
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-06 14:58:52 +00:00
Vincent Ambo
9dd1fa875d refactor(tvix/eval): optimise initialisation of locals
Instead of looking up the local to be initialised by its name again,
we can simply track the index at which it was declared from the point
where the declaration was made.

This reduces some string cloning and removes unnecessary logic. It
also theoretically makes the *current* index available during locals
compilation, which can be used to optimise some recursion cases.

Change-Id: I06f403603d4f86c3d319debfe74b5a52eec00990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6327
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:45:43 +00:00
Vincent Ambo
900a92935d refactor(tvix/eval): declare all locals before compiling them
This actually makes things full-circle, as this tree already had this
implementation once before all the other required components were in
place.

With this commit, the compiler can resolve recursive upvalues within
the same scope (though they will not yet work at runtime).

Change-Id: I6267e477d08f367257c3a6dde054b880d7b47211
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6326
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:45:43 +00:00
Vincent Ambo
47b3562867 refactor(tvix/eval): decouple local depth & initialisation tracking
In order to resolve recursive references correctly, these two can not
be initialised the same way as a potentially large number of (nested!)
locals can be declared without initialising their depth.

This would lead to issues with detecting things like shadowed
variables, so making both bits explicit is preferable.

Change-Id: I100cdf1724faa4a2b5a0748429841cf8ef206252
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6325
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:45:43 +00:00
Vincent Ambo
705c1b9d53 fix(tvix/eval): correct runtime error for missing dynamic upvalue
Change-Id: I75c351619780fdc5186a54f3df9b244ada984069
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6324
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:45:43 +00:00
Vincent Ambo
776120de05 fix(tvix/eval): instantiate *new* closures from blueprints each time
The previous closure refactoring introduced a bug in which the same
closure object would get mutated constantly for each instance of a
closure, which is incorrect behaviour.

This commit instead introduces an explicit new Value variant for the
internal "blueprint" that the compiler generates (essentially just the
lambda) and uses this variant to construct the closure at runtime.

If the blueprint ever leaks out to a user somehow that is a critical
bug and tvix-eval will panic.

As a ~treat~ test for this, the fibonacci function is being used as it
is a self-recursive closure (i.e. different instantiations of the same
"blueprint") getting called with different values and it's good to
have it around.

Change-Id: I485de675e9bb0c599ed7d5dc0f001eb34ab4c15f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6323
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:45:43 +00:00
Vincent Ambo
ccfb971dc5 fix(tvix/eval): correctly thread through dynamic upvalues
This puts together the puzzle pieces for threading dynamic
upvalues (that is, upvalues resolved from the `with`-stack) all the
way through.

Reading the test case enclosed in this commit and walking through it
is recommended to understand what problem is being tackled here.

In short, because the compiler can not statically know *which*
with-scope a dynamic argument is resolved from it needs to lay the
groundwork for resolving from *all* possible scopes.

There are multiple different approaches to doing this. The approach
chosen in this commit is that if a dynamic upvalue is detected, the
compiler will emit instructions to close over this dynamic value
in *all* enclosing lambda contexts.

It uses a new instruction for this that will leave around a sentinel
value in case an identifier could not be resolved, and wire the
location of this found value (or sentinel) up through the upvalues to
the next level of nesting.

In this tradeoff, tvix potentially closes over more upvalues than are
needed (but in practice, how often do people create *really* deep
`with`-stacks? and in *this* kind of code situation? maybe we should
even warn for this!) but avoids keeping the entire attribute sets
themselves around.

Looking at the test case, each surrounding closure will close
over *all* dynamic identifiers that are referenced later on visible to
it, but only the last one for each identifier will actually end up
being used.

This also covers our bases for an additional edge-case this creates,
in which an identifier potentially resolves to a dynamic upvalue *and*
to a dynamic value within the function's own scope (again, would
anyone really do this?) by introducing a resolution instruction for
that particular case.

There is likely some potential for cleaning up this code which is
quite ugly in some parts, but as this implementation is now carefully
calibrated to work I decided it is time to commit it and clean it up
in subsequent commits.

Change-Id: Ib701e3e6da39bd2c95938d1384036ff4f9fb3749
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6322
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:45:43 +00:00
Vincent Ambo
8982e16e26 refactor(tvix/eval): thread dynamic upvalues through all contexts
With this change, encountering a dynamic upvalue will thread through
all contexts starting from the lowest context that has a non-empty
`with`-stack.

The additional upvalues are not actually used yet, so the effective
behaviour remains mostly the same. This is done in preparation for an
upcoming change, which will implement proper dynamic resolution for
complex cases of nested dynamic upvalues.

Yes, this whole upvalue + dynamic values thing is a little bit
mind-bending, but we would like to not give up being able to resolve a
large chunk of the scoping behaviour statically.

Change-Id: Ia58cdd47d79212390a6503ef13cef46b6b3e19a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6321
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:45:43 +00:00
Vincent Ambo
f6de4434c3 feat(tvix/eval): allow ignoring locals by prefixing with _
This is a common idiom in both Nix and other languages when a local is
declared without actually being used.

Since Tvix warns for unused locals, having this available is useful
and can be included in the final error message as a suggestion if an
unused variable is intentional.

Change-Id: Ia85f704ba183499a3bae657c58166e2e29f9bde5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6320
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:45:43 +00:00
Vincent Ambo
a5e22c532b fix(tvix/eval): correctly resolve dynamic upvalues one scope up
This does not yet correctly resolve them if they are more than one
scope up, however.

Change-Id: I6687073c60aee0282f2b6ffc98b34c1e96a60f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6319
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:29:25 +00:00
Vincent Ambo
e46a2ce3ef fix(tvix/eval): account for stack offset when pushing a with scope
Change-Id: I4b98eaea3ed5059c29938a117a9d59499a0bb95d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6318
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:29:25 +00:00
Vincent Ambo
ae531a2245 feat(tvix/eval): implement capture of self-recursive upvalues
With this change, it becomes possible for functions to call themselves
as they are being defined in local bindings.

Change-Id: Ib46a39ba17b1452b5673d96fa729d633d237241a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6314
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:29:25 +00:00
Vincent Ambo
3089e46eb1 refactor(tvix/eval): encapsulate internal mutability within Closure
This is required to efficiently construct the upvalue array at
runtime, as there are situations where during Closure construction
multiple things already have a reference to the closure (e.g. a
self-reference).

Change-Id: I35263b845fdc695dc873de489f5168d39b370f6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6312
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-06 07:29:25 +00:00
Vincent Ambo
932a70d0c2 refactor(tvix/eval): track with stack size as a simple integer
The `With` struct no longer contained any internals after the cleanup
logic for the stack had been moved into Compiler::compile_with,
leaving the `Vec<With>` to essentially act as a counter for the number
of things on the with stack.

That's inefficient of course, so with this commit it actually becomes
an integer (with an encapsulated API within scope::Scope).

Change-Id: I67a00987fc8b46b30d369a96d41e83c8af5b1998
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6311
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:53:08 +00:00
Vincent Ambo
2cd08d136e refactor(tvix/eval): move compiler's scope logic to separate module
The compiler module is getting quite long and this will help keep some
order.

Right now the scope internals are not very well encapsulated; this
paves a way to reducing the API surface of the `scope` type to the
things that are actually used by the compiler instead of giving access
to its internals.

Change-Id: I8c16c26d263f018baa263f395c9cd80715199241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6310
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:53:08 +00:00
Vincent Ambo
83aa5917b4 fix(tvix/eval): declare locals before marking them initialised
This has no effect yet, other than changing the way in which some
upvalue captures break (that are already not working correctly).

However, after this change the compiler correctly detects
self-recursion and can start emitting the instructions to deal with
this at runtime.

Change-Id: Id3b0ac206c0204739597a4325bcc66f9c806c242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6309
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:53:08 +00:00
Vincent Ambo
4bf096ee6e fix(tvix/eval): Account for uninitialised variables in with_idx
Calculating the with_idx (i.e. the stack offset of the "phantom"
variable from which a `with` dynamically reads at runtime) needs to
account for unitialised variables the same way as the resolution of
normal locals does.

Change-Id: I9ffe404535bf1c3cb5dfe8d9e005798c857fff94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6308
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:53:08 +00:00
Vincent Ambo
27bc8cb3d4 fix(tvix/eval): open/close additional scope around with
This is required to correctly clean up the `with` values.

At the moment, the attrset from which identifiers are being taken is
always pushed on the stack. This means that it must also be removed
again, otherwise in an expression like

  with { a = 15; }; a

The final stack is `[ { a = 15; } 15 ]` *after the last operation*,
which means that the attrset is still on there as garbage.

This has little practical impact right now because it is always
shadowed by the fact that the actual expression value is at the right
location, but becomes relevant when accounting for upvalue captures.

Change-Id: I69e9745bfaa4d6bbcb60ee71f4dc3f8d8695d16a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6303
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:40:10 +00:00
Vincent Ambo
abd8f12f5d refactor(tvix/eval): extend resolve_local logic for self-recursion
This extends the logic of `Scope::resolve_local` to detect cases where
self-recursion is occuring (i.e. an identifier is being accessed in
its own identifier).

These cases are not yet handled specially, and the logic of when
things are marked initialised (which was previously always at the same
spot as their declaration) has not changed, making this commit a
runtime no-op for now.

Change-Id: I3179642a7c55869ad4465fdd2678b0cd51a20f15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6302
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:40:10 +00:00
Vincent Ambo
a52db14820 feat(tvix/eval): detect illegally shadowed variables
Nix does not allow things like `let a = 1; a = 2; in a`, but doing it
across depths is allowed.

Change-Id: I6a259f8b01a254b433b58c736e245c9c764641b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6301
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:40:10 +00:00
Vincent Ambo
00aeb6dfaf refactor(tvix/eval): introduce Depth enum to track variable status
This does not yet change anything semantically, but will be useful for
resolving simple cases of self-recursion etc.

Change-Id: I139ecb7e4a8a81193774392a96e73e0ea6b9f85d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6300
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:40:10 +00:00
Vincent Ambo
33cde1422e feat(tvix/eval): implement upvalue resolution in with scopes
These need to be handled specially by the runtime if the compiler
determines that a given local must be resolved via `with`.

Note that this implementation has a bug: It currently allows `with`
inside of nested lambdas to shadow statically known identifiers. This
will be cleaned up in the next commit.

Change-Id: If196b99cbd1a0f2dbb4a40a0e88cdb09a009c6b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6299
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:40:10 +00:00
Vincent Ambo
10b0879c00 fix(tvix/eval): ensure OpResolveWith can be traced
The previous implementation of OpResolveWith manually controlled the
loop iteration, which skipped over the disassembler's tracing
instruction.

Instead, the resolution of dynamic variables has been delegated to a
new helper function. This has the additional benefit that the loop
labels are no longer required, making things a bit cleaner.

Change-Id: If22b74c3d49c74bf3a1ec4497cb761a9ee6cf2a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6298
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:40:10 +00:00
Vincent Ambo
7c2fdefcd8 fix(tvix/eval): pop with stack immediately after processing body
Instead of tying the popping of the with stack to scope depth, clean
up the stack immediately after processing a with body.

The previous behaviour was actually incorrect, as it would leave
things on the with-stack longer than they were supposed to be there.
This could lead to false positive resolutions in some situations
involving closures.

Change-Id: I7b0638557503f1f71eb602e3d5ff193cdfcb67cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6297
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:40:10 +00:00
Vincent Ambo
26acc2e636 test(tvix/eval): add tests for very simple closures
Change-Id: Ib8287ade4d5df6d29e1812fb2d349cee5d92ca6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6296
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-04 17:40:10 +00:00
Vincent Ambo
f88d248f48 feat(tvix/eval): implement runtime closure construction (OpClosure)
Implements the final bit of logic remaining for wiring up closures,
which is the runtime construction of closure objects.

When encountering an OpClosure, the VM walks through the bytecode
collecting all the upvalue location operands (see commit introducing
the OpCode::Data* variants for details) and stores the runtime values
in the new closures upvalue vector.

After that, the handling of the closure itself becomes functionally
identical to that of lambdas.

With this initial implementation of closures there are several large
optimisation potentials available, the two most notable ones are:

- Distinguish the runtime representation of lambdas and closures
  explicitly.

- Detect and handle multiple-arity functions directly in the compiler.

However, for both of these we should wait until we have appropriate
benchmarking infrastructure in place. This is because our test
implementations have shown that the complexity of either of these
changes is quite significant, and we do not yet know if they really
pay off.

Change-Id: I077e977810fd5cb2b1ecd7f1a119e728025dd786
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6295
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-03 21:55:04 +00:00
Vincent Ambo
0f06d0ca33 feat(tvix/eval): implement OpGetUpvalue in the VM
This resolves an upvalue at runtime by pushing it on the stack from
the closure's upvalue vector.

Change-Id: Ic3e7a7ecd9f7032f679114a1995e5bbf83062fcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6294
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-03 21:55:04 +00:00
Vincent Ambo
3d8888a13e refactor(tvix/eval): store Closure inside of the VM's call frames
In preparation for implementing calling of closures, store a closure
directly in the VMs call frame.

Change-Id: Iad24cd8c49fee4ebd4d0c84ffaa4c2505ee3dfd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6293
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-03 21:55:04 +00:00
Vincent Ambo
5ecf573dff feat(tvix/eval): add Value::to_closure
... same as the others

Change-Id: I9c8868388c10b0b6484c5bdd3799d801296c6979
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6292
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-03 21:55:04 +00:00
Vincent Ambo
69d8f17a26 feat(tvix/eval): compile creation of closure objects
Fully implements the instructions for compiling closure
objects (without runtime handling yet).

Closure (and thunk) objects are created at runtime by capturing all
known upvalues. To represent this, the instructions for creating them
need to have a variable number of arguments. Due to that, this commit
introduces new variants in OpCode that are not actually operations,
but data.

If the VM is implemented correctly, the instruction pointer should
never point at these. Due to this, the VM will panic if it sees a data
operand during an execution run.

Change-Id: Ic56b49b3a42736dc437751e76df0e89c8d0619c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6291
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-03 21:55:04 +00:00
Vincent Ambo
e4fadfaaf8 chore(tvix/eval): print node representation for compiler errors
Better for development flow.

Change-Id: I038efb39caca804f28a44fd4c83457e90abbcee4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6290
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-03 21:55:04 +00:00
Vincent Ambo
1163ef3e41 feat(tvix/eval): implement compilation of upvalue access
This adds a new upvalue tracking structure in the compiler to resolve
upvalues and track their positions within a function when compiling a
closure.

The compiler will emit runtime upvalue access instructions after this
commit, but the creation of the runtime closure object etc. is not yet
wired up.

Change-Id: Ib0c2c25f686bfd45f797c528753068858e3a770d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6289
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-03 21:55:04 +00:00
Vincent Ambo
2f93ed297e refactor(tvix/eval): add opcode::Count type for less ambiguity
Change-Id: Ibde0b2baa1128a74c1364ee9a6330b62db3da699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6288
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-03 13:27:40 +00:00
Vincent Ambo
c73e84d957 refactor(tvix/eval): add opcode::StackIdx type for less ambiguity
Change-Id: I9b9de1f681972c205d4d20bc5731d2ce79858edb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6287
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-03 13:22:42 +00:00
Vincent Ambo
2cdc6192b4 docs(tvix/eval): add docstrings for usize wrappers in opcode
Change-Id: I11b9324233c0aa48bd2fbac15a484962f925e72e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6283
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-03 13:22:42 +00:00