Commit graph

205 commits

Author SHA1 Message Date
Adam Joseph
c096152953 refactor(tvix/eval): rename Opcode::DataLocalIdx to DataStackIdx
It is very confusing that this opcode is called DataLocalIdx, but it
carries a StackIdx rather than a LocalIdx.  It seems like this
really ought to be called DataStackIdx, but maybe I've
misunderstood; if so please explain it to me.

Change-Id: I91f6ffa759412beef0b91d3c19ec0d873fe51b99
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7088
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-05 21:58:29 +00:00
Adam Joseph
f93f138c6c fix(tvix/eval): inline mis-named Local::above()
If self.depth > other.depth then self is deeper than other, so self
is *below* other, not above it.  Let's just inline the function.

Change-Id: I8dda3d90cbc86c8a6fa01bc4a5e506a2e403bd20
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7090
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-04 01:33:22 +00:00
Adam Joseph
4d83fd84b4 refactor(tvix/eval): search-and-replace changes
This commit contains two search-and-replace renames which are broken
out from I04131501029772f30e28da8281d864427685097f in order to
reduce the noise in that CL:

- `is_thunk -> is_suspended_thunk`, since there are now
  OpThunkClosure and OpThunkSuspended

- `compile_lambda_or_thunk` -> `compile_lambda_or_suspension`

Change-Id: I7cc5bbb75ef6605e3428c7be27e812f41a10c127
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7037
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-27 22:00:39 +00:00
Adam Joseph
8616f13a71 feat(tvix/eval): builtins.import without RefCell
CL/6867 added support for builtins.import, which required a cyclic
reference import->globals->builtins->import.  This was implemented
using a RefCell, which makes it possible to mutate the builtins
during evaluation.  The commit message for CL/6867 expressed a
desire to eliminate this possibility:

  This opens up a potentially dangerous footgun in which we could
  mutate the builtins at runtime leading to different compiler
  invocations seeing different builtins, so it'd be nice to have
  some kind of "finalised" status for them or some such, but I'm not
  sure how to represent that atm.

This CL replaces the RefCell with Rc::new_cyclic(), making the
globals/builtins immutable once again.  At VM runtime (once opcodes
start executing) everything is the same as before this CL, except
that the Rc<RefCell<>> introduced by CL/6867 is turned into an
rc::Weak<>.

The function passed to Rc::new_cyclic works very similarly to
overlays in nixpkgs: a function takes its own result as an argument.
However instead of laziness "breaking the cycle", Rust's
Rc::new_cyclic() instead uses an rc::Weak.  This is done to prevent
memory leaks rather than divergence.

This CL also resolves the following TODO from CL/6867:

  // TODO: encapsulate this import weirdness in builtins

The main disadvantage of this CL is the fact that the VM now must
ensure that it holds a strong reference to the globals while a
program is executing; failure to do so will cause a panic when the
weak reference in the builtins is upgrade()d.

In theory it should be possible to create strong reference cycles
the same way Rc::new_cyclic() creates weak cycles, but these cycles
would cause a permanent memory leak -- without either an rc::Weak or
RefCell there is no way to break the cycle.  At some point we will
have to implement some form of cycle collection; whatever library we
choose for that purpose is likely to provide an "immutable strong
reference cycle" primitive similar to Rc::new_cyclic(), and we
should be able to simply drop it in.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I34bb5821628eb97e426bdb880b02e2097402adb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7097
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-27 21:36:01 +00:00
Adam Joseph
4384418877 docs(tvix/eval): StackIdx, LocalIdx UpvalueIdx
This adds a comment noting that StackIdx is an offset relative to
the base of the current CallFrame, whereas UpvalueIdx is an absolute
index into the upvalues array.

It also removes the confusing mention of StackIdx in the descriptive
comment for LocalIdx.  They index into totally different structures;
one exists at runtime and the other exists at compile time.

Change-Id: Ib932b1b0679734c15001e8c5c95a08293fa016b4
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7017
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-26 14:27:37 +00:00
Vincent Ambo
d4569cb504 feat(tvix/eval): initial attempt at setting lambda names
When compiling a lambda, take the name of the outer slot (if
available) and store it as the name on the lambda.

These names are then shown in the observer, and nowhere else (so far).

It is of course common for these things to thread through many
different context levels (e.g. `f = a: b: c: ...`), in this setup only
the outermost closure or thunk gains the name, but it's better than
nothing.

Change-Id: I681ba74e624f2b9e7a147144a27acf364fe6ccc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7065
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-23 15:58:53 +00:00
Vincent Ambo
5612ec0f41 fix(tvix/eval): thunk let-expression
There are some rare scope cases with deferred access where this
doesn't behave correctly otherwise.

Change-Id: I6c774f5e62c1cb50b598026c54727017a52cd22d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7064
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-10-23 15:50:35 +00:00
Vincent Ambo
5b305fab40 fix(tvix/eval): fix condition for useless inherit warning
The warning needs to consider whether it is occuring inside of a
thunk, i.e. the dynamic ancestry chain of lambda contexts must be
inspected and not just the current scope.

Change-Id: I5cf5482d67a8bbb9f03b0ecee7a62f58754f8e59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7063
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
9b52e5b9c2 refactor(tvix/eval): simplify check for deferring upvalue resolution
This check is now actually simply equivalent to checking whether the
target has been initialised or not.

Change-Id: I30660d11073ba313358f3a64234a90ed81abf74c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7062
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
Vincent Ambo
546fcf51cd refactor(tvix/eval): simplify self-reference check
Checking the computed depth and stack slot against the computed depth
and stack slot is equivalent to just checking the indices into the
locals vector against each other (i.e. "is the slot we're compiling
into the slot we're accessing?")

Change-Id: Ie85a68df073e3b2e3d9aba7fe8634c48eada81fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7059
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-23 15:50:35 +00:00
sterni
64bb501de1 fix(tvix): distinguish search- and relative path resolution errors
Failures to resolve a nix search path lookup in angle brackets can be
caught using tryEval (if it reaches the runtime). Resolving relative
paths (either to the current directory or the current user's home) can
never be caught, even if they happen inside a thunk at runtime (which is
currently the case for home-relative paths).

Change-Id: I7f73221df66d82a381dd4063358906257826995a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7025
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-21 00:11:29 +00:00
Adam Joseph
d978b556e6 feat(tvix/eval): deduplicate overlap between Closure and Thunk
This commit deduplicates the Thunk-like functionality from Closure
and unifies it with Thunk.

Specifically, we now have one and only one way of breaking reference
cycles in the Value-graph: Thunk.  No other variant contains a
RefCell.  This should make it easier to reason about the behavior of
the VM.  InnerClosure and UpvaluesCarrier are no longer necessary.

This refactoring allowed an improvement in code generation:
`Rc<RefCell<>>`s are now created only for closures which do not have
self-references or deferred upvalues, instead of for all closures.
OpClosure has been split into two separate opcodes:

- OpClosure creates non-recursive closures with no deferred
  upvalues.  The VM will not create an `Rc<RefCell<>>` when executing
  this instruction.

- OpThunkClosure is used for closures with self-references or
  deferred upvalues.  The VM will create a Thunk when executing this
  opcode, but the Thunk will start out already in the
  `ThunkRepr::Evaluated` state, rather than in the
  `ThunkRepr::Suspeneded` state.

To avoid confusion, OpThunk has been renamed OpThunkSuspended.

Thanks to @sterni for suggesting that all this could be done without
adding an additional variant to ThunkRepr.  This does however mean
that there will be mutating accesses to `ThunkRepr::Evaluated`,
which was not previously the case.  The field `is_finalised:bool`
has been added to `Closure` to ensure that these mutating accesses
are performed only on finalised Closures.  Both the check and the
field are present only if `#[cfg(debug_assertions)]`.

Change-Id: I04131501029772f30e28da8281d864427685097f
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7019
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-19 10:38:54 +00:00
Vincent Ambo
13a5e7dd5b fix(tvix/eval): wrap dynamic resolution in an extra thunk
Without this change it was possible to cause situations (see the new
test) in which a `with`-namespace was forced prematurely.

Change-Id: I879ea7763b43edc693feace2c73c890d426fafd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7031
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-10-18 09:15:13 +00:00
Griffin Smith
2a3d498104 feat(tvix/eval): Validate closed formals
Validate "closed formals" (formal parameters without an ellipsis) via a
new ValidateClosedFormals op, which checks the arguments (in an attr set
at the top of the stack) against the formal parameters on the Lambda in
the current frame, and returns a new UnexpectedArgument error (including
the span of the formals themselves!!) if any arguments aren't allowed

Change-Id: Idcc47a59167a83be1832a6229f137d84e426c56c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7002
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-17 11:29:49 +00:00
Griffin Smith
e63d14419f feat(tvix/eval): Record formals on lambda
In preparation for both implementing the `functionArgs` builtin and
adding support for validating closed formals, record information about
the formal arguments to a function *on the Lambda itself*. This may seem
a little odd for the purposes of just closed formal checking, but is
something we have to have anyway for builtins.functionArgs so I figured
I'd do it this way to kill both birds with one stone.

Change-Id: Ie3770a607bf352a1eb395c79ca29bb25d5978cd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-17 11:29:49 +00:00
Adam Joseph
f05a1d27d9 refactor(tvix/eval): unify compile_lambda() with thunk()
This resolves a TODO.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: If4d2124648ac88094e547e1ad7f1b446feb26182
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7010
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 23:42:30 +00:00
sterni
4dcb8f38c2 fix(tvix/eval): resolve home relative paths at runtime
Home relative paths depend on the environment to be resolved. We have
elected to do everything that depends on the environment, e.g. resolving
SPATH expressions using NIX_PATH, at runtime, so tvix evaluation would
continue to behave correctly even if we separated the compilation and
execution phases more, e.g. via serializing bytecode. Then the value of
HOME, NIX_PATH etc. could reasonably change in the time until execution,
yielding wrong results if the resolution results were cached in the
bytecode.

We also take the opportunity to fix the broken path concatenation
previously found in the compiler, fixing b/205.

Another thing we could consider is emitting a warning for home relative
path literals, as they are by nature relatively fragile.

One sideeffect of this change is that home path resolution errors
become catchable which is not the case in C++ Nix. This will need to be
fixed up in a subsequent change.

Change-Id: I30bd69b575667c49170a9fdea23a020565d0f9ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7024
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2022-10-16 19:11:23 +00:00
sterni
0624d78af0 refactor(tvix/eval): make OpFindFile use internal UnresolvedPath
To assert that OpFindFile is only emitted for specially compiled SPATH
expressions, as well as make sure it doesn't accidentally operate on
“ordinary values”, introduce an UnresolvedPath internal value. If
OpFindFile sees a non-UnresolvedPath value, it'll crash.

Note that this change is not done purely for OpFindFile: We may want to
compile SPATH expressions as function calls to __findFile (like C++ Nix
does) in the future, so the UnresolvedPath value would definitely need
to be an ordinary string again then. Rather, this change is done in
preparation for resolving home dir relative paths at runtime (since they
depend on the environment) for which we'll need a similar mechanism to
OpFindFile.

Change-Id: I6acf287f35197cd9e13377079f972b9d36e5b22e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7023
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 19:11:23 +00:00
Vincent Ambo
0f24120a6c fix(tvix/eval): fix Compiler::new on wasm
This path normalisation business causes runtime panics on WebAssembly
because those operations are unsupported.

Maybe this shouldn't be happening in the compiler anyways, not sure,
but for now this commit adds a workaround based on the target to
disable the normalisation if we're compiling for wasm.

Change-Id: I908a84fbdffc3401f8d443e2c73ec673e9f397ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7004
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-13 16:29:49 +00:00
Adam Joseph
78d3d9150b fix(tvix/eval): src/compiler: ensure root_dir is absolute
Cppnix immediately absolutizes pathnames at parse time; if you write
`./foo`, it is immediately converted to `$(pwd)/foo` and manipulated
as an absolute path at all times.

To avoid having to introduce filesystem access operations in the
implementation of otherwise-pure builtins, let's guarantee that the
`root_dir` of the VM is always an absolute path.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I7cbbae2cba4b2716ff3f5ff7c9ce0ad529358c8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6995
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-13 09:48:32 +00:00
Adam Joseph
25336fc47b refactor(tvix/eval): factor out all calls to canon_path
Right now we're pretending that the Rust library path_clean does the
same thing that cppnix's canonPath() does.  This is not true.  It's
close enough for the test suite, but may come back to bite us.

Let's create our own canon_path() function and call that in all the
places where we intend to match the behavior of cppnix's
canonPath().  That way when we fix this we can fix it once, in one
place.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ia6f9577f62f49ef352ff9cfa5efdf37c32d31b11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6993
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-13 09:07:47 +00:00
Griffin Smith
06ec4bebe7 fix(tvix/eval): Actually trace spans for thunks
Currently, the span on *all* thunk force errors is the span at which the
thunk is forced, which for recursive thunk forcing ends up just being
the same span over and over again. This changes the span on thunk force
errors to be the span at which point the thunk is *created*, which is a
bit more helpful (though the printing atm is a little... crowded). To
make this work, we have to thread through the span at which a thunk is
created into a field on the thunk itself.

Change-Id: I81474810a763046e2eb3a8f07acf7d8ec708824a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6932
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-10 23:51:09 +00:00
Griffin Smith
2592113435 fix(tvix/eval): Thunk if expr
Since the body of an `if` expr can refer to deferred upvalues, it needs
to be thunked so when we actually compile those deferred upvalues we
have something for the finalize op to point at. Without this all sorts
of weird things can happen due to the finalize op being run in the wrong
lambda context, up to and including a panic.

Change-Id: I040d5e1a7232fd841cfa4953539898fa49cbbb83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6929
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-10 20:35:11 +00:00
Griffin Smith
273ba73754 feat(tvix/eval): Initial resolution of <...> paths
This commit implements (lazy) resolution of `<...>` paths via either the
NIX_PATH environment variable, or the -I command-line flag - both
handled via EvalOptions. As a result, EvalOptions can no longer derive
Copy, meaning we have to clone it at each line of the repl - this is
probably not a huge deal as repl performance is not exactly an inner
loop and we're not cloning very much.

Internally, this works by creating a thunk which pushes a constant
containing the string inside the brackets to the stack, then a new
opcode to resolve that path via the `NixPath`. To get that opcode to
work, we now have to pass in the NixPath when constructing the VM.

This (intentionally) leaves out proper implementation of path resolution
via `findFile` (cppnix just calls whatever identifier called findFile is
in scope!!!) as that's widely considered a bit of a misfeature, but if
we do decide to implement that down the road it likely wouldn't be more
than a few extra ops within the thunk introduced here.

Change-Id: Ibc979b7e425b65cbe88599940520239a4a10cee2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6918
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-10 20:23:41 +00:00
Griffin Smith
899fbdbddb refactor(tvix/eval): Compile OpAssert using conditional jumps
In order to behave nicely with tryEval, asserts need to leave the
instruction pointer in a reasonable place even if they fail - whereas
with the previous implementation catching a failed assert would still
end up running the op for the *body* of the assert. With this change, we
compile asserts much more like an `if` expression with conditional jumps
rather than having an OpAssert op.

Change-Id: I1b266c3be90185c84000da6b1995ac3e6fd5471b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6925
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-10 17:51:22 +00:00
Vincent Ambo
207f3dd47e fix(tvix/eval): end scope after compiling legacy let bindings
When contrasting the compilation of the desugared version to the
"sugared" version, this was the noticeable difference.

This fixes b/203.

Change-Id: Iae02ffc56e06de1de091b84cdc59d8fe83a17d69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6898
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-08 19:10:09 +00:00
Vincent Ambo
50baf0bcfc refactor(tvix/eval): move spans module to crate root
This is also useful for error-handling related logic, outside of just
the compiler module.

Change-Id: I5c386e2b4c31cda0a0209b31136ca07f00e39e45
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6869
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-08 10:58:42 +00:00
Vincent Ambo
4b9178fa2a feat(tvix/eval): insert import into the builtins itself
Adding `import` to builtins causes causes a bootstrap cycle because
the `import` builtin needs to be initialised with the set of globals
before being inserted into the globals, which also must contain
itself.

To break out of the cycle this hack wraps the builtins passed to the
compiler in an `Rc` (probably sensible anyways, as they will end up
getting cloned a bunch), containing a RefCell which gives us mutable
access to the builtins.

This opens up a potentially dangerous footgun in which we could mutate
the builtins at runtime leading to different compiler invocations
seeing different builtins, so it'd be nice to have some kind of
"finalised" status for them or some such, but I'm not sure how to
represent that atm.

Change-Id: I25f8d4d2a7e8472d401c8ba2f4bbf9d86ab2abcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6867
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-07 14:24:36 +00:00
Vincent Ambo
2ff764ceb7 refactor(tvix/eval): remove unnecessary clones in compiler
There's basically nothing that needs *ownership* of an AST
node (which is just a little box full of references to other things
anyways), so we can thread this through as references all the way.

Change-Id: I35a1348a50c0e8e07d51dfc18847829379166fbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6853
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-04 21:27:30 +00:00
Vincent Ambo
b69b50feb1 refactor(tvix/eval): split observer traits in two
There are actually two different types of observers, the ones that
observe the compiler (and emitted chunks from different kinds of
expressions), and the ones that trace runtime execution.

Use of the NoOpObserver is unchanged, it simply implements both
traits.

Change-Id: I4277b82674c259ec55238a0de3bb1cdf5e21a258
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6852
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-04 21:27:30 +00:00
Vincent Ambo
6c5e83f8bc chore(tvix/eval): remove unused field in TrackedBindings
Change-Id: I65e31e9173e4f5bba19cc4e3d45eb4f8bf91b424
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6808
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-30 12:31:14 +00:00
Vincent Ambo
da1e3e9ac5 feat(tvix/eval): implement nested keys
This finishes up the implementation of nested keys after the key
insight that the nesting level does not need to be tracked, and
instead the attribute iterator can simply be retained inside the
structures as is (in an advanced state).

With this implementation, when encountering a nested key, the Tvix
compiler will first analyse whether there is already a matching
binding that can be merged (i.e. a binding that is a literal attribute
set), and perform the merge, or otherwise create a new recursive set
of bindings in which the entry is inserted with the path iterator
advanced beyond the first name component.

With this, all the logic simply applies recursively until there are no
more nested bindings (i.e. until all iterators are "empty").

Note that this has one (potentially insignificant) deviation from Nix
currently: If a non-mergable value is supplied (e.g. `a.b = 1; a =
2;`), Tvix will emit a *runtime* error (whereas it is *parse* time in
Nix) as the branch which could statically analyse this is currently
unreachable. There's a TODO for this, so we can fix it up later.

Change-Id: I53df70e09614ff4281a70b80eac7da3beca12da9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6806
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-30 12:31:14 +00:00
Vincent Ambo
9cd5f03835 refactor(tvix/eval): split out AttributeSet::from_ast helper
Change-Id: Id43dbd06aef14cf01b4901d9b3668d790cd2b5ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6805
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-30 07:07:17 +00:00
Vincent Ambo
9e9dde0106 chore(tvix/eval): remove nesting_level tracking
This is actually quite useless, as we can just pass
`AstChildren<ast::Attr>` around after partially consuming it.

Change-Id: If0aefa2b53fc801fced1ae0709bff93966bf19f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6804
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-30 07:07:17 +00:00
Vincent Ambo
ccf9dd651b refactor(tvix/eval): clean up representation flip in bindings
When encountering a nested binding for the first time, cleanly flip
the representation to `Binding::Set` in `Binding::merge` before
proceeding with the actual merge.

This reduces the number of points where we have to deal with the (soon
to be slightly more complex) construction of the nested binding
representation.

Change-Id: Ifd43aac7b59ebd15a72c3ec512386a5bcf26ec13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6802
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 17:46:02 +00:00
Vincent Ambo
18fcf0d79d feat(tvix/eval): (partially) track nesting level of attrsets
This adds the scaffolding required for tracking the nesting level (and
appropriately skipping the correct amount of attrpath entries when
inserting nested sets).

In order for all of this to work correctly, we can no longer track
`AttrpathValue` directly in the entries vector as rnix does not allow
us to construct values of that type - so instead we have to track its
inner components.

Change-Id: Icb18e105586bf6c247c2e66c302cde5609ad9789
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6801
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 17:46:02 +00:00
Vincent Ambo
cece9eae4a feat(tvix/eval): merge attribute sets in bindings
This is a significant step towards correctly implemented nested
bindings. All attribute sets defined within the same binding scope
will now be merged as in Nix, if they use the same key.

Change-Id: I13e056693d5e73192280043c6dd93b47d1306ed6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6800
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 17:46:02 +00:00
Vincent Ambo
0cee44838c feat(tvix/eval): add error kind for unmergeable nested attributes
Change-Id: Ic5e6d1bf2625c33938360affb0d1a7c922af11bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6799
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 17:46:02 +00:00
Vincent Ambo
3f34af205f feat(tvix/eval): add scaffolding for merging nested attribute sets
This sets up the required logic for finding and merging attribute sets
into nested bindings if they exist. This is absolutely not complete
yet and can, at this commit, probably cause undefined runtime
behaviour if nested attributes are specified.

The basic idea is that a new helper function on the `TrackedBindings`
struct is called with each encountered attribute and determines
whether the new entry can be merged into an existing attribute or not.

Right now the only effect this has in practice is that a new error
becomes available if somebody attempts to cause a merge into an
inherited key.

Change-Id: Id010df3605055eb1ad7fa65241055889dd21bab0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6798
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 17:46:02 +00:00
Vincent Ambo
09a57e7857 refactor(tvix/eval): emit OpAttrs inside of compile_bindings
This needs to move here so that we can reuse compile_bindings for the
nested attribute sets we're about to start constructing.

Change-Id: Ie83f52f7e1d128886e96a1da47792211fa826f21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6796
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-29 11:47:47 +00:00
Vincent Ambo
82df0b432a refactor(tvix/eval): introduce TrackedBindings struct
This struct will be the key to correctly compiling nested bindings, by
having insertions flow through some logic that will attempt to bind
attribute-set-like things when encountering them.

Change-Id: I8b5b20798de60688f3b6dc4526a460ebb2079f6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6795
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-29 11:47:47 +00:00
Vincent Ambo
e96f94ac88 refactor(tvix/eval): compile_recursive_scope -> compile_bindings
Change-Id: Iff18d0f84ba2b7a4194797e6c52c55b1c37e419c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6794
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-29 11:47:47 +00:00
Vincent Ambo
e31f8f735f chore(tvix/eval): fix all current clippy lints
Change-Id: I28d6af8cb408f8427a75d30b9120aaa809a1ea40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6784
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-29 11:47:47 +00:00
Vincent Ambo
3f21606278 refactor(tvix/eval): merge all bindings creation logic
As of this commit, all three types of bindings scopes are compiled the
same way (i.e. compilation of non-recursive attribute sets has been
switched over to the new code paths).

This sets us up for doing the final implementation of nested attribute
sets.

HOWEVER, this breaks the existing implementation of nested attributes
in non-recursive attribute sets. That implementation is flawed and
unworkable in practice, so we need to do this dance to be able to
implement it correctly.

Change-Id: Iba2545c0d1d6b51f5e1a31a5d005b8d01da546d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6782
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-29 11:47:47 +00:00
Vincent Ambo
949897651e feat(tvix/eval): implement dynamic keys in recursive attrs
This wires up the new bindings setup logic to be able to thread
through & compile dynamic attributes in recursive attrs.

It seems like we don't actually need to retain the phasing of Nix
exactly, as we can use the phantom mechanism to declare all locals
without making the dynamic ones accessible.

Change-Id: Ic2d43dd8fd97d7ccd56d8c6adf2ff97274cd837a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6781
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-09-29 11:47:47 +00:00
Vincent Ambo
3d9eb14e7a feat(tvix/eval): add KeySlot::Dynamic variant for dynamic keys
Another slice of the salami, but no functionality changes yet (other
than opening a code path that can reach a `todo!()`, but this will be
removed soon).

Change-Id: I56b4ed323f70754ed1ab27964ee3c99cf3bf3292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6780
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 00:48:33 +00:00
Vincent Ambo
d0636f1e24 refactor(tvix/eval): clean up unused attrpath normalisation logic
The previous way of sanitising dynamic keys is going away as we're
slowly introducing the new nested key logic.

While touching this stuff, I've also changed all the related string
types to SmolStr as that is more sensible for identifiers.

Change-Id: If30c74151508719d646d0e68e7d6f62c36f4d23f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6779
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 00:48:33 +00:00
Vincent Ambo
e253e5239d chore(tvix/eval): reflow comments in compiler::bindings
Change-Id: I6d74f71ecd671feaec96ee4ff39f218907c517fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6777
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 00:48:33 +00:00
Vincent Ambo
001e0520dc refactor(tvix/eval): merge inherits logic between all binding kinds
Removes the `compile_inherit_attrs` logic which was only used for
BindingsKind::Attrs (i.e. non-recursive attrs).

This brings us a step closer to fully merging all the binding logic
into one block that can dispatch based on the kind of bindings (and
thus giving us a good point to introduce the final logic for nested
bindings).

Change-Id: If48d7a9497fc084a5cc03a130c2a7da5e2b8ef0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6776
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 00:48:33 +00:00
Vincent Ambo
1a4486c92d refactor(tvix/eval): Factor out declare_bindings helper
This helper is responsible for declaring the bindings in the
compiler's scope tracking structures.

It is almost equivalent to the previous logic, but also accounts for
`BindingsKind::Attrs` - though those code paths do not end up here
yet.

Change-Id: I44f617b99b10f2a7b9675f7b23e2c803a4a93d29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6775
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-29 00:48:33 +00:00