Commit graph

21 commits

Author SHA1 Message Date
Adam Joseph
8eb32fb2d7 feat(tvix/eval): crude caching builtins.import
Before this, tvix was spending most of its time furiously re-parsing
and re-compiling nixpkgs, each time hoping to get a different result...

Change-Id: I1c0cfbf9af622c276275b1f2fb8d4e976f1b5533
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7361
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-02 13:32:11 +00:00
Griffin Smith
76d7671c8a feat(tvix/eval): Add docstrings as documentation for builtins
Add a new `documentation: Option<&'static str>` field to Builtin, and
populate it in the `#[builtins]` macro with the docstring of the builtin
function, if any.

Change-Id: Ic68fdf9b314d15a780731974234e2ae43f6a44b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7205
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-08 13:42:37 +00:00
Griffin Smith
a1015ba1d7 feat(tvix/eval): Give names to builtin arguments
Refactor the arguments of a Builtin to be a vec of a new BuiltinArgument
struct, which contains the old strictness boolean and also a static
`name` str - this is automatically determined via the ident for the
corresponding function argument in the proc-macro case, and passed in in
the cases where we're still manually calling Builtin::new.

Currently this name is unused, but in the future this can be used as
part of a documentation system for builtins.

Change-Id: Ib9dadb15b69bf8c9ea1983a4f4f197294a2394a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7204
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-08 13:42:37 +00:00
Griffin Smith
a4c9cc8d5e refactor(tvix/eval): Define impure builtins using the macro
Similar to what we did with pure builtins, define the impure builtins
within a module at the top-level using the new #[builtins] attribute
macro

Change-Id: Ie5d5135d00bb65e651531df6eadba642cd4eb08e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7202
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-08 13:42:37 +00:00
Adam Joseph
8616f13a71 feat(tvix/eval): builtins.import without RefCell
CL/6867 added support for builtins.import, which required a cyclic
reference import->globals->builtins->import.  This was implemented
using a RefCell, which makes it possible to mutate the builtins
during evaluation.  The commit message for CL/6867 expressed a
desire to eliminate this possibility:

  This opens up a potentially dangerous footgun in which we could
  mutate the builtins at runtime leading to different compiler
  invocations seeing different builtins, so it'd be nice to have
  some kind of "finalised" status for them or some such, but I'm not
  sure how to represent that atm.

This CL replaces the RefCell with Rc::new_cyclic(), making the
globals/builtins immutable once again.  At VM runtime (once opcodes
start executing) everything is the same as before this CL, except
that the Rc<RefCell<>> introduced by CL/6867 is turned into an
rc::Weak<>.

The function passed to Rc::new_cyclic works very similarly to
overlays in nixpkgs: a function takes its own result as an argument.
However instead of laziness "breaking the cycle", Rust's
Rc::new_cyclic() instead uses an rc::Weak.  This is done to prevent
memory leaks rather than divergence.

This CL also resolves the following TODO from CL/6867:

  // TODO: encapsulate this import weirdness in builtins

The main disadvantage of this CL is the fact that the VM now must
ensure that it holds a strong reference to the globals while a
program is executing; failure to do so will cause a panic when the
weak reference in the builtins is upgrade()d.

In theory it should be possible to create strong reference cycles
the same way Rc::new_cyclic() creates weak cycles, but these cycles
would cause a permanent memory leak -- without either an rc::Weak or
RefCell there is no way to break the cycle.  At some point we will
have to implement some form of cycle collection; whatever library we
choose for that purpose is likely to provide an "immutable strong
reference cycle" primitive similar to Rc::new_cyclic(), and we
should be able to simply drop it in.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I34bb5821628eb97e426bdb880b02e2097402adb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7097
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-27 21:36:01 +00:00
Adam Joseph
d978b556e6 feat(tvix/eval): deduplicate overlap between Closure and Thunk
This commit deduplicates the Thunk-like functionality from Closure
and unifies it with Thunk.

Specifically, we now have one and only one way of breaking reference
cycles in the Value-graph: Thunk.  No other variant contains a
RefCell.  This should make it easier to reason about the behavior of
the VM.  InnerClosure and UpvaluesCarrier are no longer necessary.

This refactoring allowed an improvement in code generation:
`Rc<RefCell<>>`s are now created only for closures which do not have
self-references or deferred upvalues, instead of for all closures.
OpClosure has been split into two separate opcodes:

- OpClosure creates non-recursive closures with no deferred
  upvalues.  The VM will not create an `Rc<RefCell<>>` when executing
  this instruction.

- OpThunkClosure is used for closures with self-references or
  deferred upvalues.  The VM will create a Thunk when executing this
  opcode, but the Thunk will start out already in the
  `ThunkRepr::Evaluated` state, rather than in the
  `ThunkRepr::Suspeneded` state.

To avoid confusion, OpThunk has been renamed OpThunkSuspended.

Thanks to @sterni for suggesting that all this could be done without
adding an additional variant to ThunkRepr.  This does however mean
that there will be mutating accesses to `ThunkRepr::Evaluated`,
which was not previously the case.  The field `is_finalised:bool`
has been added to `Closure` to ensure that these mutating accesses
are performed only on finalised Closures.  Both the check and the
field are present only if `#[cfg(debug_assertions)]`.

Change-Id: I04131501029772f30e28da8281d864427685097f
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7019
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-19 10:38:54 +00:00
sterni
db70c672cf fix(tvix/eval): don't coerce variable name to string
Change-Id: I8aa878dee009901feb453c489ce37c12fa3a31a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7026
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-16 09:20:02 +00:00
Griffin Smith
06ec4bebe7 fix(tvix/eval): Actually trace spans for thunks
Currently, the span on *all* thunk force errors is the span at which the
thunk is forced, which for recursive thunk forcing ends up just being
the same span over and over again. This changes the span on thunk force
errors to be the span at which point the thunk is *created*, which is a
bit more helpful (though the printing atm is a little... crowded). To
make this work, we have to thread through the span at which a thunk is
created into a field on the thunk itself.

Change-Id: I81474810a763046e2eb3a8f07acf7d8ec708824a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6932
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-10 23:51:09 +00:00
Griffin Smith
90ec632fd1 feat(tvix/eval): Implement builtins.getEnv
Change-Id: I09f512a60989a37184e73e521d4a3aa23f33a1a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6922
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2022-10-10 23:38:25 +00:00
Griffin Smith
9669fa8ff1 feat(tvix/eval): Implement builtins.readFile
Change-Id: If3fd0b087009a2bfbad8bb7aca0aa20de906eb12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6921
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-10 23:38:25 +00:00
Griffin Smith
0b8a967aca feat(tvix/eval): Implement builtins.pathExists
Change-Id: Ife8a690e9036868964771893ab29a9ae3a2d2365
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6919
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-10 20:23:41 +00:00
William Carroll
41ddc37725 feat(tvix/eval): Support builtins.readDir
Co-authored-by: Griffin Smith <root@gws.fyi>
Change-Id: I5ff19efbe87d8f571f22ab0480500505afa624c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6552
Autosubmit: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-10 19:36:49 +00:00
Griffin Smith
b4d978cd6a feat(tvix/tests): Import default.nix inside directory
This requires actually passing the source directory into `interpret` in
the eval tests, but otherwise this is fairly straightforward - if we're
trying to import a directory, just push `default.nix` onto it and import
that instead.

Change-Id: I0b7d4234f81977e78d14dfa651bf0cf9721017e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6893
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-08 19:02:25 +00:00
Vincent Ambo
1e2d323a7c feat(tvix/eval): fancy-format parse errors returned by rnix
This change is quite verbose, so a little bit of explaining:

1. To correctly format parse errors, errors must be able to return
   more than one annotated span (the parser returns a list of errors
   for each span).

   To accomplish this, the structure of how the `Diagnostic` struct
   which formats an error is constructed has changed to delegate the
   creation of the `SpanLabel` vector to the kind of error.

2. The rnix structures don't have human-readable output formats by
   default, so some verbose methods for formatting them in
   human-readable ways have been added in the errors module. We might
   want to move these out into a submodule.

3. In many cases, the errors returned by rnix are a bit strange - so
   while we format them with all information that is easily available
   they may look weird or not necessarily help users. Consider this CL
   only a first step in the right direction.

Change-Id: Ie7dd74751af9e7ecb35d751f8b087aae5ae6e2e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6871
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-08 17:27:57 +00:00
Vincent Ambo
2478a6c5ba feat(tvix/eval): coerce values to paths when importing
This enables the use of string paths (and, in the future,
derivations), as long as their string values represent an absolute
path.

Change-Id: I4b198efeb70415ed52f58bd1da6fa79a24dad14c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6866
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-07 14:24:36 +00:00
Vincent Ambo
07e03498f2 feat(tvix/eval): add method for emitting runtime warnings
This lets the VM emit warnings when it encounters situations that
should only be warned about at runtime.

For starters, this is used to pass through compilation warnings that
come up when `import` is used.

Change-Id: I0c4bc8c534d699999887c430d93629fadfa662c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6868
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-07 14:24:36 +00:00
Vincent Ambo
4b9178fa2a feat(tvix/eval): insert import into the builtins itself
Adding `import` to builtins causes causes a bootstrap cycle because
the `import` builtin needs to be initialised with the set of globals
before being inserted into the globals, which also must contain
itself.

To break out of the cycle this hack wraps the builtins passed to the
compiler in an `Rc` (probably sensible anyways, as they will end up
getting cloned a bunch), containing a RefCell which gives us mutable
access to the builtins.

This opens up a potentially dangerous footgun in which we could mutate
the builtins at runtime leading to different compiler invocations
seeing different builtins, so it'd be nice to have some kind of
"finalised" status for them or some such, but I'm not sure how to
represent that atm.

Change-Id: I25f8d4d2a7e8472d401c8ba2f4bbf9d86ab2abcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6867
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-07 14:24:36 +00:00
Vincent Ambo
b530e496a5 feat(tvix/eval): initial implementation of builtins.import
This adds an initial working version of builtins.import which
encapsulates the entire functionality of `import` within the builtin
itself, without requiring any changes in the compiler or VM.

The key insight that enables this is that we can simply return a Thunk
from `import` that is constructed from the output of running the
compiler and - ta-da! - no other component needs to know about it.

A couple of notes:

* builtins.import needs to capture variables like the SourceCode
  structure. This means it can not currently be constructed the same
  way as other builtins and has special handling, which leaks out to
  `eval.rs`. I have postponed dealing with that until we have this
  working a bit more.

* the `globals` are not yet passed through

* the error representation for the new variants is absolutely not done
  yet, we probably want to switch to something that supports
  cause-chaining now (like miette)

* there is no mechanism for emitting warnings at runtime; we need to
  add that

Change-Id: I3117a7ae3ff2432bf44f5ff05ad35f47faca31d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6857
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-06 15:22:32 +00:00
Vincent Ambo
89dc26cece feat(tvix/eval): implement builtins.currentTime
Returns time since epoch in seconds.

This has a slight behaviour difference from Nix, in that we don't pin
the time between REPL entries (Nix pins it for the program lifetime),
but this is probably inconsequential as long as it is pinned during an
evaluation.

Change-Id: I010c02e93097a209d8ad69e278397c7e30e54c86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6846
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-10-04 08:03:50 +00:00
Vincent Ambo
f0179c92d3 refactor(tvix/eval): allow impure Value builtins
Allows impure builtins that have a different shape than a Rust
function pointer; specifically this is required for
builtins.currentTime which does not work in WASM.

Change-Id: I1362d8eeafe770ce4d1c5ebe4d119aeb0abb5c9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6849
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-10-04 08:03:50 +00:00
Griffin Smith
9e16d70809 feat(tvix/eval): Sketch out impure builtins
Sketch out a new set of "impure" builtins, which supplement the existing
set of "pure" builtins but are gated behind a feature flag, which allows
them to be omitted by crates depending on tvix-eval that only want pure
evaluation, such as tvixbolt.

Change-Id: I2736017b5c9b4776bbba8758e108ec84887abd66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6655
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-22 01:36:44 +00:00