Commit graph

930 commits

Author SHA1 Message Date
sterni
8adc9c56f2 docs(tvix): document when pointer equality is preserved in C++ Nix
This explicitly documents behavior of C++ Nix that goes against the
intuition you'd gather from this document: that e.g. a simple select
from an attribute set causes a value to no longer be pointer equal to
its former self.

The point of documenting this is that we can show in a to be written
section on the use of pointer equality in nixpkgs that pointer equality
is only needed in a limited sense for evaluating it (C++ Nix's exterior
pointer equality). Tvix's pointer equality is far more powerful since
value identity preserving operations also preserve pointer equality,
generally speaking (this is because we implement interior pointer
equality in my made up terminology). This should eventually also be
documented.

Change-Id: I6ce7ef2d67b012f5ebc92f9e81bba33fb9dce7d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8856
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-11 16:17:42 +00:00
sterni
4ba624efae fix(tvix/eval): use byte, not codepoint index for slicing in escape
This fixes a subtle issue which would occasionally lead to a crash (e.g.
when evaluating (pkgs.systemd.outPath with --trace-runtime): With each
character in the string that has a multi byte representation in UTF-8,
the actual byte position and what tvix thought it was would get out of
sync. This could either lead to

* Tvix swallowing characters or jumbling characters if multi byte
  characters would cause the tracked index to become out of sync with
  the byte position before the first character to be escaped, or

* Tvix crashing if (in the same situation) the out of sync index would
  be within a UTF-8 byte sequence.

Luckily, std's `char_indices()` iterator implements exactly what
`nix_escape_char()`'s original author had in mind with
`.chars().enumerate()`. Using `i + 1` for continuing is safe, since all
characters that need (in fact, can) to be escaped in Nix are represented
as a single byte in UTF-8.

Change-Id: I1c836f70cde3d72db1c644e9112852f0d824715e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8952
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-07-11 16:11:40 +00:00
Florian Klink
5796563f1f chore(tvix/eval/vm): drop unused import
Change-Id: Ia04778391c198fde21da217bf697aa40157898b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8846
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-30 12:42:59 +00:00
Evgeny Zemtsov
c8fcdca4eb feat(tvix/eval): allow extending builtins outside of tvix_eval
The change allows applications that use tvix_serde for parsing
nix-based configuration to extend the language with domain-specific
set of features.

Change-Id: Ia86612308a167c456ecf03e93fe0fbae55b876a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8848
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-22 20:19:06 +00:00
sterni
66047063e0 fix(tvix/eval): use realpaths for import cache
I've noticed this behavior when writing the admittedly cursed test case
included in this CL. Alternatively we could use some sort of machinery
using `builtins.trace`, but I don't think we capture stderr anywhere.

I've elected to put this into the eval cache itself while C++ Nix does
it in builtins.import already, namely via `realisePath`. We don't have
an equivalent for this yet, since we don't support any kind of IfD, but
we could revise that later. In any case, it seems good to encapsulate
`ImportCache` in this way, as it'll also allow using file hashes as
identifiers, for example.

C++ Nix also does our equivalent of canon_path in `builtins.import`
which we still don't, but I suspect it hardly makes a difference.

Change-Id: I05004737ca2458a4c67359d9e7d9a2f2154a0a0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8839
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-21 07:48:52 +00:00
sterni
4516cd09c5 fix(tvix/eval): only finalise formal arguments if defaulting
When dealing with a formal argument in a function argument pattern that
has a default expression, there are two different things that can happen
at runtime: Either we select its value from the passed attribute
successfully or we need to use the default expression. Both of these may
be thunks and both of these may need finalisers. However, in the former
case this is taken care of elsewhere, the value will always be finalised
already if necessary. In the latter case we may need to finalise the
thunk resulting from the default expression. However, the thunk
corresponding to the expression may never end up in the local's stack
slot. Since finalisation goes by stack slot (and not constants), we need
to prevent a case where we don't fall back to the default expression,
but finalise anyways.

Previously, we worked around this by making `OpFinalise` ignore
non-thunks. Since finalisation of already evaluated thunks still
crashed, the faulty compilation of function pattern arguments could
still cause a crash.

As a new approach, we reinstate the old behavior of `OpFinalise` to
crash whenever encountering something that is either not a thunk or
doesn't need finalisation. This can also help catching (similar)
miscompilations in the future. To then prevent the crash, we need to
track whether we have fallen back or not at runtime. This is done using
an additional phantom on the stack that holds a new `FinaliseRequest`
value. When it comes to finalisation we check this value and
conditionally execute `OpFinalise` based on its value.

Resolves b/261 and b/265 (partially).

Change-Id: Ic04fb80ec671a2ba11fa645090769c335fb7f58b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8705
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-20 10:07:44 +00:00
sterni
77b0dddc3d chore(tvix/eval): fix markdown labeled link syntax
Change-Id: I639dc0801090eaba56b61858e28204b5a0e631b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8784
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-15 19:40:08 +00:00
sterni
3b8c9ec9c8 test(tvix/eval): update nix_tests suite to C++ Nix master
Adds new tests for foldl', intersectAttrs as well as fills in missing
.exp files.

New test cases we don't pass:

- fromTOML timestamp capabilities
- path antiquotation
- replaceStrings is lazier on C++ Nix master

The C++ Nix revision used is 7066d21a0ddb421967980094222c4bc1f5a0f45a.

Change-Id: Ic619c96e2d41e6c5ea6fa93f9402b12e564af3c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8778
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-06-15 19:28:16 +00:00
sterni
9278a0cd16 docs(tvix/eval): update test suite documentation
Change-Id: Ie9153c00b95ede4837a8eeab341e68bc90e97921
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8777
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-15 19:28:16 +00:00
sterni
0f71d8f813 test(tvix/eval): genericClosure (pointer) comparison support
genericClosure has very limited support for pointer equality: It relies
on comparison (not equality!) in C++ Nix, so as soon as C++ Nix supports
comparing lists (langVersion >= 6) we can rely on pointer equality for
key.

Since Tvix uses equality, not comparison for the insert, our behavior is
currently different, as documented by the notyetpassing tests.

Change-Id: Ifcd741ed4fc3ccc3825f7038875d56a9918b786a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8720
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-15 11:01:06 +00:00
sterni
0005737f11 fix(tvix/eval): make tvix display values like nix-instantiate(1)
In order for the test suite we have currently to be comparable to C++
Nix, we need to display values in the same way. This was largely the
case except in some weird cases.

* <CODE> for thunks and <CYCLE> for repeated thunks (?) are already in
  use. <CODE> formatting is tested by the oracle test suite already.

* Instead of lambda, we need to use <LAMBDA>

* <<primop>> and <<primop-app>> (a formatting C++ Nix uses nowhere)
  now are <PRIMOP> and <PRIMOP-APP>.

We'll probably want to have a fancier display of values (in a separate
trait) down the line. This could be used for interactive usage, e.g. the
REPL or a potential debugger.

There is a peculiarity with C++ Nix 2.3 formatting primops: import is
considered a <<PRIMOP-APP>>, since it is internally implemented by means
of scopedImport. This implementation detail no longer leaks in C++ Nix
2.13 nor in Tvix.

<CYCLE> display is untested at the moment, since we exhibit a
discrepancy to C++ Nix 2.3. Our current detection is more similar to C++
Nix 2.13—luckily it is also the more consistent of the two. See also
b/245.

Change-Id: I1d534434b02e470bf5475b3758920ea81e3420dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8760
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-06-15 11:01:06 +00:00
sterni
e409d9cc7e docs(tvix/eval): fix link to tvixbolt
Change-Id: Iff3f74ab6d5177246811bd3d58d171088915370f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8775
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-14 16:22:07 +00:00
sterni
80a71abb48 test(tvix/eval): move division by zero tests into tvix_tests
These were added by us in r/5276, so they should go into our test suite.

Change-Id: I6dc74fc242f33c22a17e0b4aee546ccae886ac85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-14 16:22:07 +00:00
sterni
1125b6b7b7 test(tvix/eval): add test case for builtins set pointer equality
Unsupported by Tvix at the moment. Documents b/280.

Change-Id: I48844feeefa9da8ed7e5d85300d52bb5650f82d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8772
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-14 16:01:19 +00:00
sterni
d5b989ddc0 test(tvix/eval): re-enable blackhole test
Change-Id: Id933f3bd708aa3342b9fd6a5584e65ee11751ff8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8773
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-14 15:59:45 +00:00
sterni
8e5551329a fix(tvix/eval): don't thunk home relative paths
C++ Nix resolves home relative paths at [parse] time. This is not an
option for us, since it prevents being able to separate the compilation
and execution phase later (e.g. precompiled nix expressions). However, a
practical consequence of this is that paths expressions are always
literals (strict) and never thunks.

[parse]: 7066d21a0d/src/libexpr/parser.y (L518-L527)

Change-Id: Ie4b9dc68f62c86d6c7fd5f1c9460c850d97ed1ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7041
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 10:57:28 +00:00
sterni
5139cc45c2 test(tvix/eval): builtins.substring's behavior with negative args
Change-Id: Ie8b97d174e9d58e33bf08c9b9e0afeeddd089ba8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8700
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-12 09:38:08 +00:00
Linus Heckemann
5733274876 fix(tvix/eval): allow negative substring lengths
Nix uses string::substr without checking the sign of the length[1].
The NixOS testing infrastructure relies on this[2], and on the
implicit conversion of that to the maximum possible value for a
size_t.

[1]: ecae62020b/src/libexpr/primops.cc (L3597)
[2]: c7c2984716/nixos/lib/testing/driver.nix (L29)

Change-Id: I6d0caf6830b6bda3fdf44c40c81de6a1befeca7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8746
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-12 08:38:49 +00:00
Linus Heckemann
2b4ad47c16 fix(tvix/eval): emit only warnings on shadowed outputs
Unfortunately, nixpkgs has at least one case[1] where the out environment
variable is shadowed -- though it doesn't cause a problem, since it's
shadowed with the correct value, odd as this may be!

[1]: c7c2984716/pkgs/development/python-modules/pybind11/default.nix (L19)

Change-Id: Ibf6790d2484dc9cce8e424feeb5886664d498dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8696
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-11 20:59:34 +00:00
sterni
a2303da01f fix(tvix/eval): use normal thunking behavior for default in formals
When comparing to C++ Nix, we notice that the thunking of default
expressions in function formals corresponds to their normal thunking,
e.g. literals are not thunked. This means that we can just invoke
compile() without much of a care and trust that it will sort it out
correctly.

If function formals blow up as a result of this, it likely indicates
that the expression is treated incorrectly by compile(), not
compile_param_pattern().

Change-Id: I64acbff2f251423eb72ce43e56a0603379305e1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8704
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-07 15:19:21 +00:00
sterni
10c6cb7251 fix(tvix/eval): type check function argument with set pattern
C++ Nix forces and typechecks the passed argument even if it is not
necessary in order to compute the return value of the function. I
discovered this when I thought our formals miscompilation might be that
we are too strict, but doesn't look like it in this case.

Change-Id: Ifb3c92592293052c489d1e3ae8c7c54e4b6b4dc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8701
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-07 15:17:20 +00:00
sterni
617130b088 refactor(tvix/eval): don't track idx twice in compile_param_pattern
Change-Id: I27f9105ddb20d84342550b2a73b479a7764ee3fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8699
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-06-07 15:17:19 +00:00
sterni
b492067c19 test(tvix/eval): check thunking behavior of basic exprs
nix_oracle.rs now gives us the possibility to check this by stuffing the
expressions in a list. In fact, the incorrect behavior fixed in

- cl/8656
- cl/8655
- cl/8662

was discovered using this test suite.

Change-Id: Id0ab01ee6be0b875791214e0a72a2ac941c46c96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8658
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-29 19:36:42 +00:00
sterni
0ab6494286 refactor(tvix/eval/nix_oracle): allow specifying eval strictness
This will be useful for comparing thunking behavior to C++ Nix. I
considered adding this capability to the tvix_tests/nix_tests
infrastructure, but as it would require changing the test file naming
scheme to do it in a clean way, I've postponed it–it's nice that our
tests are compatible with C++ Nix's test suite.

Change-Id: I60bcdd98ed25140e716f0858f8dc28f21ab957aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8657
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-29 19:36:41 +00:00
sterni
d09f333d0e fix(tvix/eval): thunk lambda expressions
As cl/8658 and b/274 reveal, lambda expressions are also wrapped in
thunks.

Resolves b/274.

Change-Id: I02fe5c8730ac76748d940e4f4427116587875275
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8662
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-29 19:36:41 +00:00
sterni
2aab01ac29 fix(tvix/eval): thunk HasAttr expressions
HasAttrs was weird because with longer attribute paths it would
sometimes not turn out to be a thunk. If it was a thunk, it'd usually
still do some eval strictly which we'll want to avoid.

Verified against C++ Nix using a new test suite introduced in a later
CL.

Change-Id: I6d047ccc68d046bb268462f170a3c4f3c5ddeffe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8656
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-29 12:44:09 +00:00
sterni
9d0425acc0 fix(tvix/eval): thunk legacy let to match regular one
Probably no real world code broken by this overzealous evaluation, but
let's be thorough!

Change-Id: Ib405a677182eab7940ace940c68e107573473a54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8655
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-05-29 12:44:08 +00:00
sterni
385c797884 fix(tvix/eval): thunk unary operator applications
Unary operator applications are thunked which can easily be observed by

  nix-instantiate --eval -E '[ (!true) (-1) ]'

Unfortunately, there are few simple expressions where this makes a
difference in the end result. Thus it only cropped up when using nixpkgs
for cross compilation: Here we would compile the expression

  !(stdenv.cc.isGNU or false)

to assemble python3Minimal's passthru attribute set (at least this seems
to be the most likely explanation from the backtraces I've studied).
This means that an unthunked

    <stdenv.cc.isGNU or false>
    OpForce
    OpInvert

would be performed in order to assemble this attribute set, causing
stdenv.cc to be evaluated too early, causing an infinite recursion.

Resolves b/273.

It seems that having a test suite that doesn't use --strict and relies
on thunks rendered as <CODE> would be beneficial for catching such
issues. I've not been able to find a test case with --strict that
demonstrates the problem fixed in this CL.

Change-Id: I640a5827b963f5b9d0f86fa2142e75e3a6bbee78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8654
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-29 12:44:08 +00:00
sterni
3b33c19a9c fix(tvix): don't call function eagerly in genList, map & mapAttrs
mapAttrs, map and genList call Nix functions provided by the caller and
store the result of applying them in a Nix data structure that does not
force all of its contents when forced itself. This means that when such
a builtin application is forced, the Nix function calls performed by the
builtin should not be forced: They may be forced later, but it is also
possible that they will never be forced, e.g. in

    builtins.length (builtins.map (builtins.add 2) [ 1 2 3 ])

it is not necessary to compute a single application of builtins.add.

Since request_call_with immediately performs the function call
requested, Tvix would compute function applications unnecessarily before
this change. Because this was not followed by a request_force, the
impact of this was relatively low in Nix code (most functions return a
new thunk after being applied), but it was enough to cause a lot of
bogus builtins.trace applications when evaluating anything from
`lib.modules`. The newly added test includes many cases where Tvix
previously incorrectly applied a builtin, breaking a working expression.

To fix this we add a new helper to construct a Thunk performing a
function application at runtime from a function and argument given as
`Value`s. This mimics the compiler's compile_apply(), but does itself
not require a compiler, since the necessary Lambda can be constructed
independently.

I also looked into other builtins that call a Nix function to verify
that they don't exhibit such a problem:

- Many builtins immediately use the resulting value in a way that makes
  it necessary to compute all the function calls they do as soon as
  the outer builtin application is forced:

  * all
  * any
  * filter
  * groupBy
  * partition

- concatMap needs to (shallowly) force the returned list for
  concatenation.

- foldl' is strict in the application of `op` (I added a comment that
  makes this explicit).

- genericClosure needs to (shallowly) force the resulting list and some
  keys of the attribute sets inside.

Resolves b/272.

Change-Id: I1fa53f744bcffc035da84c1f97ed25d146830446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8651
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-26 22:35:39 +00:00
Vincent Ambo
22776280b5 feat(tvix/eval): unthunk empty lists and attribute sets
Change-Id: Ie66cb1b163a544d45d113fd0f866286f230b0188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7960
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-05-25 11:28:13 +00:00
Vincent Ambo
5d3fb33baa feat(tvix/eval): implement unthunking in compiler
This feature allows the compiler to detect situations where the
created thunk is useless and can be merged into the parent chunk
instead.

The only case where the compiler does this initially is when
statically optimising a select expression.

For example, previously the expression `builtins.length` compiled into
two thunks:

1. An "inner" thunk which contained an `OpConstant` that had the
   optimised `length` builtin in it.

2. An "outer" thunk which contained an `OpConstant` to access the
   inner thunk, and the trailing OpForce of the top-level program.

With this change, the inner thunk is skipped completely and the outer
chunk directly contains the `length` builtin access.

This can be applied in several situations, some easier than others,
and we will add them in as we go along.

Change-Id: Ie44521445fce1199f99b5b17712833faea9bc357
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7959
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-05-25 11:28:13 +00:00
Vincent Ambo
3aea9bf527 feat(tvix/eval): implement Chunk::extend method
This method extends the contents of one chunk with that of another,
effectively merging the thunks together.

This will be used for the upcoming "unthunking" functionality.

Change-Id: I6ad74232cd7f3eca198ed921e455205e00d76e6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7958
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-25 11:28:13 +00:00
Florian Klink
d25962b9a4 refactor(tvix/eval): stop borrowing &mut self
This does undo cl/8571.

Change-Id: Ib14b4e7404f906e346304b6113860ae811afc94a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8631
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 11:11:59 +00:00
Florian Klink
11771a06ae refactor(tvix/eval): use &Path instead of PathBuf
This allows getting rid of some clones in eval/src/vm/generators.rs.

Change-Id: I330390307d3bcfeef19c98954c753ee55b1ccee3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8604
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-22 09:43:33 +00:00
Florian Klink
b4bb9062ea fix(tvix/eval): add path where useful to ErrorKind::IO
These two places didn't add the path from the context to the
ErrorKind, but simply relied on the
impl From<std::io::Error> for tvix_eval::ErrorKind, which doesn't add
the path.

Change-Id: Ifc7dbbe305d24242b0705de1dea34e8923e9d2cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8603
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-22 09:43:21 +00:00
Florian Klink
b7ab6c0856 refactor(tvix/eval/io): use io::Error instead of tvix_eval errors
We didn't return anything useful other than ErrorKind::IO anyways.

We can use io::ErrorKind::Unsupported for DummyIO.

Fixes b/271.

Change-Id: Icb231e9b38168e8b6fa473bfa405d160357b317f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8602
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-22 09:42:58 +00:00
Florian Klink
8bd7ced1fb feat(tvix/eval/io): allow &mut self in EvalIO
It's okay if these calls mutate some internal state inside an
implementation.

Change-Id: I12bb11bde0310778c3da1275696bf7de058863a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8571
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-14 18:59:55 +00:00
Vincent Ambo
969fbcd6d4 fix(tvix/eval): builtins.trace prints to stderr
Change-Id: Icf577396035474d6977e627058aba5805c61985e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8563
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-05-12 12:50:59 +00:00
Florian Klink
6a30eb69e0 chore(tvix/*): bump to smol_str 0.2.0
Change-Id: Ic9ac1b6fecb564eafb41b265bf317cd385fdc170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8560
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-11 18:19:20 +00:00
Vincent Ambo
ce502bdc89 refactor(tvix/eval): improve representation of chunk/span mapping
This switches out the previous compressed representation (count of
instructions per span) with a representation where the chunk's span
list stores the index of the first operation that belongs to a span,
and finds the right span by using a binary search when looking them
up.

This improves the lookup complexity from O(n) to O(log n).

This improvement was suggested and (mostly) implemented by GPT-4. I
only fixed up some names and updated the logic for deleting
spans (which it only did not do because I didn't tell it about that).

The code was verified by producing a complex error before/after the
change and ensuring that all spans in the error match exactly.

Co-Authored-By: GPT-4
Change-Id: Ibfa12cc6973af1c9b0ae55bb464d1975209771f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8385
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-03-31 16:46:19 +00:00
Vincent Ambo
2d305fd5b3 refactor(tvix/eval): retain call frames when entering calls
This grows the frame stack as the call stack grows, which yields *much*
better user-facing error messages.

I haven't measured the performance impact this has yet, for now I'm
still just trying to add more information to errors and then cut down
again where necessary.

Change-Id: I89f058ef31979edacf4667775d460b60704ce4d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8334
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-03-27 09:02:43 +00:00
Vincent Ambo
ba138712e4 feat(tvix/eval): add Evaluation::strict to toggle top-level deepseq
This makes it possible for callers to control whether they can receive
partially evaluated values from an evaluation or not.

We're actually flipping the default behaviour to non-strict top-level
evaluation, which means that callers have to set `strict = true` on
the Evaluation to get the previous behaviour.

Change-Id: Ic048e9ba09c88866d4c3177d5fa07db11c4eb20e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8325
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-22 13:44:20 +00:00
Vincent Ambo
a5f28eea94 fix(tvix/eval): print unevaluated thunks like Nix does
Change-Id: Ie4c563e933f571f45cb4f4efe650d1b65f119e8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8324
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2023-03-22 13:44:19 +00:00
Vincent Ambo
80aaadfc19 fix(tvix/eval): use span of set for OpForce in attribute access
Emits the span of the `set` that is being accessed in the `force`
operation of an attribute access.

Looking at traces, it's a lot more useful to get information about
*what* is being forced, as in cases like `foo.bar` it can be
misleading to have an error highlight `bar`, when the error occured
while forcing `foo` to be able to access `bar` in the first place.

Change-Id: Id46ff28f20c67cb4971727ac52cc4811795cea2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8272
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-17 19:40:40 +00:00
Vincent Ambo
8c13f18d11 feat(tvix/eval): report all known spans on infinite recursion
This reports the span

1. of the code within a thunk,
2. of the place where the thunk was instantiated,
3. of the place where the thunk was first forced,
4. of the place where the thunk was forced again,

when yielding an infinite recursion error, which hopefully makes it
easier to debug them.

The spans are tracked in the ThunkRepr::Blackhole variant when putting
a thunk under evaluation.

Note that we currently have some loss of span precision in the VM loop
when switching between frame types, so spans 3/4 are currently a bit
wonky. Working on it.

Change-Id: Icbd2a9df903d00e8c2545b3fc46dcd2a9e3e3e55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8270
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-03-17 19:32:38 +00:00
Vincent Ambo
3fa6b13c1e feat(tvix/eval): track span of first force in a thunk blackhole
This is step 1 towards being able to use all 4 spans that we know when
dealing with infinite recursion. It tracks the span at which the
force of a thunk was first requested when constructing a blackhole, so
that we can highlight the spans of the first and second forces.

These are actually the least relevant spans, but the easiest to put in
place, more coming soon.

Change-Id: I4c7e82f6211b98756439d4148a4191457cc46807
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8269
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-03-17 19:31:37 +00:00
Vincent Ambo
5095e4f269 feat(tvix/eval): add generator "name" to NativeError kind
This produces traces in which we can see what kind of native code was
run. Note that these "names" are named after the generator message, so
these aren't *really* intended for end-user consumption, but we can
give them saner names later.

Example:
https://gist.github.com/tazjin/82b24e92ace8e821008954867ee05057

This already makes the traces a little easier to parse.

Change-Id: Idcd601baf84f492211b732ea0f04b377112e10d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8268
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-03-17 19:31:37 +00:00
Vincent Ambo
ea80e0d3f8 feat(tvix/eval): enrich errors with VM's frame stack information
When emitting an error at runtime, the VM will now use the new
`NativeError` and `BytecodeError` error kinds (which just wrap inner
errors) to create a set of diagnostics to emit.

The primary diagnostic is emitted last, with `error` type (so it will
be coloured red in terminals), the other ones will be emitted with
`note` type, highlighting the causal chain.

Example:
https://gist.github.com/tazjin/25feba7d211702453c9ebd5f8fd378e4

This is currently quite verbose, and we can cut down on this further,
but the purpose of this commit is to surface more information first of
all before worrying about the exact display.

Change-Id: I058104a178c37031c0db6b4b3e4f4170cf76087d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8266
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-17 19:31:37 +00:00
Vincent Ambo
b78ae941a4 fix(tvix/eval): use coerce_to_string in builtins.substring
This actually uses coercion under the hood in C++ Nix. See the test
for an example.

Change-Id: Id56b364acf269225b6829d0b600e0222f8b3608d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8322
Reviewed-by: andi <andi@notmuch.email>
Tested-by: BuildkiteCI
2023-03-17 19:17:29 +00:00
Vincent Ambo
b5f5a1595d chore(tvix/eval): remove some dead code
This was commented out and forgotten during the generator refactor, oh
well.

Change-Id: I474b685159a955a846db462da0dd0067af177b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8321
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-17 11:01:07 +00:00