mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(tvix/eval): update test suite documentation

Browse source 
Change-Id: Ie9153c00b95ede4837a8eeab341e68bc90e97921
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8777
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit is contained in:
sterni 2023-06-14 17:20:39 +02:00 committed by clbot
parent a5e308587e
commit 9278a0cd16
1 changed files with 43 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

52
tvix/eval/README.md
View file

@ -31,19 +31,53 @@ git clone https://code.tvl.fyi/depot.git:/tvix/eval.git tvix-eval
cd tvix-eval && cargo build cd tvix-eval && cargo build
``` ```
## Nix test suite ## Tests
C++ Nix implements a language test suite in the form of Nix source Tvix currently has three language test suites for tvix-eval:
code files, and their expected output. The coverage of this test suite
is not complete, but we intend to be compatible with it.
We have ported the test suite to Tvix, but do not run it by default as * `nix_tests` and `tvix_tests` are based on the same mechanism
we are not yet compatible with it. borrowed from the C++ Nix implementation. They consist of
Nix files as well as expected output (if applicable).
The test cases are split into four categories:
`eval-okay` (evaluates successfully with the expected output),
`eval-fail` (fails to evaluate, no expected output),
`parse-okay` (expression parses successfully, no expected output)
and `parse-fail` (expression fails to parse, no expected output).
Tvix currently ignores the last two types of test cases, since
it doesn't implement its own parser.
You can run the test suite by enabling the `nix_tests` feature in Both test suites have a `notyetpassing` directory. All test cases
Cargo: in here test behavior that is not yet supported by Tvix. They are
considered to be expected failures, so you can't forget to move
them into the test suite proper when fixing the incompatibility.
cargo test --features nix_tests Additionally, separate targets in the depot pipeline, under
`//tvix/verify-lang-tests`, check both test suites (including
`notyetpassing` directories) against
C++ Nix 2.3 and the default C++ Nix version in nixpkgs.
This way we can prevent accidentally introducing test cases
for behavior that C++ Nix doesn't exhibit.
* `nix_tests` has the test cases from C++ Nix's language test
suite and is sporadically updated by manually syncing the
directories. The `notyetpassing` directory shows how far
it is until we pass it completely.
* `tvix_tests` contains test cases written by the Tvix contributors.
Some more or less duplicate test cases contained in `nix_tests`,
but many cover relevant behavior that isn't by `nix_tests`.
Consequently, it'd be nice to eventually merge the two test
suites into a jointly maintained, common Nix language test suite.
It also has a `notyetpassing` directory for missing behavior
that is discovered while working on Tvix and isn't covered by the
`nix_tests` suite.
* `nix_oracle` can evaluate Nix expressions in Tvix and compare the
result against C++ Nix (2.3) directly. Eventually it should gain
the ability to property test generated Nix expressions.
An additional feature is that it can evaluate expressions without
`--strict`, so thunking behavior can be verified more easily.
## rnix-parser ## rnix-parser