mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(web/tvl/blog): another editing pass over 2024-02 tvix update

Browse source 
Change-Id: I1396ab24c918fc91e37ce7492b041356272a018c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10885
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This commit is contained in:
Vincent Ambo 2024-02-15 17:54:33 +07:00 committed by clbot
parent 57cdebff8e
commit 4dc112cb9c
1 changed files with 31 additions and 28 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

59
web/tvl/blog/2024-02-tvix-update.md
View file

@ -95,7 +95,7 @@ content-addressed systems, as it's a very Nix-specific format.
### The more granular Tvix model ### The more granular Tvix model
After experimenting with some concepts and ideas in Golang, mostly around how to After experimenting with some concepts and ideas in Golang, mostly around how to
improve binary cache performance [^3], both on-disk as well as over the network, improve binary cache performance[^3], both on-disk as well as over the network,
we settled on a more granular, content-addressed and general-purpose format. we settled on a more granular, content-addressed and general-purpose format.
Internally, it behaves very similar to how git handles tree objects, except Internally, it behaves very similar to how git handles tree objects, except
@ -138,7 +138,7 @@ storages, as well as on more granular seeking and content-defined chunking for
blobs. blobs.
### FUSE/virtiofs ### FUSE/virtiofs
A tvix-store can be mounted via FUSE, or exposed through virtiofs [^4]. A tvix-store can be mounted via FUSE, or exposed through virtiofs[^4].
While doing the obvious thing - allowing mounting and browsing the contents While doing the obvious thing - allowing mounting and browsing the contents
of the store, this will allow lazy substitution of builds on remote builders, be of the store, this will allow lazy substitution of builds on remote builders, be
in containerized or virtualized workloads. in containerized or virtualized workloads.
@ -161,7 +161,7 @@ The necessary parsers for NARInfo, signatures etc are also available in the
## EvalIO / builtins interacting with the store more closely ## EvalIO / builtins interacting with the store more closely
tvix-eval itself is designed to be quite pure when it comes to IO - it doesn't tvix-eval itself is designed to be quite pure when it comes to IO - it doesn't
do any IO directly on its own, but for the very little IO functionality it do any IO directly on its own, but for the very little IO functionality it
does as part of "basic interaction with paths"[^2] (like importing other does as part of "basic interaction with paths" (like importing other
`.nix` files), it goes through an `EvalIO` interface, which is provided to the `.nix` files), it goes through an `EvalIO` interface, which is provided to the
Evaluator struct on instantiation. Evaluator struct on instantiation.
@ -170,10 +170,10 @@ which becomes interesting for specific store implementations that might not
expose a POSIX filesystem directly, or targets where we don't have a filesystem expose a POSIX filesystem directly, or targets where we don't have a filesystem
at all (like WASM). at all (like WASM).
Using the `EvalIO` trait also allows avoiding the `tvix-eval` crate to get too Using the `EvalIO` trait also lets `tvix-eval` avoid becoming too strongly
strongly coupled to a specific store implementation, hashing scheme etc [^2]. coupled to a specific store implementation, hashing scheme etc[^2]. As we can
As we can extend the set of builtins available to the evaluator with "foreign extend the set of builtins available to the evaluator with "foreign builtins",
builtins", these can live in other crates. these can live in other crates.
Following this pattern, we started implementing some of the "basic" builtins Following this pattern, we started implementing some of the "basic" builtins
that deal with path access in `tvix-eval`, like: that deal with path access in `tvix-eval`, like:
@ -198,38 +198,40 @@ remaining changes should land quite soon.
## Catchables / tryEval ## Catchables / tryEval
As you may know, Nix has a limited exception system for dealing with Nix has a limited exception system for dealing with user-generated errors:
user-generated errors: `builtins.tryEval` can be used to detect if an expression `builtins.tryEval` can be used to detect if an expression fails (if
fails (if `builtins.throw` or `assert` are used to generate it). This feature `builtins.throw` or `assert` are used to generate it). This feature requires
requires extra support in any Nix implementation, as errors may not necessarily extra support in any Nix implementation, as errors may not necessarily cause the
cause the Nix program to abort. Nix program to abort.
The C++ Nix implementation reuses the C++ language-provided Exception system for
`builtins.tryEval` which Tvix can't (even if Rust had an equivalent system):
The C++ Nix implementation just reuses the C++ language-provided Exception
system for `builtins.tryEval` which Tvix can't (even if Rust had an equivalent
system):
In C++ Nix the runtime representation of the program in execution corresponds In C++ Nix the runtime representation of the program in execution corresponds
to the Nix expression tree of the relevant source files. This means that an to the Nix expression tree of the relevant source files. This means that an
exception raised in C++ code will automatically bubble up correctly since the exception raised in C++ code will automatically bubble up correctly since the
C++ and Nix call stacks are equivalent to each other. C++ and Nix call stacks are equivalent to each other.
Tvix compiles the Nix expressions to a byte code program which may be mutated
by extra optimization rules (for example, we hope to eliminate as many thunks as Tvix compiles the Nix expressions to a byte code program which may be mutated by
possible in the future). This means that such a correspondence between Nix and extra optimization rules (for example, we hope to eliminate as many thunks as
the (in this case) VM runtime is not guaranteed. possible in the future). This means that such a correspondence between the state
of the runtime and the original Nix code is not guaranteed.
Previously, `builtins.tryEval` (which is implemented in Rust and can access VM Previously, `builtins.tryEval` (which is implemented in Rust and can access VM
internals) just allowed the VM to recover from certain kinds of errors. This internals) just allowed the VM to recover from certain kinds of errors. This
proved to be insufficient as it [blew up as soon as a `builtins.tryEval`-ed thunk proved to be insufficient as it [blew up as soon as a `builtins.tryEval`-ed
is forced again][tryeval-infrec]–extra bookkeeping was needed. As a thunk is forced again][tryeval-infrec] – extra bookkeeping was needed. As a
solution, we now store thunk evaluation errors that can be recovered from as solution, we now store recoverable errors as a separate runtime value type.
`Value::Catchable` which mitigates this problem.
As you can imagine, storing evaluation failures as "normal" values quickly leads As you can imagine, storing evaluation failures as "normal" values quickly leads
to all sorts of bugs because most VM/builtins code is written with only ordinary to all sorts of bugs because most VM/builtins code is written with only ordinary
values like attribute sets, strings etc. in mind. values like attribute sets, strings etc. in mind.
While ironing those out, we made sure to supplement those fixes with as many While ironing those out, we made sure to supplement those fixes with as many
test cases for `builtins.tryEval` as possible. This will hopefully prevent any test cases for `builtins.tryEval` as possible. This will hopefully prevent any
regressions if or rather when we touch this system again. We already have some regressions if or rather when we touch this system again. We already have some
ideas for replacing the `Catchable` value type with a cleaner representation. ideas for replacing the `Catchable` value type with a cleaner representation,
but first we want to pin down all the unspoken behaviour.
## String contexts ## String contexts
@ -248,12 +250,12 @@ and added support for string contexts into our `NixString` implementation,
implemented the context-related builtins, and added more unit tests that verify implemented the context-related builtins, and added more unit tests that verify
string context behaviour of various builtins. string context behaviour of various builtins.
## Strings as bstr ## Strings as byte strings
C++ Nix uses C-style zero-terminated strings internally - however, until C++ Nix uses C-style zero-terminated strings internally - however, until
recently, Tvix has used Rust `String` and `str` for string values. Since those recently, Tvix has used standard Rust strings for string values. Since those are
are required to be valid utf-8, we haven't been able to properly represent all required to be valid UTF-8, we haven't been able to properly represent all the
the string values that Nix supports. string values that Nix supports.
We recently converted our internal representation to byte strings, which allows We recently converted our internal representation to byte strings, which allows
us to treat a `Vec<u8>` as a "string-like" value. us to treat a `Vec<u8>` as a "string-like" value.
@ -309,6 +311,7 @@ you run into any snags, or have any questions.
[^4]: Strictly speaking, not limited to tvix-store - literally anything [^4]: Strictly speaking, not limited to tvix-store - literally anything
providing a listing into tvix-castore nodes. providing a listing into tvix-castore nodes.
[Tvix]: https://tvix.dev
[aterm]: http://program-transformation.org/Tools/ATermFormat.html [aterm]: http://program-transformation.org/Tools/ATermFormat.html
[bazel-remote]: https://github.com/buchgr/bazel-remote/pull/715 [bazel-remote]: https://github.com/buchgr/bazel-remote/pull/715
[castore-docs]: https://cs.tvl.fyi/depot/-/blob/tvix/castore/docs [castore-docs]: https://cs.tvl.fyi/depot/-/blob/tvix/castore/docs