151 lines
6 KiB
Markdown
151 lines
6 KiB
Markdown
|
We've now been working on our rewrite of Nix, [Tvix][], on-and-off for
|
||
|
|
over a year.
|
||
|
|
|
||
|
|
Of course, for many of us, it's been a pretty turbulent time period.
|
||
|
|
While steady progress has been made, we haven't really had the
|
||
|
|
bandwidth to communicate and publicise what has been going on - this
|
||
|
|
blog post aims to rectify that!
|
||
|
|
|
||
|
|
## Nix language evaluator
|
||
|
|
|
||
|
|
The most significant progress in the last months has been made on our
|
||
|
|
Nix language evaluator. To address a big question right away: Yes, you
|
||
|
|
can play with it - in [Tvixbolt][]!
|
||
|
|
|
||
|
|
We developed the evaluator to the current state by enumerating the
|
||
|
|
various problems we were likely to encounter, and writing small-scale
|
||
|
|
solutions to them before assembling them into a whole. Due to the
|
||
|
|
nature of this process, we briefly ended up with a very large private
|
||
|
|
source tree, which we [integrated]() into our monorepo in the last
|
||
|
|
couple of weeks.
|
||
|
|
|
||
|
|
This process was slow mostly due to code review bandwidth, but
|
||
|
|
remember that we are just volunteers and such bottlenecks are to be
|
||
|
|
expected!
|
||
|
|
|
||
|
|
Most of this code was written or reviewed by [tazjin][], [grfn][]
|
||
|
|
and [sterni][].
|
||
|
|
|
||
|
|
### So, what's working now?
|
||
|
|
|
||
|
|
The answer is *most things*! However, there are some unfinished and
|
||
|
|
important feature areas:
|
||
|
|
|
||
|
|
1. The majority of Nix's `builtins` are not yet implemented (including
|
||
|
|
fundamental ones such as `import` and `derivation`).
|
||
|
|
|
||
|
|
2. Recursive attribute sets (`rec`) are not yet implemented. This is
|
||
|
|
actually not because of the recursion in itself, but because of the
|
||
|
|
handling of nested keys (such as `a.b`), for which we are designing
|
||
|
|
a more efficient solution than what is currently in place.
|
||
|
|
|
||
|
|
In both cases we have mostly figured out how to do the remaining work
|
||
|
|
and it is simply a question of time until we've done it. Progress is
|
||
|
|
steady and can of course be tracked [in the source][src] (viewer
|
||
|
|
without Javascript [here][src-noscript]).
|
||
|
|
|
||
|
|
At the same time, we've already implemented a variety of basics that
|
||
|
|
are hopefully going to have a huge impact further down, such as:
|
||
|
|
|
||
|
|
* The Tvix compiler is built to be able to emit warnings & errors
|
||
|
|
without failing early, as well as retaining as much source
|
||
|
|
information as possible. This will enable developer tooling, such as
|
||
|
|
language servers, to be based on Tvix.
|
||
|
|
|
||
|
|
* The Tvix compiler performs very in-depth scope analysis, which
|
||
|
|
allows it to both generate efficient bytecode for accessing
|
||
|
|
identifiers, as well as alert users about problems in their code
|
||
|
|
before runtime.
|
||
|
|
|
||
|
|
* The runtime supports tail-call optimisation in many (but not all
|
||
|
|
(yet!)) cases, allowing us to evaluate many recursive expressions in
|
||
|
|
constant stack space.
|
||
|
|
|
||
|
|
### How does this all work?
|
||
|
|
|
||
|
|
Tvix's evaluator is implemented using a custom abstract machine with a
|
||
|
|
very Nix-specific instruction set, as well as a compiler that
|
||
|
|
traverses a parsed Nix AST to emit this bytecode and perform a set of
|
||
|
|
optimisations and other analysis. The most important benefit of this
|
||
|
|
is that we can plan and lay out the execution of a program in a way
|
||
|
|
that is better suited to an efficient runtime than directly traversing
|
||
|
|
the AST.
|
||
|
|
|
||
|
|
TIP: You can see the generated bytecode in [Tvixbolt][]!
|
||
|
|
|
||
|
|
This is all written in Rust (of course) and is currently made up of
|
||
|
|
less than 5000 lines of code (some of which look deceptively simple,
|
||
|
|
especially around scope-handling!).
|
||
|
|
|
||
|
|
We run the evaluator against many custom tests we have written as part
|
||
|
|
of our CI suite (through `cargo test`), as well as against the
|
||
|
|
upstream Nix test suite (which we do not yet pass, but are working
|
||
|
|
towards).
|
||
|
|
|
||
|
|
### What's next for tvix-eval?
|
||
|
|
|
||
|
|
Apart from the missing language features outlined above, the next
|
||
|
|
steps are:
|
||
|
|
|
||
|
|
* Comprehensive benchmarking. We are standing up an infrastructure for
|
||
|
|
continuous benchmarking to measure the impact of changes, and to be
|
||
|
|
able to identify and optimise hotspots.
|
||
|
|
|
||
|
|
* Implementing known optimisations. There are some areas of the code
|
||
|
|
where we are aware of (significant) possible speed gains, but we are
|
||
|
|
holding off of implementing them until the evaluator is feature
|
||
|
|
complete and passes the Nix language test suite.
|
||
|
|
|
||
|
|
* Finishing our language specification. Based on the behaviours we've
|
||
|
|
learned, we are writing a specification of the Nix language that
|
||
|
|
captures its various (sometimes subtly tricky) behaviours.
|
||
|
|
|
||
|
|
Once we can evaluate `nixpkgs`, focus is likely to shift towards the
|
||
|
|
other areas of Tvix.
|
||
|
|
|
||
|
|
## The Other Areas of Tvix
|
||
|
|
|
||
|
|
Speaking of these other areas (most importantly, the builder and store
|
||
|
|
implementation), some progress has been made there also.
|
||
|
|
|
||
|
|
While we haven't begun piecing together the final implementations,
|
||
|
|
[flokli][] and [adisbladis][] have been hard at work on [go-nix][]
|
||
|
|
which aims to implement many of the low-level primitives required for
|
||
|
|
Nix (hashing and encoding schemes, archive formats, reference scanning
|
||
|
|
...).
|
||
|
|
|
||
|
|
We're looking forward to being able to tell you more about this in the
|
||
|
|
next update!
|
||
|
|
|
||
|
|
## Outro ...
|
||
|
|
|
||
|
|
We'd be delighted to onboard new contributors to Tvix! Please take a
|
||
|
|
look at the main [TVL page](https://tvl.fyi) to find out how to get in
|
||
|
|
touch with us if you'd like to join!
|
||
|
|
|
||
|
|
Thanks also, of course, to [NLNet](https://nlnet.nl/) for sponsoring
|
||
|
|
some of this work!
|
||
|
|
|
||
|
|
And finally, we would like to thank and pay our respects to jD91mZM2 -
|
||
|
|
the original author of
|
||
|
|
[rnix-parser](https://github.com/nix-community/rnix-parser) - who
|
||
|
|
sadly passed away. We use `rnix-parser` in our compiler, and its
|
||
|
|
well-designed internals (also thanks to its new maintainers!) have
|
||
|
|
saved us a lot of time.
|
||
|
|
|
||
|
|
That's it for this update, go play with [Tvixbolt][], tell us about
|
||
|
|
the weird ways in which you break it, get in touch, and we'll see you
|
||
|
|
around!
|
||
|
|
|
||
|
|
[Tvix]: https://tvl.fyi/blog/rewriting-nix
|
||
|
|
[Tvixbolt]: https://tvixbolt.tvl.su
|
||
|
|
[integrated]: https://cl.tvl.fyi/q/status:merged+%2522tvix/eval%2522+mergedbefore:2022-09-09
|
||
|
|
[src]: https://cs.tvl.fyi/depot/-/tree/tvix/eval
|
||
|
|
[src-noscript]: https://code.tvl.fyi/tree/tvix/eval
|
||
|
|
[tazjin]: https://tazj.in
|
||
|
|
[grfn]: https://gws.fyi/
|
||
|
|
[sterni]: https://github.com/sternenseemann
|
||
|
|
[go-nix]: https://github.com/nix-community/go-nix
|
||
|
|
[flokli]: https://github.com/flokli
|
||
|
|
[adisbladis]: https://github.com/adisbladis
|