tvl-depot/web/tvl/blog/tvix-status-202209.md

We've now been working on our rewrite of Nix, Tvix, on-and-off for over a year.

Of course, for many of us, it's been a pretty turbulent time period. While steady progress has been made, we haven't really had the bandwidth to communicate and publicise what has been going on - this blog post aims to rectify that!

Nix language evaluator

The most significant progress in the last months has been made on our Nix language evaluator. To address a big question right away: Yes, you can play with it - in Tvixbolt!

We developed the evaluator to the current state by enumerating the various problems we were likely to encounter, and writing small-scale solutions to them before assembling them into a whole. Due to the nature of this process, we briefly ended up with a very large private source tree, which we integrated into our monorepo in the last couple of weeks.

This process was slow mostly due to code review bandwidth, but remember that we are just volunteers and such bottlenecks are to be expected!

Most of this code was written or reviewed by tazjin, grfn and sterni.

So, what's working now?

The answer is most things! You can enter many Nix language expressions in Tvixbolt and observe how they are evaluated.

There's a lot of interesting stuff going on under-the-hood, notably:

• The Tvix compiler is built to be able to emit warnings & errors without failing early, as well as retaining as much source information as possible. This will enable developer tooling, such as language servers, to be based on Tvix.

• The Tvix compiler performs very in-depth scope analysis, which allows it to both generate efficient bytecode for accessing identifiers, as well as alert users about problems in their code before runtime.

• The runtime supports tail-call optimisation in many (but not all (yet!)) cases, allowing us to evaluate many recursive expressions in constant stack space.

• The runtime supports having different backing representations for the same Nix type. For example, an attribute set may be represented differently based on whether it is empty, a name/value pair or a larger set.

We've also (constrained by the available features) run some initial benchmarks against C++ Nix, and in most cases Tvix evaluation is an order of magnitude faster. However, these benchmarks are in no way indicative of real-life performance when evaluating something like nixpkgs yet, so stay tuned for more information on performance.

How does it all work?

Tvix's evaluator is implemented using a custom abstract machine with a very Nix-specific instruction set, as well as a compiler that traverses a parsed Nix AST to emit this bytecode and perform a set of optimisations and other analysis. The most important benefit of this is that we can plan and lay out the execution of a program in a way that is better suited to an efficient runtime than directly traversing the AST.

TIP: You can see the generated bytecode in Tvixbolt!

This is all written in Rust (of course) and is currently made up of less than 5000 lines of code (some of which look deceptively simple, especially around scope-handling!).

We run the evaluator against many custom tests we have written as part of our CI suite (through cargo test), as well as against the upstream Nix test suite (which we do not yet pass, but are working towards).

What's next for tvix-eval?

Despite everything, there are some unfinished and important feature areas:

1. The majority of Nix's builtins are not yet implemented (including fundamental ones such as import and derivation).

2. Recursive attribute sets (rec) are not yet implemented. This is actually not because of the recursion in itself, but because of the handling of nested keys (such as a.b), for which we are designing a more efficient solution than what is currently in place.

In both cases we have mostly figured out how to do the remaining work and it is simply a question of time until we've done it. Progress is steady and can of course be tracked in the source (viewer without Javascript here).

Apart from that, the next steps are:

• Comprehensive benchmarking. We are standing up an infrastructure for continuous benchmarking to measure the impact of changes, and to be able to identify and optimise hotspots.

• Implementing known optimisations. There are some areas of the code where we are aware of (significant) possible speed gains, but we are holding off of implementing them until the evaluator is feature complete and passes the Nix language test suite.

• Finishing our language specification. Based on the behaviours we've learned, we are writing a specification of the Nix language that captures its various (sometimes subtly tricky) behaviours.

Once we can evaluate nixpkgs, focus is likely to shift towards the other areas of Tvix.

The Other Areas of Tvix

Speaking of these other areas (most importantly, the builder and store implementation), some progress has been made there also.

While we haven't begun piecing together the final implementations, flokli and adisbladis have been hard at work on go-nix which aims to implement many of the low-level primitives required for Nix (hashing and encoding schemes, archive formats, reference scanning ...).

We're looking forward to being able to tell you more about this in the next update!

Outro ...

We'd be delighted to onboard new contributors to Tvix! Please take a look at the main TVL page to find out how to get in touch with us if you'd like to join!

Thanks also, of course, to NLNet for sponsoring some of this work!

And finally, we would like to thank and pay our respects to jD91mZM2 - the original author of rnix-parser - who sadly passed away. We use rnix-parser in our compiler, and its well-designed internals (also thanks to its new maintainers!) have saved us a lot of time.

That's it for this update, go play with Tvixbolt, tell us about the weird ways in which you break it, get in touch, and we'll see you around!