docs(tvix): add a list of TODOs/ideas
This contains a rough collection of ideas on the TODO list, trying to keep track of it somewhere. Change-Id: Ifc5b0cf9f7ac38f7a8e56515882bdf70e349544b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11512 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
This commit is contained in:
Florian Klink
clbot
parent
7345986ff1
commit
f5bb9b8467
2 changed files with 130 additions and 0 deletions
|
|
@ -2,6 +2,7 @@
|
||||||
|
|
||||||
# Tvix
|
# Tvix
|
||||||
- [Architecture & data flow](./architecture.md)
|
- [Architecture & data flow](./architecture.md)
|
||||||
|
- [TODOs](./TODO.md)
|
||||||
|
|
||||||
# Nix
|
# Nix
|
||||||
- [Specification of the Nix Language](./language-spec.md)
|
- [Specification of the Nix Language](./language-spec.md)
|
||||||
|
|
|
||||||
129
tvix/docs/src/TODO.md
Normal file
129
tvix/docs/src/TODO.md
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
# TODO
|
||||||
|
|
||||||
|
This contains a rough collection of ideas on the TODO list, trying to keep track
|
||||||
|
of it somewhere.
|
||||||
|
|
||||||
|
Of course, there's no guarantee these things will get addressed, but it helps
|
||||||
|
dumping the backlog somewhere.
|
||||||
|
|
||||||
|
Feel free to add new ideas. Before picking something, ask in `#tvix-dev` to make
|
||||||
|
sure noone is working on this, or has some specific design in mind already.
|
||||||
|
|
||||||
|
## Cleanups
|
||||||
|
- move some of the rstest cases in `tvix-glue` to the `.nix`/`.exp` mechanism.
|
||||||
|
- Parts requiring test fixtures need some special convention.
|
||||||
|
Some of these also cannot be checked into the repo, like the import tests
|
||||||
|
adding special files to test filtering.
|
||||||
|
- add `nix_oracle` mechanism from `tvix-eval` to `tvix-glue`.
|
||||||
|
|
||||||
|
## Fixes towards correctness
|
||||||
|
- `builtins.toXML` is missing string context. See b/398.
|
||||||
|
- `builtins.toXML` self-closing tags need to be configurable in a more granular
|
||||||
|
fashion, requires third-party crate support. See b/399.
|
||||||
|
- `rnix` only supports string source files, but `NixString` uses bytes (and Nix
|
||||||
|
source code might be no valid UTF-8).
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
Extend the other pages in here. Some ideas on what should be tackled:
|
||||||
|
- Document what Tvix is, and what it is not yet. What it is now, what it is not
|
||||||
|
(yet), explaining some of the architectural choices (castore, more hermetic
|
||||||
|
`Build` repr), while still being compatible. Explain how it's possible to
|
||||||
|
plug in other frontends, and use `tvix-{[ca]store,build}` without Nixlang even.
|
||||||
|
And how `nix-compat` is a useful crate for all sorts of formats and data
|
||||||
|
types of Nix.
|
||||||
|
- Update the Architecture diagram to model the current state of things.
|
||||||
|
There's no gRPC between Coordinator and Evaluator.
|
||||||
|
- Add a dedicated section/page explaining the separation between tvix-glue and
|
||||||
|
tvix-eval, and how more annoying builtins get injected into tvix-eval through
|
||||||
|
tvix-glue.
|
||||||
|
Maybe restructure to only explain the component structure potentially
|
||||||
|
crossing process boundaries (those with gRPC), and make the rest more crate
|
||||||
|
and trait-focused?
|
||||||
|
- Restructure docs on castore vs store, this seems to be duplicated a bit and
|
||||||
|
is probably still not too clear.
|
||||||
|
- Describe store composition(s) in more detail. There's some notes on granular
|
||||||
|
fetching which probably can be repurposed.
|
||||||
|
- Absorb the rest of //tvix/website into this.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
### CLI
|
||||||
|
- `nix repl` can set variables and effectively mutates a global scope. We
|
||||||
|
should update the existing / add another repl that allows the same. We don't
|
||||||
|
want to mutate the evaluator, but should construct a new one, passing in the
|
||||||
|
root scope returned from the previous evaluation.
|
||||||
|
|
||||||
|
### Fetchers
|
||||||
|
Some more fetcher-related builtins need work:
|
||||||
|
- `fetchGit`
|
||||||
|
- `fetchTree` (hairy, seems there's no proper spec and the URL syntax seems
|
||||||
|
subject to change/underdocumented)
|
||||||
|
|
||||||
|
### Convert builtins:fetchurl to Fetches
|
||||||
|
We need to convert `builtins:fetchurl`-style calls to `builtins.derivation` to
|
||||||
|
fetches, not Derivations (tracked in `KnownPaths`).
|
||||||
|
|
||||||
|
### Derivation -> Build
|
||||||
|
While we have some support for `structuredAttrs` and `fetchClosure` (at least
|
||||||
|
enough to calculate output hashes, aka produce identical ATerm), the code
|
||||||
|
populating the `Build` struct doesn't exist it yet.
|
||||||
|
|
||||||
|
Similarly, we also don't properly populate the build environment for
|
||||||
|
`fetchClosure` yet. (Note there already is `ExportedPathInfo`, so once
|
||||||
|
`structuredAttrs` is there this should be easy.
|
||||||
|
|
||||||
|
### Builders
|
||||||
|
Once builds are proven to work with real-world builds, and the corner cases
|
||||||
|
there are ruled out, adding other types of builders might be interesting.
|
||||||
|
|
||||||
|
- bwrap
|
||||||
|
- gVisor
|
||||||
|
- Cloud Hypervisor (using similar technique as `//tvix//boot`).
|
||||||
|
|
||||||
|
Long-term, we want to extend traits and gRPC protocol to expose more telemetry,
|
||||||
|
logs etc, but this is something requiring a lot of designing.
|
||||||
|
|
||||||
|
### Store composition
|
||||||
|
- Combinators: list-by-priority, first-come-first-serve, cache
|
||||||
|
- How do describe hierarchies. URL format too one-dimensional, but we might get
|
||||||
|
quite far with a similar "substituters" concept that Nix uses, to construct
|
||||||
|
the composed stores.
|
||||||
|
### Store Config
|
||||||
|
There's already serde for some store options (bigtable uses `serde_qs`).
|
||||||
|
We might also have common options global over all backends, like chunking
|
||||||
|
parameters for chunking blobservices. Think where this would fit in.
|
||||||
|
- Rework the URL syntax for object_store. We should support the default s3/gcs
|
||||||
|
URLs at least.
|
||||||
|
|
||||||
|
### BlobService
|
||||||
|
- On the trait side, currently there's no way to distinguish reading a
|
||||||
|
known-chunk vs blob, so we might be calling `.chunks()` unnecessarily often.
|
||||||
|
At least for the `object_store` backend, this might be a problem.
|
||||||
|
- While `object_store` recently got support for `Content-Type`
|
||||||
|
(https://github.com/apache/arrow-rs/pull/5650), there's no support on the
|
||||||
|
local filesystem yet. We'd need to add support to this (through xattrs).
|
||||||
|
|
||||||
|
### DirectoryService
|
||||||
|
- Add an `object_store` variant, storing a Directory *closure* keyed by the
|
||||||
|
root `Directory` digest. This won't allow indexing intermediate Directory
|
||||||
|
nodes, but once we have `DirectoryService` composition, it shouldn't be an
|
||||||
|
issue.
|
||||||
|
- [redb](https://www.redb.org/) backend
|
||||||
|
|
||||||
|
### PathInfoService
|
||||||
|
- [redb](https://www.redb.org/) backend
|
||||||
|
- sqlite backend (different schema than the Nix one, we need the root nodes data!)
|
||||||
|
|
||||||
|
### Nix-compat
|
||||||
|
- Async NAR reader (@edef?)
|
||||||
|
|
||||||
|
### Nix Daemon protocol
|
||||||
|
- Some work ongoing on the worker operation parsing. Partially blocked on the async NAR reader.
|
||||||
|
|
||||||
|
### O11Y
|
||||||
|
- gRPC trace propagation (cl/10532)
|
||||||
|
- `tracing-tracy` (cl/10952)
|
||||||
|
- `[tracing-]indicatif` for progress/log reporting (floklis stash)
|
||||||
|
- unification into `tvix-tracing` crate, currently a lot of boilerplate
|
||||||
|
in `tvix-store` CLI entrypoint, and half of the boilerplate copied over to
|
||||||
|
`tvix-cli`.
|
||||||
Loading…
Reference in a new issue