docs(tvix): Update components.md
This extends various ideas around the store and its slightly different internal model. Thanks to store composition, lazy substitution and more efficient transfer, we can also simplify some complicated logic while building. Change-Id: Ib3380af650fe06e114f54e8dc2df231f18af876b Reviewed-on: https://cl.tvl.fyi/c/depot/+/6585 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
This commit is contained in:
Florian Klink
flokli
parent
f859751c6a
commit
d8f667e01e
2 changed files with 63 additions and 31 deletions
|
|
@ -28,7 +28,7 @@ note right
|
|||
Immediately starts streaming derivations as they are instantiated across
|
||||
the dependency graph so they can be built while the evaluation is still running.
|
||||
|
||||
There are two types of build requests: One for regular "fire and forget" builds
|
||||
There are two types of build requests: One for regular "fire and forget" builds,
|
||||
and another for IFD (import from derivation).
|
||||
|
||||
These are distinct because IFD needs to be fed back into the evaluator for
|
||||
|
|
@ -42,27 +42,13 @@ loop while has more derivations
|
|||
Coord<--Store: Success response
|
||||
else Store does not have path
|
||||
Coord-->Build: Request derivation to be built
|
||||
note left
|
||||
The build request optionally includes a desired store.
|
||||
If a builder is aware of how to push to the store it will do so
|
||||
directly when the build is finished.
|
||||
|
||||
If the store is not known by the builder results will be streamed
|
||||
back to the coordinator for store addition.
|
||||
end note
|
||||
|
||||
alt Build failure
|
||||
Coord<--Build: Fail response
|
||||
note left: It's up to the coordinator whether to exit on build failure
|
||||
else Build success
|
||||
alt Known store
|
||||
Build-->Store: Push outputs to store
|
||||
Build<--Coord: Send success & pushed response
|
||||
else Unknown store
|
||||
Build<--Coord: Send success & not pushed response
|
||||
Coord<--Build: Stream build outputs
|
||||
Coord-->Store: Push outputs to store
|
||||
end
|
||||
end
|
||||
|
||||
end
|
||||
|
|
|
|||
|
|
@ -63,12 +63,14 @@ to generate configuration without any build or store involvement.
|
|||
command itself. We give it filesystem access to handle things like
|
||||
imports or `builtins.readFile`.
|
||||
|
||||
In the future, we might abstract away raw filesystem access by
|
||||
allowing the evaluator to request files from the coordinator (which
|
||||
will query the store for it). This might get messy, and the benefits
|
||||
are questionable. We might be okay with running the evaluator with
|
||||
filesystem access for now and can extend the interface if the need
|
||||
arises.
|
||||
To support IFD, the Evaluator also needs access to store paths. This
|
||||
could be implemented by having the coordinator provide an interface to retrieve
|
||||
files from a store path, or by ensuring a "realized version of the store" is
|
||||
accessible by the evaluator (this could be a FUSE filesystem, or the "real"
|
||||
/nix/store on disk.
|
||||
|
||||
We might be okay with running the evaluator with filesystem access for now and
|
||||
can extend the interface if the need arises.
|
||||
|
||||
## Builder
|
||||
|
||||
|
|
@ -95,20 +97,64 @@ dominant Linux containerisation technology, by default.
|
|||
With a well-defined builder abstraction, it's also easy to imagine
|
||||
other backends such as a Kubernetes-based one in the future.
|
||||
|
||||
The environment in which builds happen is currently very Nix-specific. We might
|
||||
want to avoid having to maintain all the intricacies of a Nix-specific
|
||||
sandboxing environment in every builder, and instead only provide a more
|
||||
generic interface, receiving build requests (and have the coordinator translate
|
||||
derivations to that format). [^1]
|
||||
|
||||
To build, the builder needs to be able to mount all build inputs into the build
|
||||
environment. For this, it needs the store to expose a filesystem interface.
|
||||
|
||||
## Store
|
||||
|
||||
*Purpose:* Store takes care of storing build results. It provides a
|
||||
unified interface to get file paths and upload new ones.
|
||||
unified interface to get store paths and upload new ones, as well as querying
|
||||
for the existence of a store path and its metadata (references, signatures, …).
|
||||
|
||||
Most likely, we will end up with multiple implementations of store, a
|
||||
few possible ones that come to mind are:
|
||||
Tvix natively uses an improved store protocol. Instead of transferring around
|
||||
NAR files, which don't provide an index and don't allow seekable access, a
|
||||
concept similar to git tree hashing is used.
|
||||
|
||||
- Local
|
||||
- SSH
|
||||
- GCP
|
||||
- S3
|
||||
- Ceph
|
||||
This allows more granular substitution, chunk reusage and parallel download of
|
||||
individual files, reducing bandwidth usage.
|
||||
As these chunks are content-addressed, it opens up the potential for
|
||||
peer-to-peer trustless substitution of most of the data, as long as we sign the
|
||||
root of the index.
|
||||
|
||||
Tvix still keeps the old-style signatures, NAR hashes and NAR size around. In
|
||||
the case of NAR hash / NAR size, this data is strictly required in some cases.
|
||||
The old-style signatures are valuable for communication with existing
|
||||
implementations.
|
||||
|
||||
Old-style binary caches (like cache.nixos.org) can still be exposed via the new
|
||||
interface, by doing on-the-fly (re)chunking/ingestion.
|
||||
|
||||
Most likely, there will be multiple implementations of store, some storing
|
||||
things locally, some exposing a "remote view".
|
||||
|
||||
A few possible ones that come to mind are:
|
||||
|
||||
- Local store
|
||||
- SFTP/ GCP / S3 / HTTP
|
||||
- NAR/NARInfo protocol: HTTP, S3
|
||||
|
||||
A remote Tvix store can be connected by simply connecting to its gRPC
|
||||
interface, possibly using SSH tunneling, but there doesn't need to be an
|
||||
additional "wire format" like the Nix `ssh(+ng)://` protocol.
|
||||
|
||||
Settling on one interface allows composition of stores, meaning it becomes
|
||||
possible to express substitution from remote caches as a proxy layer.
|
||||
|
||||
It'd also be possible to write a FUSE implementation on top of the RPC
|
||||
interface, exposing a lazily-substituting /nix/store mountpoint. Using this in
|
||||
remote build context dramatically reduces the amount of data transferred to a
|
||||
builder, as only the files really accessed during the build are substituted.
|
||||
|
||||
# Figures
|
||||
|
||||
|
||||
|
||||
[^1]: There have already been some discussions in the Nix community, to switch
|
||||
to REAPI:
|
||||
https://discourse.nixos.org/t/a-proposal-for-replacing-the-nix-worker-protocol/20926/22
|
||||
|
|
|
|||
Loading…
Reference in a new issue