mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore(tvix/castore): move data model docs to here

Browse source 
These describe the castore data model, so it should live in the castore
crate.
Also, some minor edits to //tvix/store/docs/api.md, to honor the move of
the castore bits to tvix-castore.

Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit is contained in:
Florian Klink 2023-10-19 15:01:53 +01:00 committed by clbot
parent d545f11819
commit beae3a4bf1
3 changed files with 20 additions and 15 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
tvix/store/docs/castore.md → tvix/castore/docs/data-model.md
View file

@ -1,8 +1,8 @@
# //tvix/store/docs/castore.md
# Data model
This provides some more notes on the fields used in castore.proto.
It's meant to supplement `//tvix/store/docs/api.md`.
See `//tvix/store/docs/api.md` for the full context.
## Directory message
`Directory` messages use the blake3 hash of their canonical protobuf

0
tvix/store/docs/why-not-git-trees.md → tvix/castore/docs/why-not-git-trees.md
View file

31
tvix/store/docs/api.md
View file

@ -1,10 +1,12 @@
tvix-store API
tvix-[ca]store API
==============
This document outlines the design of the API exposed by tvix-store, as
well as other implementations of this store protocol.
This document outlines the design of the API exposed by tvix-castore and tvix-
store, as well as other implementations of this store protocol.
This document is meant to be read side-by-side with [castore.md](./castore.md) which describes the data model in more detail.
This document is meant to be read side-by-side with
[castore.md](../../tvix-castore/docs/castore.md) which describes the data model
in more detail.
The store API has four main consumers:
@ -115,8 +117,9 @@ content-addressed world to a physical path.
### PathInfo
As most paths in the Nix store currently are input-addressed [^input-addressed],
we need something mapping from an input-addressed "output path hash" to the
contents in the content- addressed world.
and the `tvix-castore` data model is also not intrinsically using NAR hashes,
we need something mapping from an input-addressed "output path hash" (or a Nix-
specific content-addressed path) to the contents in the `tvix-castore` world.
That's what `PathInfo` provides. It embeds the root node (Directory, File or
Symlink) at a given store path.
@ -215,13 +218,15 @@ This is useful for people running a Tvix-only system, or running builds on a
In a system with Nix installed, we can't simply manually "extract" things to
`/nix/store`, as Nix assumes to own all writes to this location.
In these use cases, we're probably better off exposing a tvix-store as a local
binary cache (that's what nar-bridge does).
binary cache (that's what `//tvix/nar-bridge` does).
Assuming we are in an environment where we control `/nix/store` exclusively, a
"realize to disk" would either "extract" things from the tvix-store to a
filesystem, or expose a FUSE filesystem. The latter would be particularly
interesting for remote build workloads, as build inputs can be realized on-
demand, which saves copying around a lot of never-accessed files.
"realize to disk" would either "extract" things from the `tvix-store` to a
filesystem, or expose a `FUSE`/`virtio-fs` filesystem.
The latter is already implemented, and particularly interesting for (remote)
build workloads, as build inputs can be realized on-demand, which saves copying
around a lot of never- accessed files.
In both cases, the API interactions are similar.
* The *PathInfoService* is asked for the `PathInfo` of the requested store path.
@ -253,7 +258,7 @@ As already described above, the only non-content-addressed service is the
This means, all other messages (such as `Blob` and `Directory` messages) can be
substituted from many different, untrusted sources/mirrors, which will make
plugging in additional substitution strategies like IPFS, local network
neighbors super simple.
neighbors super simple. That's also why it's living in the `tvix-castore` crate.
As for `PathInfo`, we don't specify an additional signature mechanism yet, but
carry the NAR-based signatures from Nix along.
@ -268,7 +273,7 @@ rather than a whole NAR file.
A future signature mechanism, that is only signing (parts of) the `PathInfo`
message, which only points to content-addressed data will enable verified
partial access into a store path, opening up opportunities for lazy filesystem
access, which is very useful in remote builder scenarios.
access etc.