This adds a function that consumes a [proto::node::Node] pointing to the
root of a (store) path, and writes the contents in NAR serialization to
the passed [std::io::Write].
We need this in various places:
- tvix-store's calculate_nar() RPC method needs to render a NAR stream
to get the nar hash, which is necessary to give things imported in
the store a "NAR-based" store path.
- communication with (remote) Nix (via daemon protocol) needs a NAR
representation.
- Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a
NAR representation.
Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This provides a `Directory.nodes()` function, returning an iterator over
all three node types in an ordered fashion.
Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Make the trait public, so consumers can use it.
Also, implement it for node::Node, so we can later use this to access
the name from all three if we don't care about the enum type.
Change-Id: Iae530a16b705493645e61947852c03273876cc55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Move nixbase32 and store_path into this.
This allows //tvix/cli to not pull in //tvix/store for now.
Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Replace our data_encoding usage with the implementation taken from
https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32
Also uncomment some of the unit tests, and add a regression test for a
NIXBASE32.encode with a 32 bytes sequence.
The previous implementation of NIXBASE32.encode is wrong in that case:
```
❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA=
0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk
❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C
00000000 b3 a2 4d e9 7a 8f db c8 35 b9 83 31 69 50 10 30 |..M.z...5..1iP.0|
00000010 b8 97 70 31 bc b5 4b 3b 3a c1 37 40 f8 46 ab 30 |..p1..K;:.7@.F.0|
00000020
```
Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
There's gonna be more tests coming, let's move them into a separate file
everything Directory-related into a separate module.
Change-Id: I78a0f263925528907a22724b028f75cce644b329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Also add a `NamedNode` trait. We'll later use this to access names from
all three individually.
Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is used for content-defined chunking.
Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This calls out to Directory::validate() for all received Directory
messages, and also makes sure the sizes we refer a Directory message as
matches the sizes that have been calculated.
Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We can omit sending Directory messages to clients that have already been
sent in the same stream.
We can also omit storing a Directory message if we already have it -
they're content-addressed anyways.
Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Import more things, and use expect_err to unpack the response.
Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This embeds the build config directly at the point where `Cargo.nix`
is imported, making it transparent to library consumers.
Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
As discussed in #tvl, this is a more common term for it.
Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This provides a function returning a string starting with the store
path prefix, the counterpart of `from_absolute_path`.
Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows constructing a NixPath from an absolute path. It pops off
the STORE_DIR prefix and the trailing slash and returns an error if it
couldn't be found.
Change-Id: Ib540e353c63cc247ac15e20414b0db2caf695ef4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7751
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Rename PATH_HASH_SIZE to DIGEST_SIZE.
It's a digest, not hash (we don't necessarily have the internal hash
state anymore), and the fact it's about (Nix)Paths is already visible
from the module name.
Also expose ENCODED_DIGEST_SIZE, so we don't need to do the calculation
inside from_string() method, and it becomes more clear this is a
constant.
Change-Id: I0e7577dd7a6e503039037b986313b214e995d826
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7725
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Tested-by: BuildkiteCI
These can be accessed directly.
Change-Id: I71dc84f982820d53f319efefbed9b9055034954d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7724
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This provides validation of PathInfo messages, and ensures the output
hashes are properly parsed from the root node names.
NixPath already has a more extensive test suite for various wrong
NixPaths, so it's omitted from here.
Change-Id: I5d69118df5816daabb521ddb19d178bddd1caacf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7684
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows other crates to import tvix_store.
Rename the bin crate to tvix-store-bin, to avoid having multiple crates
with the same name (https://github.com/rust-lang/cargo/issues/6313)
Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/
tracing.
Annotate all method handlers with an instrument macro, and warn! a
message for them being unimplemented.
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools
like evans automatically discover available services, without having to
specify the path to the .proto files client-side.
It's behind a reflection feature flag, which is enabled by default.
Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This refactors how the original digest type (Vec<u8>) is converted
to [u8; PATH_HASH_SIZE].
Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This replaces the hello world example from tvix-store with an actual
gRPC endpoint, implementing all of BlobService, DirectoryService and
PathInfoService.
All RPC methods currently respond with the unimplemented gRPC status.
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Similar to cl/7682, we also want to make that error configurable.
Change-Id: I64f1a4570b3d75af4741abe10c2855959766e107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7708
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This implements the NixPath structure. NixPath allow to parse a string to a nix path. If the parsing fails, a DecodeError will be raised.
Change-Id: I28363cdcfb27f04bf21a11c0d130b461667e3720
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This implements the nix-specific base32 encoding and decoding, exposing
a subset of the API that the data-encoding crate provides.
Nix uses a custom alphabet, no padding, and encodes bytes in reverse
order. The latter one is the reason we can't just use the data-encoding
crate directly.
Three odd corner case tests ported over from go-nix failed. We opened
b/235 to further investigate.
Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We'll be using validate_node_name in other places in a bit, where
returning a ValidateDirectoryError is not appropriate.
Accept a function mapping a string to error as a second argument, and
pass ValidateDirectoryError::InvalidName at the current call sites.
Change-Id: I45cbb0deb4763061ad912c6b18a112c727795a17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7682
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
The only warnings left are unused warnings, but that'll change once we
have a real implementation, and not just tests.
Change-Id: I28912281b5e66735be37e999cc8ef4b8b09028fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7669
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
lazy_static is only used in tests, and anyhow isn't used at all (yet).
This can be dropped.
Change-Id: Ic41ff3f9bb93cfa600c3485e85464f78a3976504
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7668
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Move them from the bottom of src/proto.rs to its own src/tests/mod.rs.
Also drop the test_ prefix, this is not golang.
Change-Id: I2e0b6b9812264f3d9721c0766936f08157fadc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7667
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Stat exposes metadata about a given blob,
such as more granular chunking, baos.
It implicitly allows checking for existence too, as asking this for a
non-existing Blob will return a Status::not_found grpc error.
The previous version returned a Status::not_found error on the Get
request too, but there was no chance to prevent the server from starting
to stream (except sending an immediate cancellation).
Being able to check whether something exists in a BlobStore helps to
prevent from uploading in first place.
The granular chunking bits are an optional optimization - if the
BlobStore implements no more granular chunking, the Stat response can
simply contain a single chunk.
Read returns a stream of BlobChunk, which is just a stream of bytes -
not necessarily using the chunking that's returned in the reply of a
Stat() call. It can be used to read blobs or chunks.
Change-Id: I4b6030ef184ace5484c84ca273b49d710433731d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7652
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This changes the RPC methods to return/consume a stream of chunks, instead of a
very big message containing the whole blob, to keep message sizes in manageable
sizes (less than 4MiB).
Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Expose the NAR calculation to a separate `CalculateNAR` method, which
responds with the NAR size and sha256 hash.
Contrary to what cl/7618 and cl/7620 initially did, don't add different
other request types.
In the CalculateNARResponse message, there's now some duplication in the
(optional) `narinfo` field of a PathInfo, but I'm not entirely sure if
we want to drop the fields from there yet.
Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
