Commit graph

36 commits

Author SHA1 Message Date
Florian Klink
066179651c refactor(tvix/store/blobsvc): move from Vec<u8> to B3Digest
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-23 10:48:55 +00:00
Florian Klink
b8ff08b1b0 refactor(tvix/store/directorysvc): move from Vec<u8> to B3Digest
This introduces a new struct, B3Digest, which internally holds a
Vec<u8>, but only allows construction with 32 bytes.

It also implements display, which will print the base64 representation.
This should reduce some boilerplate when parsing Vec<u8>.

Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-23 10:48:55 +00:00
Florian Klink
0e7bbf0d01 fix(tvix/store/proto/grpc_blobsvc): clippy
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-18 19:29:53 +00:00
Florian Klink
a6c7869393 fix(tvix/store/src/proto/mod.rs): fix error string
This was missed while renaming NixPath to StorePath.

Change-Id: Ibcc929c43b111e4370e8222c1dd86d403548367f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8577
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-17 13:06:52 +00:00
Florian Klink
e815b680c0 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directly
We currently only support querying by the output hash digest.
This makes the interface a bit simpler.

Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-16 10:39:06 +00:00
Florian Klink
616fa4476f refactor(tvix/store): remove ChunkService
Whether chunking is involved or not, is an implementation detail of each
Blobstore. Consumers of a whole blob shouldn't need to worry about that.
It currently is not visible in the gRPC interface either. It
shouldn't bleed into everything.

Let the BlobService trait provide `open_read` and `open_write` methods,
which return handles providing io::Read or io::Write, and leave the
details up to the implementation.

This means, our custom BlobReader module can go away, and all the
chunking bits in there, too.

In the future, we might still want to add more chunking-aware syncing,
but as a syncing strategy some stores can expose, not as a fundamental
protocol component.

This currently needs "SyncReadIntoAsyncRead", taken and vendored in from
https://github.com/tokio-rs/tokio/pull/5669.
It provides a AsyncRead for a sync Read, which is necessary to connect
our (sync) BlobReader interface to a GRPC server implementation.

As an alternative, we could also make the BlobReader itself async, and
let consumers of the trait (EvalIO) deal with the async-ness, but this
is less of a change for now.

In terms of vendoring, I initially tried to move our tokio crate to
these commits, but ended up in version incompatibilities, so let's
vendor it in for now.

Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-11 14:27:57 +00:00
John Ericson
26c68f8e89 refactor(nix-compat): Properly encapsulate store path construction
Before there was code scattered about (e.g. text hashing module and
derivation output computation) constructing store paths from low level
building blocks --- there was some duplication and it was easy to make
nonsense store paths.

Now, we have roughly the same "safe-ish" ways of constructing them as
C++ Nix, and only those are exposed:

- Make text hashed content-addressed store paths

- Make other content-addressed store paths

- Make input-addressed fixed output hashes

Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-04-09 15:12:21 +00:00
Florian Klink
6edc2182d1 docs(tvix/store): update comment regarding reachability check
This might be more understandable.

Change-Id: I91371d304a9a316a2934ce41929998eaef67e11c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:35:39 +00:00
Florian Klink
60751b70eb docs(tvix/store/grpc_directorysvc_wrapper): add TODO
The logic validating connectivity of Directory nodes should be moved
to SimplePutter, and this use whatever DirectoryPutter the store comes
with.

Change-Id: Id68a86a96cc49ff73920017839788859ea9c5161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8358
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:35:39 +00:00
Florian Klink
5f2c2e79e1 refactor(tvix/nix-compat): move build_store_path out of derivation
This doesn't have anything to do with ATerms, we just happen to be using
the aterm representation of a Derivation as contents.

Moving this into store_path/utils.rs makes these things much cleaner -
Have a build_store_path_from_references function, and a
build_store_path_from_fingerprint helper function that makes use of it.

build_store_path_from_references is invoked from the derivation module
which can be used to calculate the derivation path.

In the derivation module, we also invoke
build_store_path_from_fingerprint during the output path calculation.

Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-30 14:03:35 +00:00
Florian Klink
2fe53cce40 feat(tvix/store/directorysvc): add DirectoryService::get_recursive()
This moves the recursive BFS traversal of Directory closures from the
GRPCDirectoryServiceWrapper out into a a DirectoryTraverser struct
implementing Iterator.

It is then used from various implementors of DirectoryService in the
`get_recursive()` method.

This allows distinguishing between recursive requests and non-recursive
requests in the gRPC client trait implementation.

Change-Id: I50bfd4a0d9eb11832847329b78c587ec7c9dc7b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8351
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-27 09:03:34 +00:00
Florian Klink
bde10682a5 docs(tvix/store): fix typo in comment
We query the blob service for detailled blob info, not the chunk
service.

Change-Id: I85a6a57b1dae74a950f734be7d4455c5c35ae355
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8348
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 08:44:39 +00:00
Florian Klink
b025ebb2a1 refactor(tvix/store/nar): pass in &proto::node::Node
Passing in a &proto::node::Node into all this allows us consumers to
keep ownership of the proto::node::Node.

Change-Id: I44882a86c46826b06a8a8a0b24c18adfc7052662
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8316
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-17 07:49:38 +00:00
Florian Klink
985f842e32 chore(tvix/store/grpcblobsvc): clippy lint
Change-Id: Idb78e0417a962599cdfdef5e7346f7fa41e3fa1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8320
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-17 07:49:38 +00:00
Florian Klink
530cb920b5 refactor(tvix/store/chunksvc): use [u8; 32] instead of Vec<u8>
Change-Id: Ie2b94aa5d69ff2c61fb77e13ae844f81f6270273
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8314
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-16 13:47:32 +00:00
Florian Klink
ee23220564 refactor(tvix/store/directorysvc): use [u8; 32] instead of Vec<u8>
Also, simplify the trait interface, only allowing lookups of Directory
objects by their digest.

Change-Id: I6eec28a8cb0557bed9b69df8b8ff99a5e0f8fe35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8313
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-16 13:47:32 +00:00
Florian Klink
c8bbddd5e5 refactor(tvix/store): use read_all_and_chunk in gRPC blobservice
This was the last piece of code using BlobWriter.

We can also use `read_all_and_chunk`, it's just requires a bit more
plumbing:

 - The data coming from the client (stream) needs to be mapped (we
   extract the .data field).
 - The stream needs to be turned into an (async) reader
 - The reader needs to be made sync, and that code using the sync reader
   needs to be in a `task::spawn_blocking`.

Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 10:05:21 +00:00
Florian Klink
b049b88d2d refactor(tvix/store): factor out hash update into function
We're using this in a bunch of places. Let's move it into a helper
function.

Change-Id: I118fba35f6d343704520ba37280e4ca52a61da44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8251
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
ceb9d670bf refactor(tvix/store): rename C to DS, client to directory_service
This matches the semantics in other gRPC wrappers.

Change-Id: I505c901aa77a58deab216316d08542877b0b2c14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8247
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 16:15:48 +00:00
Florian Klink
ac3e470b15 feat(tvix/store): clippy
warning: the following explicit lifetimes could be elided: 'set
   --> store/src/proto/mod.rs:201:1
    |
201 | / fn update_if_lt_prev<'set, 'n>(
202 | |     prev_name: &'set mut &'n str,
203 | |     name: &'n str,
204 | | ) -> Result<(), ValidateDirectoryError> {
    | |_______________________________________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes
    = note: `#[warn(clippy::needless_lifetimes)]` on by default

Change-Id: I053d370a34e5c6721ef86502c7f225c3fbfb8746
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8244
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:23:04 +00:00
Florian Klink
a4f6c4181a feat(tvix/store): add new_temporary for all Sled services
This provides a service using /dev/shm, that's deleted once the
reference is dropped.

Refactor all tests to use these, which allows getting rid of most
TempDir usage in the tests.

The only place where we still use TempDir is in the importer tests,
which work on a filesystem path.

Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
535e1b15ab fix(tvix/store/proto/grpc_blobservice_wrapper): buffer recv data
While we don't want to keep all of the data in memory, we want to
feed a reasonably-enough buffer to the chunking function, to prevent
unnecessarily trying to chunk over and over again.

Change-Id: I5bbe2d55e8c1c63f8f7ce343889d374b528b559e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8160
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
e7279b6063 refactor(tvix/store/tests): move fixtures into separate module
Change-Id: I362dbf0899e4dc42114fd2e6a8fa7f537e9ea138
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8156
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
28a862976b refactor(tvix/store/tests): move gen_*_service() into helper
This allows hiding to tests what exact implementation we're using, when
testing things that do something with a store, but don't care what's
used for underlying storage.

Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
0baaabc43e refactor(tvix/store): move blob splitting into a BlobWriter struct
This will moves the chunking-as-we-receive logic that so far only lived
in grpc_blobservice_wrapper.rs into a generic BlobWriter.

Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
d8ab140d25 feat(tvix/store): do not buffer blob data
Use the FastCDC::cut function to ask fastcd for cutting points as we
receive the data. Make sure to keep the last chunk in the temporary
buffer, as we might not actually cut at the end.

Also, use rayon to calculate the blake3 hash if the input data is
> 128KiB.

Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
a40d2dcdcd feat(tvix/store): bump fastcdc, use v2020 version
This switches away from the less canonical "ronomon" version to the
implementation as described in the
[paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et
al., in 2020.

That version uses 64-bit hash values and tends to be faster than both
the ronomon and v2016 versions, and produces the same chunking as the
2016 version.

As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's
the recommended choice.

The crate also gained support for streaming version of chunkers:
https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be
useful.

Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
cd22a8aa6d feat(tvix/store): add tests for GRPCPathInfoServiceWrapper
Change-Id: I22de33c28c132fc9710ddecd0e9158ee3b3ef94b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8108
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
dd1ee7a2e7 feat(tvix/store): add GRPCPathInfoServiceWrapper
This exposes a proto::pathinfo_service_server::PathInfoService for a
directoryservice::PathInfoService and a way to calculate NARs.

Change-Id: I30cd058562f83d063c78b84976ec97190de49400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8097
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
cdb9458310 feat(tvix/store): add tests for GRPCDirectoryServiceWrapper
Change-Id: Iccf70664cd5dd8ea22e0b43002d2ff1a9680870a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8102
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
1540c7e276 feat(tvix/store): add GRPCDirectoryServiceWrapper
This exposes a proto::directory_service_server::DirectoryService for a
directoryservice::DirectoryService.

Change-Id: I191a6e44e5bce10d904b5f10f0073d6adaf1f572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8094
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
35ea0b0d2e feat(tvix/store): add tests for GRPCBlobServiceWrapper
Change-Id: I0c9a2d887b5b94c43afee97a8a1e701cc9f74f23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8098
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
69516f1f68 feat(tvix/store/blobservice): add GRPCBlobServiceWrapper
This takes a BlobService and ChunkService in the constructor, and
provides a [proto::blob_service_server::BlobService] trait for it.

Implementing proto::blob_service_server::BlobService is a lot of surface
to cover, and providing this wrapper will make individual
implementations taking care of how to store chunks or chunking
information much simpler.

Change-Id: Ia7b46484fb3ac9104354d496ff2922dca96ff7b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8092
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
bdf0725def refactor(tvix/store): remove anyhow::Result in directory it test
There's no `?` being used.

Change-Id: Ifbfd88f6056e0064d11458cb335d05b333ec7b6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8127
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
95aec51b2c refactor(tvix/store): remove useless clone in test
Change-Id: If62887cbc3338110cb7c815ec2ee2160fae04353
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8111
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
80f68bf828 chore(tvix/store): move protos into separate mod.rs
This allows adding more stuff into this namespace, from different files.

Also move tests on proto-related code from src/tests to src/proto/tests.

Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00