Commit graph

314 commits

Author SHA1 Message Date
Florian Klink
4b4443240e feat(tvix/castore): add HashingReader, B3HashingReader
HashingReader wraps an existing AsyncRead, and allows querying for the
digest of all data read "through" it.
The hash function is configurable by type parameter, and we define
B3HashingReader.

Change-Id: Ic08142077566fc08836662218f5ec8c3aff80be5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11087
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:31:31 +00:00
Florian Klink
8383e9e02e feat(tvix/castore/digests): impl From digest::Output<_> for B3Digest
This allows calling .into() to get a B3Digest.

Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:18:19 +00:00
Florian Klink
7bebf492ec refactor(tvix/castore/blobsvc/chunked_reader): refactor, document
The public-consumable thing here is ChunkedReader, not ChunkedBlob.

ChunkedBlob is a helper that can be used to get a new AsyncRead, but
not AsyncSeek. It is used internally by ChunkedReader whenever the
client seeks.

Make this more obvious, by extending the documentation, and putting
ChunkedReader at the top of this file.

Also make ChunkedBlob and its methods private, and give ChunkedReader a
more useful constructor (from_chunks, instead of from_chunked_blob).

Change-Id: I2399867591df923faa73927b924e7c116ad98dc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11079
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-03 11:22:56 +00:00
Florian Klink
53fb9ff4c6 feat(tvix/castore/blobsvc): BlobReader for more trivial types
Change-Id: I80e4f26c41a504fa4c6a013c2a1e76de613ba294
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11078
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-02 17:05:23 +00:00
Florian Klink
982459d343 fix(tvix/castore/blobwriter): don't require Sync + 'static
There's no reason for these two.

Change-Id: Ie6f238bbb0b17971c9877b11b61ea7ebca573c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11075
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-02 05:56:21 +00:00
Florian Klink
b38badf206 docs(tvix/castore/directorysvc): K/V is not necessarily flat
Some implementations of DirectoryService might not allow retrieval of
intermediate Directory nodes, that are not at the "root".

Think about an object store implementation. The client is doing a
get_recursive anyways to reduce the number of roundtrips.

By documenting the fact we don't need to support looking up intermediate
Directory messages, we can just batch all directories into the same
object, keyed by the root.

Change-Id: I019d720186d03c4125cec9191e93d20586a20963
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10988
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 09:17:38 +00:00
Peter Kolloch
5777050821 feat(tvix/castore): Compile fix for Darwin
Towards https://b.tvl.fyi/issues/264

Change-Id: If8fa912ae3fb2987b761f649ab738529ebf3b2e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10970
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-19 17:14:24 +00:00
Florian Klink
28f5c13c53 fix(tvix/castore): don't emit ret as INFO
This otherwise gets a bit spammy.

Change-Id: I288350a600d79a394c239f253424ad55bc3cefc5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10954
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-18 07:12:27 +00:00
Florian Klink
34a1ff291a feat(tvix/castore/fs): make allow_other configurable
Also add a cli argument to the `tvix-store` binary.

Change-Id: Id07d7fedb60d6060543b195f3a810a46137f9ad5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10945
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-17 07:00:41 +00:00
Florian Klink
d10c5309bc feat(tvix/castore/blobsvc): add Chunked{Blob,Reader}
These provide seekable access into a Blob for which we have more
granular chunking information.

There's no support for verified streaming in here yet, this simply
produces a stream of readers for each chunk, skipping irrelevant chunks
and data from the first chunk at the beginning.

A seek simply does produce a new reader using the same process.

Change-Id: I37f76b752adce027586770475435f3990a6dee0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10731
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-10 14:24:51 +00:00
Florian Klink
40d81d0c74 docs(tvix/castore/blobstore): reorganize docs
docs/verified-streaming.md explained how CDC and verified streaming can
work together, but didn't really highlight enough how chunking in
general also helps with seeking.

In addition, a lot of the thoughts w.r.t. the BlobStore protocol, both
gRPC and Rust traits, as well as why there's no support for seeking
directly in gRPC, as well as how clients should behave w.r.t. chunked
fetching was missing, or mixed together with the verified streaming
bits.

While there is no verified streaming version yet, a chunked one is
coming soon, and documenting this a bit better is gonna make it easier
to understand, as well as provide some lookout on where this is heading.

Change-Id: Ib11b8ccf2ef82f9f3a43b36103df0ad64a9b68ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10733
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-06 18:28:00 +00:00
Florian Klink
cb2cf3f6b7 fix(tvix/castore/grpc/svc_wrapper): expose chunks() over gRPC
The Stat() method was just always signalling no granular chunks are
available. However, as we now have a .chunks() method, we can expose it
over gRPC.

Change-Id: I74f0890ae083f301bb0cec62f1ea4a95463ac590
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10736
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-02 16:27:10 +00:00
Florian Klink
9504015031 feat(tvix/castore/blobsvc): validate StatBlobResponse
All chunks must have valid blake3 digests. It is allowed to send an
empty list, if no more granular chunking is available.

Change-Id: I7ecb53579cdf40fd938bb68a85685751b4d3626f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10726
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-02-02 16:26:38 +00:00
Florian Klink
5ad5a0da00 refactor(tvix/castore/grpc/blobsvc): inline stream_mapper
This can be written without the additional function.

Change-Id: Ib11c5d5254d3e44c8fa9661414835b0622eb1ac4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10735
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-02 16:25:37 +00:00
Florian Klink
1157eea710 docs(tvix/castore/blobsvc): fix doc comments on trait
The readers implement AsyncRead/AsyncSeek, not their sync counterparts.
Also update expectations around chunks.

Change-Id: Ic266688039d80d16d33f651b96ce2bcdedecfa00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10734
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-02 16:24:06 +00:00
Florian Klink
4c5d9fa356 feat(tvix/castore/docs/verified-streaming): clarify reply
"given chunksize" is misleading here. It's up to the backend to decide
if it does chunking at all, and how it chunks.

Change-Id: I4f130ca9ac34db79f18ef1d6475295806ac7f9a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10728
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-02-02 08:56:48 +00:00
Florian Klink
459a564ff1 refactor(tvix/castore/blobsvc/combinator): compact trait bounds
BlobService already implies Send and Sync, we don't need to explicitly
list it here.

Change-Id: I58a4c5912be61a60acd961565979aa01d94ee0f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10727
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-02-02 08:55:16 +00:00
Ryan Lahfa
68bba48d59 feat(tvix/castore): process_entry cannot process unsupported nodes
In the past, we had a `todo!` on unsupported node types, this returns a proper error
that can be caught by the caller.

Change-Id: Icba4c1dab33c0d670a97f162c9b358d1ed5855cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10675
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-22 14:15:42 +00:00
Connor Brewster
4e341fb5d9 chore(tvix/store): Use BoxStream type alias
The BoxStream type alias is a more concise and easier to read than
the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type.

Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-21 19:41:02 +00:00
Ryan Lahfa
7275288f0e refactor(tvix/castore): break down ingest_path
In one function that does the heavy lifting: `ingest_entries`, and three additional helpers:

- `walk_path_for_ingestion` which perform the tree walking in a very naive way and can be replaced by the user
- `leveled_entries_to_stream` which transforms a list of a list of
  entries ordered by their depth in the tree to a stream of entries in
  the bottom to top order (Merkle-compatible order I will say in the
  future).
- `ingest_path` which calls the previous functions.

Change-Id: I724b972d3c5bffc033f03363255eae448f017cef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10573
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-20 18:26:17 +00:00
Ryan Lahfa
1f1a42b4da feat(tvix/castore): ingestion does DFS and invert it
To make use of the filtering feature, we need to revert the internal walker to a real DFS.

We will therefore just invert the whole tree by storing all of its
contents in a level-keyed vector.

This is horribly expensive in memory, this is a compromise between CPU
and memory, here is the fundamental reason for why:

When you encounter a directory, it's either a leaf or not, i.e. it
contains subdirectories or not.

To know this fact, you can:

- wait until you notice subdirectories under it, i.e. you need to store
  any intermediate nodes you see in the meantime -> memory penalty.
- getdents or readdir on it to determine *NOW* its subdirectories -> CPU
  penalty and I/O penalty.

This is an implementation of the first proposal, we pay memory.

In practice, we are paying O(#nb of nodes) in memory.

There's a smarter albeit much more complicated algorithm that pays only
O(\sum_i #siblings(p_i)) nodes where (p_1, ..., p_n) is the path to a leaf.

which means for:

             A
            / \
           B   C
          /   / \
         D   E   F

We would never store D, E, F but only E, F at a given time.
But we would still store B, C no matter what.

Change-Id: I456ed1c3f0db493e018ba1182665d84bebe29c11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10567
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 17:16:01 +00:00
sterni
526295a71d chore(3p/sources): Bump channels & overlays
- Adjust to ecl 23.9.9 release

- Regenerate go protos after protoc-gen-go update

- Drop dhall fork which hasn't kept up with 1.42.*

- Address new clippy warnings:

  - Variant naming of Error::ValidationError
  - Simplify .try_into().unwrap()
  - Drop unnecessary identity function
  - Test module must be last in file
  - Drop unused `pub use`

- Update agenix to 0.15.0. Current master has a installCheckPhase that
  doesn't work with C++ Nix 2.3.*:
  a23aa271be (commitcomment-137185861)

Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-19 21:47:32 +00:00
Ryan Lahfa
93afc711f6 feat(tvix/castore): convert import error to std::io::Error
So that we can just `map_err` easily in functions returning `std::io::Error` but calling functions
returning `castore::import::Error`.

Change-Id: Id181b95e8431c69e95f3a8cd569ca10306656e1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10572
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-18 14:40:06 +00:00
Florian Klink
c5e2832cbd feat(tvix/castore): implement Ord for node::Node
This allows assembling BTreeSets of node::Node.

Change-Id: I97b83be5ffc3e891307a8ef2b5fc31e38b747a62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10625
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-15 18:19:15 +00:00
Florian Klink
4d135bcfa2 feat(tvix/castore): implement CombinedBlobService
First attempt on composition of BlobServices.

Change-Id: I6e70248007edfd322a503fd40c1c4b4300cbc30c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10587
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-09 17:32:03 +00:00
Florian Klink
719cbad871 feat(tvix/castore/blobsvc): add chunks method
This adds support to retrieve a list of chunks for a given blob to the
BlobService interface.

While theoretically all chunk-awareness could be kept private inside
each BlobService reader, we'd not be able to resolve individual chunks
from different Blobservices - and due to this, not able to substitute
chunks we already have in a more local store.

This function allows asking a BlobService for the list of chunks,
leaving any actual fetching up to the caller (be it through individual
calls to open_read), or asking another store for it.

Change-Id: I1d33c591195ed494be3aec71a8c804743cbe0dca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10586
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-09 17:31:32 +00:00
Florian Klink
9596c5caff refactor(tvix/castore): do clone inside a scope
Make it clear this is only used inside the scope.

Change-Id: Ie94f88d7f0fb58cd4bf9c2f1176000b272e6f2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10585
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 16:20:19 +00:00
Florian Klink
9de1ebf23e feat(tvix/castore/grpc): instrument some more functions
Change-Id: Icedb148c88c5f4a3b2242ed12df1dd8692af94fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10584
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 16:18:48 +00:00
Florian Klink
0009383c07 refactor(tvix/castore/directorysvc): AsRef traverse_to
Change-Id: I641bd4ab3de591a013f03137f1e16295946315f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10579
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-09 14:15:55 +00:00
Florian Klink
b1c556b7e1 refactor(tvix/castore/blobservice/grpc): remove fn pointer hack
It looks like the workaround isn't necessary anymore.

Change-Id: Ifbcef1d631b3f369cac3db25a2c793480043f697
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10583
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:13:24 +00:00
Florian Klink
89882ff9b1 refactor(tvix): use AsRef<dyn …> instead of Deref<Target= …>
Removes some more needs for Arcs.

Change-Id: I9a9f4b81641c271de260e9ffa98313a32944d760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10578
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:08:22 +00:00
Florian Klink
8fbdf72825 feat(tvix/castore/blobsvc/grpc): rm VecDec, fix docstring
The docstrings were not updated once we made the BlobService trait async.
There's no more need to turn things into a sync reader.

Also, rearrange the stream manipulation a bit, and remove the need to
create a new VecDeque for each element in the stream. bytes::Bytes
implements the Buf trait.

Fixes b/289.

Change-Id: Id2bbedca5876b462e630c144b74cc289c3916c4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10582
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:03:21 +00:00
Ryan Lahfa
cbcd078684 chore(tvix/castore): fix the docstring for process_entry
It was a `//` not a `///`.

Change-Id: Iee3e8c116d73b5dd8a41c027153714415a66695f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-06 01:39:41 +00:00
Florian Klink
5a82736122 chore(tvix): bump test-case dep to 3.3.1
Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-01-05 16:43:34 +00:00
Florian Klink
6b42aef88d fix(tvix/castore): validate Option<Node>
Extend our validation function to also check for the None case.

Change-Id: Ib75f880646d7fb3d66588f1988e61ec18be816a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10534
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-05 16:43:34 +00:00
Florian Klink
f20969de9b refactor(tvix/castore): relax trait bounds for DS
Make this an `AsRef<dyn DirectoryService>`.

This helps dropping some Clone requirements.

Unfortunately, we can't thread this through to TvixStoreIO just yet.

Change-Id: I3f07eb28d6c793d3313fe21506ada84d5a8aa3ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-05 16:43:34 +00:00
Florian Klink
597a6b6205 refactor(tvix/castore/tests): let gen_*_service return Boxes
Only convert to and reuse an Arc<…> where needed.

Change-Id: I2c1bc69cca5a4a3ebd3bdb33d6e28e1f5fb86cb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10514
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 14:45:17 +00:00
Florian Klink
1b62f82b10 refactor(tvix/castore/blobsvc/grpc/wrapper): don't require Arc<_>
Change-Id: I9655f5588c7dc98427de6af47d74b4ab7ce22071
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10516
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 14:42:36 +00:00
Florian Klink
96aa220dcf refactor(tvix/castore/directorysvc/grpc/wrapper): no Arc<_>
We can also drop the Clone requirement. Because the trait is async since
some time, there's no need to clone before moving into an async closure,
allowing us to simplify the code a bit.

Change-Id: I9b0a0e10077d8c548d218207b908bfd92c5b8de0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10515
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-01 14:40:35 +00:00
Florian Klink
54fe97e725 refactor(tvix/castore): make directorysvc more generic
This works on Box<dyn DirectoryService> too.

Change-Id: Ib869f0f4d963ef4dbaeab22db03ff6afb71ede04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10513
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-01 02:09:20 +00:00
Florian Klink
ddae4860c2 feat(tvix/castore/import): generalize ingest_path
We don't actually care if it's an Arc<dyn BlobService>, or something
else, as long as we can Deref to a BlobService and clone.

Change-Id: I0852aaf723f51c5e6b820be8db1199d17309ab08
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10510
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-01 00:54:14 +00:00
Florian Klink
41935fab70 refactor(tvix/castore/directorysvc): return Box, not Arc
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.

Change-Id: Ia05bb370340792a048e2036be30e285ef1e63870
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10483
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:18:14 +00:00
Florian Klink
9ca1353122 refactor(tvix/castore/blobsvc): return Box, not Arc
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.

Change-Id: I7bd337cd2e4c2d4154b385461eefa62c9b78345d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10482
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:18:14 +00:00
Florian Klink
8a52c7f1c5 feat(tvix/castore/fs): borrow some matches
We only do things with the reference, so we don't need to locally borrow
it.

Change-Id: I6073f7ec7aff717ae3069e28a00b1cb408a50ceb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10455
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-29 17:18:14 +00:00
Florian Klink
46a372d5d7 feat(tvix/castore/fs): instrument FuseDaemon functions
Change-Id: I696b7ab6b4c08004db147c0fda7312bbebaa0eec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10451
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-29 15:57:26 +00:00
Florian Klink
acbb613e61 chore(tvix): switch to upstream futures 0.3.30
The bugs have been fixed,
https://github.com/rust-lang/futures-rs/pull/2801 and
https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended
up in that release.

Change-Id: Iefd990d2d1719b884504093343e54e9c5258e2e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10414
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-24 21:45:04 +00:00
Florian Klink
a5865ec7fa refactor(tvix/castore/fs/tests): drop unused args
There's no need to pass in an unused directory service into the
populate_blob_* method, and considering we have one or two invocation of
each of these, we don't really gain much from having all these functions
follow the same structure, at least for now.

Also, update some function names to better describe what they're doing.

Change-Id: I92f680745c157fb0a602b07342f8838bfad23ecd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10411
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-24 16:05:52 +00:00
Florian Klink
8d86d2f409 refactor(tvix/castore): add RootNode impl for BTreeMap, mv fs tests
cl/10378 did already move store/fs to castore/fs, but we kept the tests
in tvix-store, as they were populating a PathInfoService to make nodes
appear in the mount root.

Update these tests to now just insert root nodes into a BTreeMap, and
ensure we can use that as a RootNodes too.

Change-Id: Iad7d1ee4f9423eb6e3a1da33f433842c9ae0de1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10410
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-24 15:44:30 +00:00
Florian Klink
a5167c508c chore(tvix): move store/fs to castore/fs
With the recent introduction of the RootNodes trait, there's nothing in
the fs module pulling in tvix-store dependencies, so it can live in
tvix-castore.

This allows other crates to make use of TvixStoreFS, without having to
pull in tvix-store.

For example, a tvix-build using a fuse mountpoint at /nix/store doesn't
need a PathInfoService to hold the root nodes that should be present,
but just a list.

tvix-store now has a pathinfoservice/fs module, which contains the
necessary glue logic to implement the RootNodes trait for a
PathInfoService.

To satisfy Rust orphan rules for trait implementations, we had to add a
small wrapper struct. It's mostly hidden away by the make_fs helper
function returning a TvixStoreFs.

It can't be entirely private, as its still leaking into the concrete
type of TvixStoreFS.

tvix-store still has `fuse` and `virtiofs` features, but they now simply
enable these features in the `tvix-castore` crate they depend on.

The tests for the fuse functionality stay in tvix-store for now, as
they populate the root nodes through a PathInfoService.

Once above mentioned "list of root nodes" implementation exists, we
might want to shuffle this around one more time.

Fixes b/341.

Change-Id: I989f664827a5a361b23b34368d242d10c157c756
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-22 16:55:18 +00:00
Florian Klink
9627ef15de docs(tvix/castore/protos): remove reference
This is not gonna end up as a interlinked docstring.

Change-Id: I2b0ca106aa75bae0156c0b411da5931da60c725d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10406
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-12-21 16:44:48 +00:00
Florian Klink
783a1e314c docs(tvix/castore): fix reference
Change-Id: I00b1d56d58c4d3779b57ab0056cff1c7e6053b9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10401
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-12-21 16:43:44 +00:00
Ryan Lahfa
0ae32d45f6 feat(tvix/castore): simple filesystem blob service
The simple filesystem `BlobService` enable a user to write blob store
on an existing filesystem using a prefix-style layout in the provided root directory,
e.g. the two first bytes of the blake3 hashes are used as directories prefixes.

Change-Id: I3451a688a6f39027b9c6517d853b95a87adb3a52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10071
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-17 14:34:13 +00:00
Florian Klink
923a5737e6 refactor(tvix/castore): drop is_closed() from impl DirectoryPutter
This is only used in the gRPC version (GRPCPutter), during the test
automation.

So define it as a method there, behind #[cfg(test)], and remove from
the trait.

Change-Id: Idf170884e3a10be0e96c75d946d9c431171e5e88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10340
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 23:07:31 +00:00
Florian Klink
8b0047e277 docs(tvix/castore/directorysvc): update comment
This comment didn't make a lot of sense before.

Change-Id: Ie057a133ca4b1a099ed3c885e32316b0d87c5eb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10339
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-12-13 21:11:58 +00:00
Florian Klink
3a32963b78 docs(tvix/castore): document expectations about DirectoryService
Namely, all trait implementations should reject invalid data being fed,
and detect invalid data being returned.

b/355 tracks writing some more tests for this, to ensure we're compliant
with this.

Change-Id: I3b05752932837ce208785efb21ffc21508b4b33a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10338
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2023-12-13 19:57:45 +00:00
Florian Klink
d236b08916 docs(tvix/castore): fix docstrings
There's been some copypasta errors.

Change-Id: I8fcad6cfc951ead6c789e0dce823c798adbfcf97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10337
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-13 19:40:40 +00:00
Florian Klink
81ef26ba3f fix(tvix/castore/import): don't unwrap entry
If the path specified doesn't exist, construct a proper error instead
of panicking.

Part of b/344.

Change-Id: Id5c6a91248b0a387f3e8f138f8e686e402009e8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10330
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:11 +00:00
Florian Klink
afd09c3290 feat(tvix/castore/import): log returned errors
This will emit a log event / trace in case this function returns an
error-y type.

Change-Id: I48db6807f3e42304357c422a2b6e177cb8b95228
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10329
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:11 +00:00
Florian Klink
30d82efa77 refactor(tvix/castore/blobservice): use io::Result in trait
For all these calls, the caller has enough context about what it did, so
it should be fine to use io::Result here.

We pretty much only constructed crate::Error::StorageError before
anyways, so this conveys *more* information.

Change-Id: I5cabb3769c9c2314bab926d34dda748fda9d3ccc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10328
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-12 18:06:40 +00:00
Florian Klink
ccaf10b4a6 refactor(tvix/*store/sled): make ::new() more generic
We don't really require the Path to be a PathBuf, we don't even require
it to be a Path, we only need it to be AsRef<Path>>.

This removes some conversion in the from_addr cases, which can just
reuse `url.path()` (a `&str`).

Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 22:46:50 +00:00
Florian Klink
9748543f1c chore(tvix): drop sled compression support
It's been a while since the last sled release, and that one binds to a
pretty old version of zstd, requiring workarounds like cl/10090.

Upstream sled main branch currently has zstd halfway patched out (it's
a no-op, but the feature flag and options are still there), and it's in
that state for a year.

Rather than maintaining our own fork of sled, let's just stop using the
compression feature in sled, dropping the version pin to zstd that way,
removing the need for cl/10090.

This doesn't mean we won't reintroduce per-blob compression - but we
probably just won't let sled take care of the compression, but do it
ourselves - which is necessary for more chunked blob storage anyways.

Even though we do drop the feature flag, we still need to explicitly use
use_compression(false).

Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 22:46:50 +00:00
Florian Klink
43943e2623 refactor(tvix/*/protos): separate lint target
Break up the go-bindings derivation. Keep "protos" containing all proto
files (well, and the buf config), and use it for a check phase running
linter and formatter, as well as the existing "go-bindings" attribute

Change-Id: I52cb9d08570bb76452acb831eb711c5b6c0eacfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10239
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
459d9e106f chore(tvix/castore/protof): buf format
Change-Id: Idf11de78b0d6eca69fda34a89f2c57a00ed89ad5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10237
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 10:24:44 +00:00
Florian Klink
22a669d27e refactor(tvix/castore): address clippy
We match to destructure a single pattern.

Change-Id: I564a3510b4860e90b3315a9639effc48ee88b483
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10233
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
sterni
875bb26fc3 fix(tvix/castore): correctly flag unreachable code
Change-Id: Id09afa4b77c3c70fb5695f253f6df4aa88b61e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10113
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-24 23:36:15 +00:00
Florian Klink
4e9e4b19ef refactor(tvix/castore/blobservice): rm AsyncBufRead from BlobReader
There's no need to already require this to be buffered here.

Change-Id: Ib9a11b194e0754d87ab8d2ef0b8cb0f4edc01229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10074
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 12:54:19 +00:00
Ryan Lahfa
df63b719ac feat(tvix/castore): fix tracing instrument in MemoryBlobService
Change-Id: Iedba57e8b3e1a44f14f5baa1e981275d4b02eb56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10070
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-18 19:23:55 +00:00
Florian Klink
68e473ed6b feat(tvix/castore): impl From<std::io::Error> for Error
Make it less annoying to convert from io::Error to this. We already have
one direction, doesn't hurt to have the other too.

Change-Id: I9fe2c6da608c9d54910ee8c397572aadb1d90d99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10068
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-18 19:23:47 +00:00
Florian Klink
ebfe456251 refactor(tvix/castore/tonic): use match in channel_from_url
Having random if blocks and returning from them is error-prone.

Also, turns out we only need the unprefixed scheme in the fallback case,
so move it down to there.

Change-Id: Ifcb09279c963f8a39e0dbabe145990263f3d7cf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10041
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-17 14:01:26 +00:00
Florian Klink
4e5165d0a1 refactor(tvix/castore/utils): drop unused DuplexStreamWrapper
This wasn't used at all, let's remove it.

Change-Id: I426e3d93c32ebe65247ae5cf8d05b5bf686be2d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10044
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-11-15 22:04:04 +00:00
Florian Klink
30e0c32066 refactor(tvix/castore/tonic): make async, support wait-connect=?
This moves the sync `channel::from_url` to a async
`tonic::channel_from_url`. It now allows connecting non-lazily if `wait-
connect=1` is set in the URL params.

Also, make the pingpong tests for blobsvc and directorysvc use the wait-
connect=1 codepath.

Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
c83841d3a1 refactor(tvix/castore): remove DirectoryService::from_url
Make directoryservice::from_addr use the more specific constructors.

Change-Id: I9fee2afed77692505988d631d9fe246d9843d25a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10029
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
840bee1e97 refactor(tvix/castore/blobsvc): remove BlobService::from_url
Make blobservice::from_addr use the more specific constructors.

Change-Id: Id9637e279d6910ce6d92ff0086a984be5c65a8c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10028
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
362117fbf5 feat(tvix/castore/src/channel): move from_url tests
These gRPC PathInfoService tests were actually not too useful in here,
what we're mostly testing is the channel construction, so move it to
there.

Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-11-15 06:43:31 +00:00
edef
a8e7f4eadb fix(tvix/castore): B3Digest::{to_vec -> as_slice}
Not a single call site actually makes use of the Vec.

Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-07 12:35:41 +00:00
Florian Klink
67999f0dcf feat(tvix/castore): extend blobstore protos for verified streaming
This pdates the proto docstrings a bit, especially w.r.t. verified
streaming.
It also adds send_chunks, send_bao fields to StatBlobRequest (renamed
from BlobMeta)

Change-Id: I590cc8646d86b73bca9f38a9b6d9ea15e4df5cb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9951
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 15:13:52 +00:00
Florian Klink
2546446d51 feat(tvix/castore): bump [Directory,File]Node size to u64
Having more than 4GiB files is quite possible (think about the NixOS
graphical installer, and an uncompressed iso of it).

No wire format changes.

Change-Id: Ia78a07e4c554e91b93c5b9f8533266e4bd7f22b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9950
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-05 10:57:01 +00:00
Florian Klink
5a0ae8312c docs(tvix/castore): add notes on verified streaming
These are still a bit WIP, but should already explain the idea better
than not having any documentation about how we intend to implement this
at all.

Let's put this into the repo, as a working document.

Co-Authored-By: edef <edef@edef.eu>
Change-Id: Ic0c5195cdf0a33c8382af1b5cff9064fe6ef9ffb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9894
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-02 09:08:20 +00:00
Florian Klink
beae3a4bf1 chore(tvix/castore): move data model docs to here
These describe the castore data model, so it should live in the castore
crate.
Also, some minor edits to //tvix/store/docs/api.md, to honor the move of
the castore bits to tvix-castore.

Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-02 09:08:20 +00:00
edef
621739037f feat(tvix/castore): carry name in ValidateNodeError::InvalidName
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 13:56:51 +00:00
edef
e525272019 refactor(tvix): turn nullary enum variants into unit variants
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
6a0a75c8e1 refactor(tvix): condense long bytestrings
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 07:37:26 +00:00
Florian Klink
1bb7e84241 docs(tvix/castore/directorysvc): document drop without put and close
As seen in cl/9761, we only use put() and close() in some cases.

Make sure this is documented, so we don't end up having implementations
not able to deal with that.

Change-Id: I406ad23f2f4c9604eca731d898ae41a4663fb846
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9786
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-17 21:12:43 +00:00
Florian Klink
6e8fbc830a chore(tvix/castore-go): rename go module
`code.tvl.fyi/tvix/castore/protos` now points to a directory that only
contains the `.proto` files, while all golang tooling and .pb.go files
live in tvix/castore-go.

As discussed in
https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount
of people currently using this is still small, so rename the go.mod now,
while it doesn't yet hurt.

Change-Id: Ib3c6a2dac2923b3806ebb05be00af66d0da9f698
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9791
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-17 20:25:45 +00:00
Florian Klink
e38733a955 chore(tvix): move castore golang bindings to tvix/castore-go
Have `tvix/castore/protos` only contain the protos, no go noise.

Make the `.pb.go` file generation a pure Nix build
at `//tvix/castore/protos:go-bindings`, and have a script at
`//tvix:castore-go-generate` (TBD) that copies the results to
`tvix/castore-go`.

`//tvix:castore-go`, with sources in `tvix/castore-go` now contains the
tooling around the generated bindings, and the generated bindings
themselves (So go mod replace workflows still work).

An additional CI step is added from there to ensure idempotenty of
the .pb.go files.

The code.tvl.fyi webserver config is updated to the new source code
path. I'm still unsure if we want to also update the go.mod name. While
being a backwards-incompatible change, it'll probbaly make it easier
where to find these files, and the amount of external consumers is still
low enough.

Part of b/323.

Change-Id: I2edadd118c22ec08e57c693f6cc2ef3261c62489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-17 19:51:24 +00:00
Connor Brewster
0325ae3ba3 fix(tvix/castore): Fix race when ingesting into castore
After finishing the ingestion, the directory putter was not being
closed. This caused a race where the root directory node was accessed
before the directory node had been flushed to the server.

This patch makes it so we close the putter before returning the root
node which should ensure that the root node exists on the directory
service server before the `ingest_path` function returns.

Fixes b/326

Change-Id: Id16cf46bc48962121dde76d3c9c23a845d87d0f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9761
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-17 13:01:29 +00:00
Florian Klink
1d97b4230f refactor(tvix/castore): clippy
Change-Id: I8a792d6b238190a78390c5d8f638669a2681f70f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9746
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-16 11:13:33 +00:00
Florian Klink
9757bf6377 refactor(tvix/*store): helper for channel creation from url
This moves the repetitive code to parse a URL and create a channel
connected to it into `tvix_castore::channel::from_url`.

Part of b/308

Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 12:26:40 +00:00
Florian Klink
3f011d2762 refactor(tvix/*store): reorganize from_url
Move the channel creation depending on the string-based URL into its
own block.

Change-Id: I546b769acd2296b548eb966b62c495f910266df5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-14 12:26:40 +00:00
Florian Klink
199e5e0339 refactor(tvix/*store): remove some grpc_client let bindings
We had to have these all while the traits where sync, and there was a
lot of spawning and moving.

Most of this can now be removed in favor of some inline `.clone()`.

Change-Id: Id5466c32a403100bc3347866b3172e06a792e311
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 12:26:40 +00:00
Florian Klink
0b18be3b57 feat(tvix/castore/protos): add more granular validation methods
Similar to cl/9715, this makes the validation checks more granular,
introducing a Validate on all *Node.

A check for symlink targets is added too.

Once merged, it can also be used from tvix/store/protos.

Change-Id: I0909a89fadcd74b74ef0c9a8a1f22658fccc83b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9716
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-14 12:21:18 +00:00
Florian Klink
532f414da6 refactor(tvix/castore): simplify node validation checks
We can just check the digest length to be correct, and move the symlink
target checks to a single line.

Change-Id: I41d2e3a50e7990ef6c04f02acd754b1e17b43e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9717
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 12:09:10 +00:00
Linus Heckemann
b1ab8075cd docs(tvix/castore): point out use of contents_first
Change-Id: I7620d2abe01675ea7028a478d4f8447e36d5768b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9605
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-13 17:14:48 +00:00
Florian Klink
9b7629826f refactor(tvix/castore): factor out node checks
Implement `validate()` on `node::Node`, and call it from PathInfo's
validate() too. Node-related errors are moved to a ValidateNodeError
error type.

This additionally adds some more validations for symlink targets (they
must not be empty, and not contain null bytes).

Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-12 21:05:48 +00:00
Florian Klink
04c1f151f3 refactor(tvix/castore/utils): clippy
Change-Id: Ib0fa1e1fb90d7f315f943a3f28266c0fb3714a31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9708
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-12 20:46:24 +00:00
edef
930edb69d7 fix(tvix/castore): use bstr for formatting names in errors
Much friendlier than either Base64 or raw byte slices.

Change-Id: I9b4cdd57c83ddc76c0be8103da4320207657a72b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9622
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 22:40:55 +00:00
edef
baae5ce473 fix(tvix/castore): handle Directory::size overflow explicitly
We use checked arithmetic for computing the total size, and verify
that size is in-bounds in Directory::validate.

If an out-of-bounds size makes it to the "unchecked" size method,
we either panic (in debug mode), or silently saturate to u32::MAX.

No new panic sites are added, since overflows in debug mode already
panic at the language level.

Change-Id: I95b8c066a42614fa447f08b4f8fe74e16fbe8bf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9616
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 20:33:52 +00:00
edef
e2dba089c4 test(tvix/castore): demonstrate that Directory::size may panic
Change-Id: Iebfa12df51c8c9d39b4d9eb2fccc6ffa312639cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9620
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 20:33:52 +00:00
Florian Klink
d1bccfe82f chore(tvix/*store): use google.golang.org/protobuf
The other one is deprecated, according to
https://pkg.go.dev/github.com/golang/protobuf/proto.

Change-Id: I502d5bfa7431df0dea3d564f1e0ecfea14c90e10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9597
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:07 +00:00
Florian Klink
abf9c05090 feat(tvix/castore): add RenamedNode helper
This takes a castorev1pb.Node and updates its name, making sure to copy
it.

Change-Id: I57bf2b2b4a74496a1a697bcbce72bf67a12e69fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9594
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:05 +00:00
Florian Klink
dba62157e5 test(tvix/castore): add uds pingpong test for directorysvc
This was missing so far.

Change-Id: Ic8c7781aa0e101342e8a7d94f206937b4163ebfa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9570
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 22:55:10 +00:00
Florian Klink
b6bf3a87f1 test(tvix/castore): use tokio-retry for exp backoff
Rather than using this loop, use exponential backoff while waiting for
the socket path to be created.

Change-Id: I18706a64ce06f8916a07892dfbcd409ac5b3bff1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9568
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 22:50:42 +00:00
Florian Klink
b196cbbc67 refactor(tvix/castore): simplify test_valid_unix_path_ping_pong
We don't need to spawn two tokio runtimes anymore, and can do the URL
parsing at once, too.

Change-Id: I38ab96978cb7f8c31ded2726262e0b1366655094
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9566
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 22:47:34 +00:00
Florian Klink
c63c55f052 chore(tvix): bump tokio[-util], tonic, prost
This bumps all these dependencies to their more recent versions.

We have to enable the `macros` tonic feature, and looks like we should
also enable `rt`, not only `rt-multi-thread`.

Change-Id: Icc3600848fca0bacce24f4889d088e75711594ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8984
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 21:53:20 +00:00
Florian Klink
c847cc32d9 refactor(tvix/castore): move tests to grpc client, rm tonic-mock
Similar to gen_directorysvc_grpc_client, introduce a
gen_blobsvc_grpc_client function that provides a gRPC client connected
to a blobservice.

The test is update to use that client to test against, rather than the
server trait, removing the last usage of tonic_mock, so it's removed
as well.

Fixes b/243.

Change-Id: If746e8600588da247eb53a63b70fe72f139e9e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9564
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 21:53:20 +00:00
Florian Klink
1629f3064f docs(tvix/castore): remove TODO
This probably was about passing around directory_putter at some point,
which we do, so whatever this meant, it's not actionable anymore.

Change-Id: I1b4e0cdd2119bf2b2a9cf06d186a3b476b0ff367
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9573
Reviewed-by: Linus Heckemann <git@sphalerite.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-08 16:00:44 +00:00
Florian Klink
b172c804b0 refactor(tvix/castore): use DuplexStream instead of unix socket
We can use DuplexStream to create to bidirectional pairs, which avoids
manually waiting for unix sockets to pop up and connect, and creating
temporary directoires to create the unix sockets in.

Turns out, we also don't actually need to spawn the server in a separate
runtime, it works just fine these days. This might be due to all the
sync barriers in between being gone.

Change-Id: I6b79823bc6209cbcb343b7a498c64a2ba6e0aee7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9562
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 11:25:23 +00:00
Florian Klink
3b620e6d0c refactor(tvix/castore): move gRPC directorysvc tests off tonic_mock
The tests were using the gRPC Service/Server interface, not the gRPC
clients.

Use our new gen_directorysvc_grpc_client() function to spin up a server
in a separate thread, and test via that client, rather than *just* the
server implementation.

This is more correct, because we're now really exercising the gRPC
stack, and some of the stream implementations are also only implemented
on the client.

Change-Id: I99434a2399856a44c9e6988d0b720f7a429d0ad1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9560
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 07:44:13 +00:00
Florian Klink
a77914db73 refactor(tvix/castore/directorysvc): factor out gRPC client gen
Move this code into a helper function, which we'll use in other places
in a bit.

Change-Id: Icae6f6dd2d4b2fa86fd2b836ddd7a4ca0e0354e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9559
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 07:30:34 +00:00
Florian Klink
c67ab911eb refactor(tvix/castore): move magic number to B3_LEN const
… and export it.

Change-Id: I47d2dc2f5a8174da65c614b43801d648506e2d73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9544
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 09:50:31 +00:00
edef
9a7c078a69 fix(tvix/castore): explicitly name lifetimes in process_entry
Otherwise this produces absolutely inscrutable errors:

    note: hidden type `[async fn body@castore/src/import.rs:63:1: 63:94]` captures lifetime '_#24r

Change-Id: If5d9626c9edf400de5bcec038bcaa5a3117561f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9506
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-04 08:31:40 +00:00
Florian Klink
6b02800e48 chore(tvix/[ca]store): allow building without tonic-reflection
- rename the feature to `tonic-reflection` in both tvix-store and tvix-
   castore.
 - set it to disabled in tvix-castore by default
 - enable it in tvix-store, and pull in tvix-castore with the feature
   enabled.

Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 10:07:40 +00:00
Florian Klink
80a02f96ce chore(tvix/castore): remove dependency on anyhow
We don't use this here, except in a test, where we don't really use
it either.

Change-Id: Ia6c45fccf663fe328942b1e1a2cd1c3a1b7f9ae5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9376
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-22 12:51:21 +00:00
Florian Klink
32f41458c0 refactor(tvix): move castore into tvix-castore crate
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.

Notable changes:
 - `fixtures` and `utils` had to be moved out of the `test` cfg, so they
   can be imported from tvix-store.
 - Some ad-hoc fixtures in the test were moved to proper fixtures in the
   same step.
 - The protos are now created by a (more static) recipe in the protos/
   directory.

The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)

Bug: https://b.tvl.fyi/issues/301

Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:51:21 +00:00