This provides a DirectoryService implementation which uses
redb (https://github.com/cberner/redb) as the database. It provides both
in-memory and persistent on-filesystem implementations.
Change-Id: Id8f7c812e2cf401cccd1c382b19907b17a6887bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12038
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
When retrieving a closure with get_recursive, the following could happen in the GRPC client:
- The first reference to the deduplicated directory is added to expected_directory_digests
- The deduplicated directory is obtained removed from expected_directory_digests
- The second reference to the deduplicated directory is added to expected_directory_digests
- The deduplicated directory has already been sent, but is still in the
expected_directory_digests. It looks to the GRPC client like the
closure is incomplete and the stream ended prematurely.
Change-Id: Ic62bca12e7f8fb85af5fa4dacd199f0f3b8eea8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12033
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Use the ChunkedReader in CombinedBlobService instead which also supports seeking.
Change-Id: I681331a80763172c27e55362b7044fe81aaa323b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12031
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This provides a PathInfoService implementation using redb
(https://github.com/cberner/redb) as the underlying storage engine.
Both an in-memory variant, as well as a filesystem one is provided,
similar how it's done with the sled implementation.
Supersedes: https://cl.tvl.fyi/c/depot/+/11692
Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc
Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We still have the unique store name to identify which instantiation caused the error. For recursion errors, the full chain is still retained inside the CompositionError.
Change-Id: Iaddcece445a5df331e578d7c69d710db3d5f8dcd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12002
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
This allows avoiding a `.node.unwrap()`` after validation.
Change-Id: Ieef1ffebab16cdca94c979ca6831a7ab4f6007da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11989
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This will be necessary for the PathInfoService composition, as some
PathInfoService implementations require a BlobService & DirectoryService
to ingest into.
Using the Extend trait for creating compositions allows extending the same
composition with configs of various types e.g. BlobStore, DirectoryStore
Generics are moved from the Composition struct to the functions.The storage of
the InstantiatonStates uses the TypeId in the key and a Box<dyn Any> in the
value, which is downcasted to InstantiatonState<T>.
Change-Id: I2af11f26c535029adfb1c62905e0e7c4aaed7b51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11980
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
this is useful when oneshot-instantiating a store from a single config
Change-Id: I08538fdee1d0bb26b3ae2da7d3b2339b2e93bc0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11975
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
We previously called ObjectStoreBlobService::parse_url, which passes an
empty list of options when constructing the ObjectStore.
This is most likely not what we want. The more reasonable thing to do is
pass along the query string (pairs) as options to
`object_store::parse_url_opts`, and remove them from the plain URL we
pass to object_store itself.
Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Get rid of the `let grpc_client` and `let resp` in some cases.
Change-Id: Idc1c0f566a3b1b48da62e6f1977b07620656b16c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11884
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We don't need to require these things for these impl blocks yet.
Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.
The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.
This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.
Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
By default tokio::spawn does not instrument the spawned task with the
current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do
this manually for all tokio::spawn functions in functions that are
instrumented.
Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.
Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.
Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.
Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.
So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.
Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.
Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.
Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.
Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Use the new helper introduced in CL 11708 instead of rolling our own.
Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
ClosureValidator was previously only suitable for a very narrow use case:
Validating incoming uploads, which are in leaves-to-root order.
This is because the ordering validation was hard-wired into the add()
function.
This
- Re-name ClosureValidator to DirectoryGraph, which is more suitable
since it actually stores the Directory structs and is drained in the end.
- Move the ordering-related logic to a separate OrderValidator, which
can be used independently.
- re-write DirectoryGraph to be a general purpose
validator which can accept the input in both orders
and can be drained in both orders as well.
This means the DirectoryGraph and OrderValidator can now serve
multiple new purposes:
- Validating the incoming closure on the client while downloading.
- Validating the incoming closure downloaded in a caching layer from the
`far` cache, and re-order it for insertion into the `near` cache.
Change-Id: I2b4b226348416912d7a31935bec050e53d911b70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
This adds the tracing-indicatif crate, and configures it as a layer in
our tracing_subscriber pipeline to emit progress for every span that's
configured so.
It also moves from using std::io::stderr to write logs to using their
writer, to avoid clobbering output.
Progress bar styles are defined in a lazy_static, moving this into a
general tracing is left for later.
This adds some usage of this to the `imports` and `copy` commands.
The output can still be improved a bit - we should probably split each
task up into a smaller (instrumented) helper functions, so we can create
a progress bar for each task.
Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
These tests only interact with the FUSE layer, and
import super::fuse to do its work.
However, this only works if the `fuse` feature is enabled, which we
don't do if we enable the `virtiofs` feature only, causing the tests
to fail:
```
❯ cargo test --no-default-features --features virtiofs
Compiling tvix-castore v0.1.0 (/home/flokli/dev/nixos/code.tvl.fyi-submit2/tvix/castore)
error[E0432]: unresolved import `super::fuse`
--> castore/src/fs/tests.rs:14:13
|
14 | use super::{fuse::FuseDaemon, TvixStoreFs};
| ^^^^ could not find `fuse` in `super`
```
We move src/fs/tests.rs to src/fs/fuse/tests.rs
(and src/fs/fuse.rs to src/fs/fuse/mod.rs) to better structure this,
which will automatically cause both tests and code to only be built if
we have the `fuse` feature enabled.
Change-Id: I8fbbad3e4457e326bdfd171aa5c43d25d3187b5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The archive ingester has a mechanism for concurrently uploading small
blobs to the blob service in order to hide round trip latency with the
blob service when ingesting many small blobs.
Other ingestion sources like NARs also need a similar mechanism, this
extracts the concurrent blob uploading mechanism into its own struct to
make it more reusable.
Change-Id: I05020419ff4b9ad5829fbfb5cd08d36db983b8c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11693
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Using remove_node messed up the extraction of nodes from the graph. Use
into_nodes_edges() instead, to remove the nodes without cloning.
Change-Id: Id76c7935d082d6f26192cc3cd490483594f1d1e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11684
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
We can just use the `BoxStream` directly, or a `once` with the single
`Directory`.
In the recursive case, we also did not properly close the channel after
the first error.
Change-Id: Ifad56d307fc7861107b6d3cffd28d35631d526e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11635
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Necessary to directly use this in the GRPC DirectoryService wrapper
directly.
Change-Id: Ic6a0038a40dc30071d145af5035345fcd93288ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11634
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Use try_stream! rather than stream!, and a bit more map_err and ok_err
to make things a bit more concise. Once we have proper error types here,
and impl Froms, a lot of the error mapping would disappear entirely.
Change-Id: I5240a6b0ff7818b94c151322774242b2c142e33b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11633
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Replace the loop manually driving the iterator with a for … in, and some
of the match with ok_or_else.
Change-Id: I6d7b3ef1bf1c7aa128bd6adef09390b54f79479e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We don't produce these erorrs anymore, no need to provide a conversion
to it.
Change-Id: I37933e436ad15c5d90b3ac270c4ef5742980513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11614
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
We don't want to block here, and this also means there's no poisoning to
deal with.
Change-Id: Ic375571970c48beace0005ae2c012135086a4d67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This one doesn't require us to deal with poisoning, is upgradeable and
the right thing to use when locking access to data, not IO resources.
Change-Id: I78634953a73404500d28f51f1d93a87e215c8149
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This does IO, which might take a longer amount of time than what we want
to be blocking the normal executor.
Use spawn_blocking instead. I didn't add it for the constructors, as we
only call these once.
Change-Id: I96231fcff8d10abe90cafde25a099a2db6ea9414
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11617
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This never did any chunking, and sled (rightfully) performs really bad
if values get too large.
We switched the default to using the objectstore backend with the local
filesystem a while ago, no need to keep this footgun around anymore.
Change-Id: I2c12672f2ea6a22e40d0cbf9161560baddd73d4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11616
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Once we break out with the root node, there may be no more elements in
the stream.
Change-Id: I6f5fc5662095aa2b2a56bcad506d25520d9ad00c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11592
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
We got away with not properly dealing with this for the archive case,
where everything is contained inside a toplevel dir, but NARs can encode
a single file/symlink.
Properly break if the IngestionEntry path has the ROOT as parent, and
only create filling directories in the other case.
Change-Id: Ib378d0d1040de7c3fe310912a0b0488c55afee83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11590
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
