This provides an additional configuration flag to the tvix-store mount
subcommand, and logic in the fuse module to request listing for the
root of the mountpoint.
Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: adisbladis <adisbladis@gmail.com>
Tested-by: BuildkiteCI
We don't need to explicitly describe the type of the task itself,
describing the return type of the async closure is sufficient.
Also, use io::Result<_> instead of Result<_, io::Error>.
Change-Id: I9ab3f990eb49929b0aea335b2bb07da392ab631f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9267
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This trait is eval-specific, there's no point in dealing with these
things in tvix-store.
This implements the EvalIO interface for a Tvix store.
The proper place for this glue code (for now) is tvix-cli, which knows
about both tvix-store and tvix-eval.
There's one annoyance with this move: The `tvix-store import` subcommand
previously also used the TvixStoreIO implementation (because it
conveniently did what we wanted).
Some of this code had to be duplicated, mostly logic to calculate the
NAR-based output path and create the PathInfo object.
Some, but potentially more of this can be extracted into helper
functions in a shared crate, and then be used from both TvixStoreIO in
tvix-cli as well as the tvix-store CLI entrypoint.
Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Since the refactor to use URIs for all three services, this actually
does talk to a daemon by default.
Change-Id: Ied296772b77eef514bfcae0a9dfc50f848a1c2f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9210
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Some paths might use names that are not valid UTF-8. We should be able
to represent them.
We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.
While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?
Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
A previous iteration of this code did actually connect (in the gRPC
client), which was why we had this function async.
However, as the connection there is now lazy too, we can drop the
asyncness in this function.
Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This allows constructing blob stores with a URL syntax at runtime,
by passing the --blob-service-addr arg.
We probably still want to have some builder pattern here, to allow
additional schemes to be registered.
Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This removes the use of generics, like previously done with Blob and
Directory services.
Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This allows us to blob services without closing them before putting them
in a box.
We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers
require Sync.
Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Once we support configuring services at runtime, we don't know what
DirectoryService we're using at compile time.
This also means, we can't explicitly use the is_closed method from
GRPCPutter, without making it part of the DirectoryPutter itself.
Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Putting this in the PathInfoService trait makes much more sense, we can
have direct control over where/how to cache the results in the
implementation.
This now requires each PathInfoService to hold pointers to BlobService
and DirectoryService.
Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
There's only one way to calculate NAR files, by walking through them.
Things like caching such replies should be done closer to where we use
these, composing NARCalculationService doesn't actually give us much.
Instead, expose two functions, `nar::calculate_size_and_sha256` and
`nar::writer_nar`, the latter writing NAR to a writer, the former using
write_nar to only keeping the NAR size and digest.
Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
To construct various stores at runtime, we need to eliminate associated
types from the BlobService trait, and return Box<dyn …> instead of
specific types.
This also means we can't consume self in the close() method, so
everything we write to is put in an Option<>, and during the first close
we take from there.
Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
`tvix-store mount PATH` will mount the tvix-store to the given path.
Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This now creates different store client, depending on the cli
subcommand.
The `import` command will connect to the gRPC service, and the `daemon`
command will use the sled implementation.
It might make sense to define some URI syntax to make this configurable
by the user, via the CLI.
Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Instead of instantiating it once in every loop iteration, put it in an
Arc, and clone that before passing it to the spawned task.
Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This operation is blocking, so it should be run inside a blocking tokio
task. Tokio panics if it detects a blocking operation inside a non-
blocking task, so cl/8619 would cause it to panic (as the GRPC clients
use spawn_blocking under the hood).
As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614),
we create a new instance of TvixStoreIO inside each loop iteration.
Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This introduces a new struct, B3Digest, which internally holds a
Vec<u8>, but only allows construction with 32 bytes.
It also implements display, which will print the base64 representation.
This should reduce some boilerplate when parsing Vec<u8>.
Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This distinguishes it better from the EvalIO::import_path method.
Also update the docstring to explain what it does (and what it doesn't).
Change-Id: I32a8b2869fa67a894df28532b22bf170961a2abf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8578
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Whether chunking is involved or not, is an implementation detail of each
Blobstore. Consumers of a whole blob shouldn't need to worry about that.
It currently is not visible in the gRPC interface either. It
shouldn't bleed into everything.
Let the BlobService trait provide `open_read` and `open_write` methods,
which return handles providing io::Read or io::Write, and leave the
details up to the implementation.
This means, our custom BlobReader module can go away, and all the
chunking bits in there, too.
In the future, we might still want to add more chunking-aware syncing,
but as a syncing strategy some stores can expose, not as a fundamental
protocol component.
This currently needs "SyncReadIntoAsyncRead", taken and vendored in from
https://github.com/tokio-rs/tokio/pull/5669.
It provides a AsyncRead for a sync Read, which is necessary to connect
our (sync) BlobReader interface to a GRPC server implementation.
As an alternative, we could also make the BlobReader itself async, and
let consumers of the trait (EvalIO) deal with the async-ness, but this
is less of a change for now.
In terms of vendoring, I initially tried to move our tokio crate to
these commits, but ended up in version incompatibilities, so let's
vendor it in for now.
Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
After ingestion of the contents into the store, this will use the
NonCachingNARCalculationService to create a NAR stream or the contents
of the path, and use our Derivation output path calculation machinery to
determine the output path (using recursive hashing strategy).
In a real-world scenario, we obviously want to cache these calculations,
but this should be sufficient to tinker around with it.
Change-Id: I9b2e69384414f0be1bdcb5a99a4bfd46e8db9932
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8317
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
When building tvix-store without default features, this variable doesn't
need to be mutable. Silence the warning.
Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This exposes the previous default behavior at the `tvix-store daemon`
subcommand.
It also adds a `tvix-store import` command, which will ingest a given
path into the store.
Change-Id: Ide14f1d409b9364e7f98090690c744326486e470
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8166
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
