Commit graph

270 commits

Author SHA1 Message Date
Florian Klink
f5e291cf83 feat(tvix/store/fuse): implement open explicitly
This moves from stateless I/O to actually dealing with file handles,
allowing the filesystem to keep reusing existing blobreaders, instead of
opening a new reader on every read() call.

Change-Id: I3fc35c071e4aee1021c8bbd58749d082b0abd188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8834
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 11:59:48 +00:00
Florian Klink
b25d98a84e feat(tvix/store/fuse): initial implementation
This is a first implementation of a FUSE filesystem, mounting tvix-store
to a given location.

This is mostly meant as one additional lens into a store, and could be
used for builds. It's not meant to be used as a general-purpose thing.

It still has some rough edges:

 - It doesn't implement open/close, so it doesn't use file handles.
   Which means, we need to open blobs for partial reads over and over
   again.
 - It doesn't implement seek, as BlobReader doesn't implement seek yet.
 - It doesn't track "lifetimes" of inodes by listening on forget,
   meaning it might hold more data in memory than necessary.
 - As we don't have store composition (and a caching layer) yet,
   operations might be slow.

Change-Id: Ib1812ed761dfaf6aeb548443ae939c87530b7be8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8667
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-20 10:14:00 +00:00
Florian Klink
6656b865b6 docs(tvix/store): correct some cargo doc warnings
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-19 22:54:46 +00:00
Florian Klink
ad687ccaf4 chore(tvix/store/pathinfosvc): clippy
Change-Id: Ied4bed08e989791f832922da8776d2104035e28a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8812
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-19 14:29:05 +00:00
Florian Klink
b10f008b03 chore(tvix/store/directorysvc): clippy
Change-Id: Idf45aaa0f6211ac35a9a41d0f3f60dfbe1009398
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8811
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-19 14:29:05 +00:00
Florian Klink
71093a513a chore(tvix/store/blobsvc): clippy
Change-Id: Ie384bdd27e1e9282ceda83edc74ffaad387f352b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8810
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-19 14:29:04 +00:00
Florian Klink
f087e07ca1 fix(tvix/store/blobsvc): drop stray println! debug statement
Change-Id: Ica073820ea8240d42ce8b979309f881af18a8cde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8781
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-14 23:16:09 +00:00
Florian Klink
c5851fb815 refactor(tvix/store/blobsvc): make from_addr not async
A previous iteration of this code did actually connect (in the gRPC
client), which was why we had this function async.

However, as the connection there is now lazy too, we can drop the
asyncness in this function.

Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 23:16:09 +00:00
Florian Klink
35bff2bda6 refactor(tvix/store/pathinfosvc): add from_addr
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-14 23:15:08 +00:00
Florian Klink
bb7c76739a feat(tvix/store/directorysvc): add from_addr
Add --directory-service-addr arg to tvix-store CLI.

Change-Id: Iea1e6f08f27f7157b21ccf397297c68358bd78a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8743
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 23:15:07 +00:00
Florian Klink
a1acb5bcb3 feat(tvix/store/blobsvc): add from_addr
This allows constructing blob stores with a URL syntax at runtime,
by passing the --blob-service-addr arg.

We probably still want to have some builder pattern here, to allow
additional schemes to be registered.

Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-14 20:44:36 +00:00
Florian Klink
c947a73b8e chore(tvix/store/pathinfosvc): remove GRPCPathInfoService::new
There's very little reason to instantiate a GRPCPathInfoService in a
context where we are not already in a tokio context.

Change-Id: Ib81d649387717cb98de8a8039f92472f727b10c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8755
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 21:33:59 +00:00
Florian Klink
011fbe2214 chore(tvix/store/directorysvc): remove GRPCDirectoryService::new
The only place where we did use new was also already where we've been in
a tokio context, so just using from_client is easier.

Change-Id: I39dbc18f6aaa3abc342409be623395647f968530
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8754
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 21:33:59 +00:00
Florian Klink
37e7742ba9 chore(tvix/store/blobsvc): remove GRPCBlobService::new
There's very little reason to instantiate a GRPCBlobService in a context
where we are not already in a tokio context.

Change-Id: Ic6e18809a9f2a76f1c098ed330118d8dcfba5137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8753
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-12 21:33:59 +00:00
Florian Klink
b5e37869e6 refactor(tvix/store/pathinfosvc): use Arc<dyn …>
This removes the use of generics, like previously done with Blob and
Directory services.

Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 13:27:05 +00:00
Florian Klink
64a4f6185c refactor(tvix/store/blobsvc): drop Result<_,_> around open_write
We never returned Err here anyways, and we can still return an error
during the first (or subsequent) write(s).

Change-Id: I4b4cd3d35f6ea008e9ffe2f7b71bfc9187309e2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8750
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 12:43:34 +00:00
Florian Klink
b49f7cfec6 docs(tvix/store/blobservice): fix docstrings
These are outdated.

Change-Id: I1a1ae130a55847f57a48d5e244e7e029c1ecae7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8749
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 12:43:34 +00:00
Ryan Lahfa
5de04dbc12 feat(tvix/store): increase blob chunk size
From 64 bytes to 100 KBytes.
We need to provide a custom wrapper with a different Default instance.

Change-Id: Id7c6c437b8183b355a9e388f98cef1622b363f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-06-12 12:36:56 +00:00
Florian Klink
09c5ca0a0d refactor(tvix/store/fuse): use Arc<dyn …> instead of generics
Change-Id: I5685379bd6f89d17da6843d31bef4c1fc4dc0a18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8745
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-12 10:28:59 +00:00
Florian Klink
38a7caaada refactor(tvix/store/blobsvc/sled): cargo clippy
Change-Id: I00b8b567509d5e0847270f36dadb5dcb534b9b73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8739
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-12 10:28:59 +00:00
Florian Klink
aa7bdc1199 refactor(tvix/store): use Arc instead of Box
This allows us to blob services without closing them before putting them
in a box.
We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers
require Sync.

Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-12 10:24:24 +00:00
Florian Klink
7725eb53ad refactor(tvix/store): use Box<dyn DirectoryService>
Once we support configuring services at runtime, we don't know what
DirectoryService we're using at compile time.

This also means, we can't explicitly use the is_closed method from
GRPCPutter, without making it part of the DirectoryPutter itself.

Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-12 10:24:24 +00:00
Florian Klink
6f85dbfc06 feat(tvix/store/pathinfosvc): add calculate_nar method
Putting this in the PathInfoService trait makes much more sense, we can
have direct control over where/how to cache the results in the
implementation.

This now requires each PathInfoService to hold pointers to BlobService
and DirectoryService.

Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 10:15:20 +00:00
Florian Klink
8d05c0ceaa refactor(tvix/src/nar): drop NARCalculationService
There's only one way to calculate NAR files, by walking through them.

Things like caching such replies should be done closer to where we use
these, composing NARCalculationService doesn't actually give us much.

Instead, expose two functions, `nar::calculate_size_and_sha256` and
`nar::writer_nar`, the latter writing NAR to a writer, the former using
write_nar to only keeping the NAR size and digest.

Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-12 10:15:20 +00:00
Florian Klink
27ff98000b feat(tvix/store): eliminate generics in BlobStore
To construct various stores at runtime, we need to eliminate associated
types from the BlobService trait, and return Box<dyn …> instead of
specific types.

This also means we can't consume self in the close() method, so
everything we write to is put in an Option<>, and during the first close
we take from there.

Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 10:15:20 +00:00
Florian Klink
b3ca1a78eb feat(tvix/store): add mount command to entrypoint and fuse mod
`tvix-store mount PATH` will mount the tvix-store to the given path.

Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-10 15:33:07 +00:00
Florian Klink
83c1546b9a chore(tvix/store): add fuse feature
This brings in fuse (via the `fuser` crate), and adds pkg-config and
libfuse to the dev shell, so `cargo build` can link against it.

Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-10 14:48:48 +00:00
Florian Klink
d925ec34cd docs(src/tests/nar_renderer): fix comment
This testcase tests a missing blob fails the rendering, the comment has
been copied from elsewhere.

Change-Id: I48fa3fa454e12506590fa14a3591d156bafa8b5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8722
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-09 15:33:14 +00:00
Florian Klink
485fd26334 docs(tvix/store): drop nar-bridge references
The CLs did bitrot a bit, they're based on an older version of the
protocol, and it's unclear if they'll be a separate Go Binary, or just
another HTTP handler inside tvix-store itself, considering we now have
way more NAR juggling code than before.

Change-Id: I3632035cda8d75a8ff23b3132312f0f086d9e02f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8732
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-09 15:33:14 +00:00
Florian Klink
7f5372327c docs(tvix/store): update invocation in README
This has moved to a `daemon` subcommand.

Change-Id: Iae9778d8a59e6bf84555119fabfd62db3917bb62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8731
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-09 15:33:14 +00:00
Florian Klink
b78a9b7fe1 docs(tvix/store/docs): fix grammar
Change-Id: I1d7d306e5d997a7ba47a83d613edc19a4fba1ed4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8721
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-09 09:10:51 +00:00
Florian Klink
2b0b4cadbd feat(tvix/store/bin): use sled for daemon, grpc for import cmd
This now creates different store client, depending on the cli
subcommand.
The `import` command will connect to the gRPC service, and the `daemon`
command will use the sled implementation.

It might make sense to define some URI syntax to make this configurable
by the user, via the CLI.

Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 18:10:29 +00:00
Florian Klink
48b66a8982 feat(tvix/store/bin/import): process all path imports concurrently
Change-Id: I3e1428a4725fc2e552e8f37bc0550121117fcef6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8633
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 10:13:07 +00:00
Florian Klink
dbff1289b8 refactor(tvix/store): drop Clone from BS and DS trait bound
We don't need to be able to clone these services in here.

Change-Id: Ifb69450f7ebdc8364cbf9cdfb6464f8560440e4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8645
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-30 10:13:07 +00:00
Florian Klink
a4d0a80466 refactor(tvix/store/bin): instantiating TvixStoreIO once
Instead of instantiating it once in every loop iteration, put it in an
Arc, and clone that before passing it to the spawned task.

Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 12:35:59 +00:00
Florian Klink
d25962b9a4 refactor(tvix/eval): stop borrowing &mut self
This does undo cl/8571.

Change-Id: Ib14b4e7404f906e346304b6113860ae811afc94a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8631
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 11:11:59 +00:00
Florian Klink
fcfbcf9cfa refactor(tvix/store): drop mut self borrow in ingest_path
With traverse_to not requiring a &mut anymore, we can drop the &mut self
in all these function signatures.

Change-Id: I22105376b625cb281c39e92d3206df8a6ce97a88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 11:11:58 +00:00
Florian Klink
ca06f7061c refactor(tvix/store/directorysvc): don't borrow self mutable
Change-Id: I97f183e1ef3b1209a8f05e05e152f70d1f7a9596
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8628
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 10:48:52 +00:00
Florian Klink
dbb4d5e5b2 refactor(tvix/store/bin): use std::io
Change-Id: I3f3dc9732d90790d92268c04c75eccbe92e7e05b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8634
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-25 10:48:51 +00:00
Florian Klink
92b6d15da3 fix(tvix/store/bin): use spawn_blocking to call import_path
This operation is blocking, so it should be run inside a blocking tokio
task. Tokio panics if it detects a blocking operation inside a non-
blocking task, so cl/8619 would cause it to panic (as the GRPC clients
use spawn_blocking under the hood).

As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614),
we create a new instance of TvixStoreIO inside each loop iteration.

Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 05:06:24 +00:00
Florian Klink
ad6b6b14db feat(tvix/store/pathinfosvc): derive Clone
This allows closing a TvixStoreIO if it contains a SledPathInfoService.

Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-24 16:04:08 +00:00
Florian Klink
5774117d5e feat(tvix/store): implement TvixStoreIO
This providesEvalIO, asking given PathInfoService, DirectoryService
and BlobService.

Change-Id: I32f210f5a7aa8173ad9a7d53e8a5ac03619f527a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8561
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 09:25:02 +00:00
Florian Klink
7bd9df0133 feat(tvix/store/pathinfosvc): GRPCPathInfoService::from_client
Change-Id: I4a4f6f713dd76bf95f393c094c0eb9b6d15c5436
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8613
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:37 +00:00
Florian Klink
8d392221c6 feat(tvix/store/directorysvc): GRPCDirectoryService::from_client
Change-Id: Ifa274a380683b01f0f24cd9ff1f50d22f13b38bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8612
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:36 +00:00
Florian Klink
f4a7b325e9 feat(tvix/store/blobsvc): add constructors
Change-Id: I55e06bf4e8a11dc2caf92c597558f1b820b42566
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8610
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:36 +00:00
Florian Klink
2d214eacc5 fix(tvix/store/directorysvc): stop renaming spans
We have code.namespace as a field, set to
tvix_store::directoryservice::sled, so there's no need to repeat the
name here.

Change-Id: Ic1aa8a2b24de439c6a189966bd773e9acf49d1e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8623
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 09:01:35 +00:00
Florian Klink
457fd4c325 feat(tvix/store/nar): add GRPCNARCalculationService
This asks a remote tvix-store for the nar size and digest of a given
root node.

Change-Id: If9f916d9bfc5f8dc3166e2c6c1671c0f0124d1c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8611
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-24 09:01:35 +00:00
Florian Klink
79bbf25cb2 refactor(tvix/store/nar): clippy
Change-Id: I0d864fd1b9248fe4d5c88e3d8fa396ea759fea09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8616
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
4dd954f8f5 refactor(tvix/store/directorysvc): clippy
Change-Id: I30f7e98fe79ba22d218b4aabaef88c84b6085d82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8617
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
e51d85e078 refactor(tvix/store/digests): clippy
useless conversion to the same type: `std::vec::Vec<u8>`

Change-Id: Idcb16679bb2a3350784965d8d9bbb593c760634e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8615
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
066179651c refactor(tvix/store/blobsvc): move from Vec<u8> to B3Digest
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-23 10:48:55 +00:00
Florian Klink
b8ff08b1b0 refactor(tvix/store/directorysvc): move from Vec<u8> to B3Digest
This introduces a new struct, B3Digest, which internally holds a
Vec<u8>, but only allows construction with 32 bytes.

It also implements display, which will print the base64 representation.
This should reduce some boilerplate when parsing Vec<u8>.

Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-23 10:48:55 +00:00
Florian Klink
e31008db14 fix(tvix/store): fix timing sensitivity in gRPC directorysvc test
One test in here assumed the server was fast, but when very busy, a
computer running these tests might not be fast.

Treat both cases that can occur separately.

Change-Id: Iba168ad39f2458c4fb8873211df33beeaff7c6c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8595
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-18 19:38:54 +00:00
Florian Klink
103d2eb909 refactor(tvix/store): bump CalculateNARResponse.nar_size to u64
Change-Id: I5f41981820363a5eb9982c3d19830916207c62cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8591
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
0e7bbf0d01 fix(tvix/store/proto/grpc_blobsvc): clippy
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-18 19:29:53 +00:00
Florian Klink
a0801ae220 refactor(tvix/store/blobsvc/sled): clippy
Change-Id: Icfea8ceb6b827a645c6f10efae741f6fa2114fad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8590
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
2e09e94aad refactor(tvix/store/directorysvc/traverse): clippy, use NamedNode
Also, get rid of the explicit byte comparison here, but unwrap like in
the rest of the codebase. We'll deal with this in a generic manner in
b/267 and b/189.

Change-Id: Ie5f3d27ab35b7e88d67a2796c29cdd7bc7df71f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8589
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
a1324513ad feat(tvix/store/directorysvc): add traverse_to
This walks from a node further down until it reaches the requested path.

Change-Id: I2f9a15a8601db4d06c95d7b47cd6153264e203e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8568
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-18 06:26:55 +00:00
Florian Klink
3a4e29c261 refactor(tvix/store): rename import::{import_path -> ingest_path}
This distinguishes it better from the EvalIO::import_path method.
Also update the docstring to explain what it does (and what it doesn't).

Change-Id: I32a8b2869fa67a894df28532b22bf170961a2abf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8578
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-17 13:06:53 +00:00
Florian Klink
a6c7869393 fix(tvix/store/src/proto/mod.rs): fix error string
This was missed while renaming NixPath to StorePath.

Change-Id: Ibcc929c43b111e4370e8222c1dd86d403548367f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8577
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-17 13:06:52 +00:00
Florian Klink
e815b680c0 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directly
We currently only support querying by the output hash digest.
This makes the interface a bit simpler.

Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-16 10:39:06 +00:00
Florian Klink
a9a2fce27e feat(tvix/store): add GRPCBlobService
This connects to a (remote) tvix-store BlobService over gRPC.

Change-Id: If31f706738a5c3445886c117feca8b61f3203e9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8552
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-14 18:59:55 +00:00
Florian Klink
616fa4476f refactor(tvix/store): remove ChunkService
Whether chunking is involved or not, is an implementation detail of each
Blobstore. Consumers of a whole blob shouldn't need to worry about that.
It currently is not visible in the gRPC interface either. It
shouldn't bleed into everything.

Let the BlobService trait provide `open_read` and `open_write` methods,
which return handles providing io::Read or io::Write, and leave the
details up to the implementation.

This means, our custom BlobReader module can go away, and all the
chunking bits in there, too.

In the future, we might still want to add more chunking-aware syncing,
but as a syncing strategy some stores can expose, not as a fundamental
protocol component.

This currently needs "SyncReadIntoAsyncRead", taken and vendored in from
https://github.com/tokio-rs/tokio/pull/5669.
It provides a AsyncRead for a sync Read, which is necessary to connect
our (sync) BlobReader interface to a GRPC server implementation.

As an alternative, we could also make the BlobReader itself async, and
let consumers of the trait (EvalIO) deal with the async-ness, but this
is less of a change for now.

In terms of vendoring, I initially tried to move our tokio crate to
these commits, but ended up in version incompatibilities, so let's
vendor it in for now.

Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-11 14:27:57 +00:00
Florian Klink
a56b5131c8 chore(tvix): bump cargo dependencies
Change-Id: I6b872a33885f4e29082c554062a60317db754188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8550
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Johannes Kirschbauer <hsjobeki+github@gmail.com>
2023-05-10 06:15:37 +00:00
Florian Klink
a226a3c2c4 fix(tvix/store/nar/renderer): handle digest error
Change-Id: I183580732e1dd33ed079a2593097ec790def0a55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8554
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-10 06:15:37 +00:00
John Ericson
26c68f8e89 refactor(nix-compat): Properly encapsulate store path construction
Before there was code scattered about (e.g. text hashing module and
derivation output computation) constructing store paths from low level
building blocks --- there was some duplication and it was easy to make
nonsense store paths.

Now, we have roughly the same "safe-ish" ways of constructing them as
C++ Nix, and only those are exposed:

- Make text hashed content-addressed store paths

- Make other content-addressed store paths

- Make input-addressed fixed output hashes

Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-04-09 15:12:21 +00:00
Florian Klink
6edc2182d1 docs(tvix/store): update comment regarding reachability check
This might be more understandable.

Change-Id: I91371d304a9a316a2934ce41929998eaef67e11c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:35:39 +00:00
Florian Klink
60751b70eb docs(tvix/store/grpc_directorysvc_wrapper): add TODO
The logic validating connectivity of Directory nodes should be moved
to SimplePutter, and this use whatever DirectoryPutter the store comes
with.

Change-Id: Id68a86a96cc49ff73920017839788859ea9c5161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8358
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:35:39 +00:00
Florian Klink
8263c024c9 refactor(tvix/store/import): use DirectoryPutter in import.rs
This should allow import_path to communicate to a gRPC remote store,
that actually verifies the Directory nodes are interconnected.

Change-Id: Ic5d28c33518f50dedec15f1732d81579a3afaff1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8357
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:26:36 +00:00
Florian Klink
0836450006 feat(tvix/store/directorysvc): add put_multiple_start
This provides a handle to upload multiple proto::Directory as part of
the same closure.

Change-Id: I9213dde257a260c8622239918ea541064b270484
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8356
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-04-07 09:26:36 +00:00
Florian Klink
5bf9134324 refactor(tvix/store/directorysvc): move DirectoryTraverser to utils
Change-Id: Ie60a660e0fda7c80a6c7de20404c1965fe0e0d63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8355
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-31 10:54:48 +00:00
Florian Klink
5f2c2e79e1 refactor(tvix/nix-compat): move build_store_path out of derivation
This doesn't have anything to do with ATerms, we just happen to be using
the aterm representation of a Derivation as contents.

Moving this into store_path/utils.rs makes these things much cleaner -
Have a build_store_path_from_references function, and a
build_store_path_from_fingerprint helper function that makes use of it.

build_store_path_from_references is invoked from the derivation module
which can be used to calculate the derivation path.

In the derivation module, we also invoke
build_store_path_from_fingerprint during the output path calculation.

Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-30 14:03:35 +00:00
Florian Klink
d884890e56 docs(tvix/store/directorysvc/grpc): fix comment
We do compare for equality. This comment probably was when I tried to
compare the `Result<T, E>`, and as `E` doesn't derive PartialEq it was
annoying.

Change-Id: I18bb19528c76af91c9d24d88d55dd46d0c092d20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8354
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 14:37:33 +00:00
Florian Klink
2fe53cce40 feat(tvix/store/directorysvc): add DirectoryService::get_recursive()
This moves the recursive BFS traversal of Directory closures from the
GRPCDirectoryServiceWrapper out into a a DirectoryTraverser struct
implementing Iterator.

It is then used from various implementors of DirectoryService in the
`get_recursive()` method.

This allows distinguishing between recursive requests and non-recursive
requests in the gRPC client trait implementation.

Change-Id: I50bfd4a0d9eb11832847329b78c587ec7c9dc7b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8351
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-27 09:03:34 +00:00
Florian Klink
367a5e9922 feat(tvix/store/directorysvc): add gRPC client
This provides a GRPCDirectoryService struct implementing
DirectoryService, allowing a client to Directory objects from a (remote)
tvix-store.

Remote in this case is anything outside the current process, be it
another process, or an endpoint on the network.

To keep the sync interface in the `DirectoryService` trait, a handle to
some tokio runtime needs to be passed into the constructor, and the two
methods use `self.tokio_handle.spawn` to start an async function, and
`self.tokio_handle.block_on` to wait for its completion.

The client handle, called `grpc_client` itself is easy to clone, and
treats concurrent requests internally. This means, even though we keep
the `DirectoryService` trait sync, there's nothing preventing it from
being used concurrently, let's say from multiple threads.

There's still two limitations for now:

1) The trait doesn't make use of the `recursive` request, which
   currently leads to a N+1 query problem. This can be fixed
   by `GRPCDirectoryService` having a reference to another
   `DirectoryService` acting as the local side.
   I want to wait for general store composition code to pop up before
   manually coding this here.

2) It's currently only possible to put() leaf directory nodes, as the
   request normally requires uploading a whole closure. We might want
   to add another batch function to upload a whole closure, and/or do
   this batching in certain cases. This still needs some more thinking.

Change-Id: I7ffec791610b72c0960cf5307cefbb12ec946dc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8336
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 08:48:41 +00:00
Florian Klink
b919f29752 feat(tvix/store/pathinfosvc): add gRPC client
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-27 08:48:41 +00:00
Florian Klink
93f3964cbc docs(tvix/store/blobreader): fix docstring
Change-Id: Ia8ce1dd1663663903b9f60feada12e6a16fb4af2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8353
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-27 08:44:40 +00:00
Florian Klink
bde10682a5 docs(tvix/store): fix typo in comment
We query the blob service for detailled blob info, not the chunk
service.

Change-Id: I85a6a57b1dae74a950f734be7d4455c5c35ae355
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8348
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 08:44:39 +00:00
sterni
3a74268b6e chore(3p/sources): Bump channels & overlays
* //users/wpcarro/emacs: use top level (ELPA) version of eglot, as it
  was removed from MELPA:

  dc2ead17a8

* //3p/nixpkgs:tdlib: 1.8.11 -> 1.8.12

* //tvix/store: regenerate protos after buf update

Change-Id: I782a8d91fda5ed461788055a3721104e8c032207
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8327
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-21 01:31:48 +00:00
Florian Klink
49cb2a2b1f feat(tvix/store/bin): print store path
After ingestion of the contents into the store, this will use the
NonCachingNARCalculationService to create a NAR stream or the contents
of the path, and use our Derivation output path calculation machinery to
determine the output path (using recursive hashing strategy).

In a real-world scenario, we obviously want to cache these calculations,
but this should be sufficient to tinker around with it.

Change-Id: I9b2e69384414f0be1bdcb5a99a4bfd46e8db9932
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8317
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-19 20:57:27 +00:00
Florian Klink
b025ebb2a1 refactor(tvix/store/nar): pass in &proto::node::Node
Passing in a &proto::node::Node into all this allows us consumers to
keep ownership of the proto::node::Node.

Change-Id: I44882a86c46826b06a8a8a0b24c18adfc7052662
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8316
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-17 07:49:38 +00:00
Florian Klink
985f842e32 chore(tvix/store/grpcblobsvc): clippy lint
Change-Id: Idb78e0417a962599cdfdef5e7346f7fa41e3fa1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8320
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-17 07:49:38 +00:00
Florian Klink
23ddf49229 chore(tvix/store/chunkservice): clippy lint
Change-Id: I2073408fe6e87ffadbeb35ef2f9aaa0316c54966
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8319
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-17 07:49:37 +00:00
Florian Klink
7a966b2540 chore(tvix/store/blobreader): clippy lint
Change-Id: I4c85425782878295277abe6dc91a1b570768f7fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8318
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-17 07:49:37 +00:00
Florian Klink
530cb920b5 refactor(tvix/store/chunksvc): use [u8; 32] instead of Vec<u8>
Change-Id: Ie2b94aa5d69ff2c61fb77e13ae844f81f6270273
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8314
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-16 13:47:32 +00:00
Florian Klink
ee23220564 refactor(tvix/store/directorysvc): use [u8; 32] instead of Vec<u8>
Also, simplify the trait interface, only allowing lookups of Directory
objects by their digest.

Change-Id: I6eec28a8cb0557bed9b69df8b8ff99a5e0f8fe35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8313
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-16 13:47:32 +00:00
Florian Klink
36bed4ce57 chore(tvix/store): drop separate bin package
Apparently, having multiple packages with the same path is a bad thing:

```
The bin target `tvix-store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)` has the same output filename as the lib target `tvix_store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)`.
Colliding filename is: /home/flokli/tvl/tvix/target/doc/tvix_store/index.html
The output filenames should be unique.
This is a known bug where multiple crates with the same name use
the same path; see <https://github.com/rust-lang/cargo/issues/6313>.
```

Change-Id: Ic785c0349070783baf5e8fd23f5fb60603a3c995
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8308
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 22:23:10 +00:00
Florian Klink
21782d24f5 feat(tvix/store): drop BlobWriter
All code initially using this has been replaced by the simpler and more
performant implementation with StreamCDC and read_all_and_chunk.

Change-Id: I08889e9a6984de91c5debcf2b612cb68ae5072d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8265
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 10:05:21 +00:00
Florian Klink
c8bbddd5e5 refactor(tvix/store): use read_all_and_chunk in gRPC blobservice
This was the last piece of code using BlobWriter.

We can also use `read_all_and_chunk`, it's just requires a bit more
plumbing:

 - The data coming from the client (stream) needs to be mapped (we
   extract the .data field).
 - The stream needs to be turned into an (async) reader
 - The reader needs to be made sync, and that code using the sync reader
   needs to be in a `task::spawn_blocking`.

Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 10:05:21 +00:00
Florian Klink
7ffb2676ee refactor(tvix/store): add read_all_and_chunk method
This moves the logic from src/import.rs that

 - reads over the contents of a file
 - chunks them up and uploads individual chunks
 - keeps track of the uploaded chunks in a BlobMeta structure
 - returns the hash of the blob and the BlobMeta structure

… into a generic read_all_and_chunk function in
src/chunkservice/util.rs.

It will work on anything implementing io::Read, not just files, which
will help us in a bit.

Change-Id: I53bf628114b73ee2e515bdae29974571ea2b6f6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8259
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-03-13 08:46:51 +00:00
Florian Klink
2fe7192dbc refactor(tvix/store): use update_hasher in blobwriter
Make use of the helper function here as well.

Change-Id: Ia0afd84eb3903bb897ee6aee884dc291f3e4371c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8258
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-13 08:46:51 +00:00
Florian Klink
62ecfc7001 chore(tvix/store): allow(unused_mut)
When building tvix-store without default features, this variable doesn't
need to be mutable. Silence the warning.

Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-12 16:45:37 +00:00
Florian Klink
e4687765d7 refactor(tvix/store): bump fastcdc dep
This removes the use of Box::new, switching fastcdc to version 3.0.2
with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed.

Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-11 14:12:46 +00:00
Florian Klink
b049b88d2d refactor(tvix/store): factor out hash update into function
We're using this in a bunch of places. Let's move it into a helper
function.

Change-Id: I118fba35f6d343704520ba37280e4ca52a61da44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8251
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
2dc93f8de2 docs(tvix/store): make upload_chunk docs a docstring
Change-Id: I6794e2a69b4907ab8d9ed2d55335c11db2c30c02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8250
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
ead113cdfc feat(tvix/store/import): use StreamCDC instead of blobwriter
This seems to be way faster.

Change-Id: Ica7cee95d108c51fe67365f07366634ddbbfa060
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8246
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-11 12:54:07 +00:00
Florian Klink
ceb9d670bf refactor(tvix/store): rename C to DS, client to directory_service
This matches the semantics in other gRPC wrappers.

Change-Id: I505c901aa77a58deab216316d08542877b0b2c14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8247
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 16:15:48 +00:00
Florian Klink
3506d3bd0e refactor(tvix/store): move upload_chunk out of blobwriter
This is useful not only in blobwriter contexts.

Change-Id: I4c584b5264ff7b4bb3b1a9671affc39e18bf4ccf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8245
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 15:45:45 +00:00
Florian Klink
ac3e470b15 feat(tvix/store): clippy
warning: the following explicit lifetimes could be elided: 'set
   --> store/src/proto/mod.rs:201:1
    |
201 | / fn update_if_lt_prev<'set, 'n>(
202 | |     prev_name: &'set mut &'n str,
203 | |     name: &'n str,
204 | | ) -> Result<(), ValidateDirectoryError> {
    | |_______________________________________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes
    = note: `#[warn(clippy::needless_lifetimes)]` on by default

Change-Id: I053d370a34e5c6721ef86502c7f225c3fbfb8746
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8244
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:23:04 +00:00
Florian Klink
e1783c45a4 refactor(tvix/store/blobwriter): don't clone
This already is a &CS, so we don't need to clone().

Change-Id: I5397a5948ae7fe4781f18df760a79047f83dca01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8243
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 12:23:04 +00:00