Commit graph

117 commits

Author SHA1 Message Date
Florian Klink
432222f098 feat(tvix/store/proto): use Bytes instead of Vec<u8>
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our
bytes field cheaper to clone.

It's a bit annoying to configure due to
https://github.com/hyperium/tonic/issues/908, but the workaround does
get the job done.

Change-Id: I25714600b041bb5432d3adf5859b151e72b12778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-22 09:03:02 +00:00
Florian Klink
6656b865b6 docs(tvix/store): correct some cargo doc warnings
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-19 22:54:46 +00:00
Florian Klink
ad687ccaf4 chore(tvix/store/pathinfosvc): clippy
Change-Id: Ied4bed08e989791f832922da8776d2104035e28a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8812
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-19 14:29:05 +00:00
Florian Klink
35bff2bda6 refactor(tvix/store/pathinfosvc): add from_addr
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-14 23:15:08 +00:00
Florian Klink
c947a73b8e chore(tvix/store/pathinfosvc): remove GRPCPathInfoService::new
There's very little reason to instantiate a GRPCPathInfoService in a
context where we are not already in a tokio context.

Change-Id: Ib81d649387717cb98de8a8039f92472f727b10c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8755
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 21:33:59 +00:00
Florian Klink
b5e37869e6 refactor(tvix/store/pathinfosvc): use Arc<dyn …>
This removes the use of generics, like previously done with Blob and
Directory services.

Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 13:27:05 +00:00
Florian Klink
aa7bdc1199 refactor(tvix/store): use Arc instead of Box
This allows us to blob services without closing them before putting them
in a box.
We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers
require Sync.

Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-12 10:24:24 +00:00
Florian Klink
7725eb53ad refactor(tvix/store): use Box<dyn DirectoryService>
Once we support configuring services at runtime, we don't know what
DirectoryService we're using at compile time.

This also means, we can't explicitly use the is_closed method from
GRPCPutter, without making it part of the DirectoryPutter itself.

Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-12 10:24:24 +00:00
Florian Klink
6f85dbfc06 feat(tvix/store/pathinfosvc): add calculate_nar method
Putting this in the PathInfoService trait makes much more sense, we can
have direct control over where/how to cache the results in the
implementation.

This now requires each PathInfoService to hold pointers to BlobService
and DirectoryService.

Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 10:15:20 +00:00
Florian Klink
8d05c0ceaa refactor(tvix/src/nar): drop NARCalculationService
There's only one way to calculate NAR files, by walking through them.

Things like caching such replies should be done closer to where we use
these, composing NARCalculationService doesn't actually give us much.

Instead, expose two functions, `nar::calculate_size_and_sha256` and
`nar::writer_nar`, the latter writing NAR to a writer, the former using
write_nar to only keeping the NAR size and digest.

Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-12 10:15:20 +00:00
Florian Klink
ad6b6b14db feat(tvix/store/pathinfosvc): derive Clone
This allows closing a TvixStoreIO if it contains a SledPathInfoService.

Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-24 16:04:08 +00:00
Florian Klink
7bd9df0133 feat(tvix/store/pathinfosvc): GRPCPathInfoService::from_client
Change-Id: I4a4f6f713dd76bf95f393c094c0eb9b6d15c5436
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8613
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:37 +00:00
Florian Klink
e815b680c0 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directly
We currently only support querying by the output hash digest.
This makes the interface a bit simpler.

Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-16 10:39:06 +00:00
Florian Klink
b919f29752 feat(tvix/store/pathinfosvc): add gRPC client
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-27 08:48:41 +00:00
Florian Klink
8e97c0deb0 refactor(tvix/store): impl Default for MemoryPathInfoService
Change-Id: I07f83f8aaa6312cc37ce7e7ee2b713e60df23782
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8234
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:13:00 +00:00
Florian Klink
a4f6c4181a feat(tvix/store): add new_temporary for all Sled services
This provides a service using /dev/shm, that's deleted once the
reference is dropped.

Refactor all tests to use these, which allows getting rid of most
TempDir usage in the tests.

The only place where we still use TempDir is in the importer tests,
which work on a filesystem path.

Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
9b3228959a feat(tvix/store): add pathinfoservice
This adds a PathInfoService trait, and an implementation for it using
sled, and one using a HashMap.

Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00