Commit graph

404 commits

Author SHA1 Message Date
Florian Klink
b49f7cfec6 docs(tvix/store/blobservice): fix docstrings
These are outdated.

Change-Id: I1a1ae130a55847f57a48d5e244e7e029c1ecae7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8749
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 12:43:34 +00:00
Ryan Lahfa
5de04dbc12 feat(tvix/store): increase blob chunk size
From 64 bytes to 100 KBytes.
We need to provide a custom wrapper with a different Default instance.

Change-Id: Id7c6c437b8183b355a9e388f98cef1622b363f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-06-12 12:36:56 +00:00
Florian Klink
09c5ca0a0d refactor(tvix/store/fuse): use Arc<dyn …> instead of generics
Change-Id: I5685379bd6f89d17da6843d31bef4c1fc4dc0a18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8745
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-12 10:28:59 +00:00
Florian Klink
38a7caaada refactor(tvix/store/blobsvc/sled): cargo clippy
Change-Id: I00b8b567509d5e0847270f36dadb5dcb534b9b73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8739
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-12 10:28:59 +00:00
Florian Klink
aa7bdc1199 refactor(tvix/store): use Arc instead of Box
This allows us to blob services without closing them before putting them
in a box.
We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers
require Sync.

Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-12 10:24:24 +00:00
Florian Klink
7725eb53ad refactor(tvix/store): use Box<dyn DirectoryService>
Once we support configuring services at runtime, we don't know what
DirectoryService we're using at compile time.

This also means, we can't explicitly use the is_closed method from
GRPCPutter, without making it part of the DirectoryPutter itself.

Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-12 10:24:24 +00:00
Florian Klink
6f85dbfc06 feat(tvix/store/pathinfosvc): add calculate_nar method
Putting this in the PathInfoService trait makes much more sense, we can
have direct control over where/how to cache the results in the
implementation.

This now requires each PathInfoService to hold pointers to BlobService
and DirectoryService.

Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 10:15:20 +00:00
Florian Klink
8d05c0ceaa refactor(tvix/src/nar): drop NARCalculationService
There's only one way to calculate NAR files, by walking through them.

Things like caching such replies should be done closer to where we use
these, composing NARCalculationService doesn't actually give us much.

Instead, expose two functions, `nar::calculate_size_and_sha256` and
`nar::writer_nar`, the latter writing NAR to a writer, the former using
write_nar to only keeping the NAR size and digest.

Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-12 10:15:20 +00:00
Florian Klink
27ff98000b feat(tvix/store): eliminate generics in BlobStore
To construct various stores at runtime, we need to eliminate associated
types from the BlobService trait, and return Box<dyn …> instead of
specific types.

This also means we can't consume self in the close() method, so
everything we write to is put in an Option<>, and during the first close
we take from there.

Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 10:15:20 +00:00
Florian Klink
b3ca1a78eb feat(tvix/store): add mount command to entrypoint and fuse mod
`tvix-store mount PATH` will mount the tvix-store to the given path.

Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-10 15:33:07 +00:00
Florian Klink
83c1546b9a chore(tvix/store): add fuse feature
This brings in fuse (via the `fuser` crate), and adds pkg-config and
libfuse to the dev shell, so `cargo build` can link against it.

Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-10 14:48:48 +00:00
Florian Klink
d925ec34cd docs(src/tests/nar_renderer): fix comment
This testcase tests a missing blob fails the rendering, the comment has
been copied from elsewhere.

Change-Id: I48fa3fa454e12506590fa14a3591d156bafa8b5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8722
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-09 15:33:14 +00:00
Florian Klink
485fd26334 docs(tvix/store): drop nar-bridge references
The CLs did bitrot a bit, they're based on an older version of the
protocol, and it's unclear if they'll be a separate Go Binary, or just
another HTTP handler inside tvix-store itself, considering we now have
way more NAR juggling code than before.

Change-Id: I3632035cda8d75a8ff23b3132312f0f086d9e02f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8732
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-09 15:33:14 +00:00
Florian Klink
7f5372327c docs(tvix/store): update invocation in README
This has moved to a `daemon` subcommand.

Change-Id: Iae9778d8a59e6bf84555119fabfd62db3917bb62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8731
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-09 15:33:14 +00:00
Florian Klink
b78a9b7fe1 docs(tvix/store/docs): fix grammar
Change-Id: I1d7d306e5d997a7ba47a83d613edc19a4fba1ed4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8721
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-09 09:10:51 +00:00
Florian Klink
2b0b4cadbd feat(tvix/store/bin): use sled for daemon, grpc for import cmd
This now creates different store client, depending on the cli
subcommand.
The `import` command will connect to the gRPC service, and the `daemon`
command will use the sled implementation.

It might make sense to define some URI syntax to make this configurable
by the user, via the CLI.

Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 18:10:29 +00:00
Florian Klink
48b66a8982 feat(tvix/store/bin/import): process all path imports concurrently
Change-Id: I3e1428a4725fc2e552e8f37bc0550121117fcef6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8633
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 10:13:07 +00:00
Florian Klink
dbff1289b8 refactor(tvix/store): drop Clone from BS and DS trait bound
We don't need to be able to clone these services in here.

Change-Id: Ifb69450f7ebdc8364cbf9cdfb6464f8560440e4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8645
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-30 10:13:07 +00:00
Florian Klink
a4d0a80466 refactor(tvix/store/bin): instantiating TvixStoreIO once
Instead of instantiating it once in every loop iteration, put it in an
Arc, and clone that before passing it to the spawned task.

Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 12:35:59 +00:00
Florian Klink
d25962b9a4 refactor(tvix/eval): stop borrowing &mut self
This does undo cl/8571.

Change-Id: Ib14b4e7404f906e346304b6113860ae811afc94a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8631
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 11:11:59 +00:00
Florian Klink
fcfbcf9cfa refactor(tvix/store): drop mut self borrow in ingest_path
With traverse_to not requiring a &mut anymore, we can drop the &mut self
in all these function signatures.

Change-Id: I22105376b625cb281c39e92d3206df8a6ce97a88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 11:11:58 +00:00
Florian Klink
ca06f7061c refactor(tvix/store/directorysvc): don't borrow self mutable
Change-Id: I97f183e1ef3b1209a8f05e05e152f70d1f7a9596
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8628
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 10:48:52 +00:00
Florian Klink
dbb4d5e5b2 refactor(tvix/store/bin): use std::io
Change-Id: I3f3dc9732d90790d92268c04c75eccbe92e7e05b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8634
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-25 10:48:51 +00:00
Florian Klink
92b6d15da3 fix(tvix/store/bin): use spawn_blocking to call import_path
This operation is blocking, so it should be run inside a blocking tokio
task. Tokio panics if it detects a blocking operation inside a non-
blocking task, so cl/8619 would cause it to panic (as the GRPC clients
use spawn_blocking under the hood).

As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614),
we create a new instance of TvixStoreIO inside each loop iteration.

Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 05:06:24 +00:00
Florian Klink
ad6b6b14db feat(tvix/store/pathinfosvc): derive Clone
This allows closing a TvixStoreIO if it contains a SledPathInfoService.

Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-24 16:04:08 +00:00
Florian Klink
5774117d5e feat(tvix/store): implement TvixStoreIO
This providesEvalIO, asking given PathInfoService, DirectoryService
and BlobService.

Change-Id: I32f210f5a7aa8173ad9a7d53e8a5ac03619f527a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8561
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 09:25:02 +00:00
Florian Klink
7bd9df0133 feat(tvix/store/pathinfosvc): GRPCPathInfoService::from_client
Change-Id: I4a4f6f713dd76bf95f393c094c0eb9b6d15c5436
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8613
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:37 +00:00
Florian Klink
8d392221c6 feat(tvix/store/directorysvc): GRPCDirectoryService::from_client
Change-Id: Ifa274a380683b01f0f24cd9ff1f50d22f13b38bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8612
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:36 +00:00
Florian Klink
f4a7b325e9 feat(tvix/store/blobsvc): add constructors
Change-Id: I55e06bf4e8a11dc2caf92c597558f1b820b42566
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8610
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:36 +00:00
Florian Klink
2d214eacc5 fix(tvix/store/directorysvc): stop renaming spans
We have code.namespace as a field, set to
tvix_store::directoryservice::sled, so there's no need to repeat the
name here.

Change-Id: Ic1aa8a2b24de439c6a189966bd773e9acf49d1e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8623
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 09:01:35 +00:00
Florian Klink
457fd4c325 feat(tvix/store/nar): add GRPCNARCalculationService
This asks a remote tvix-store for the nar size and digest of a given
root node.

Change-Id: If9f916d9bfc5f8dc3166e2c6c1671c0f0124d1c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8611
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-24 09:01:35 +00:00
Florian Klink
79bbf25cb2 refactor(tvix/store/nar): clippy
Change-Id: I0d864fd1b9248fe4d5c88e3d8fa396ea759fea09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8616
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
4dd954f8f5 refactor(tvix/store/directorysvc): clippy
Change-Id: I30f7e98fe79ba22d218b4aabaef88c84b6085d82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8617
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
e51d85e078 refactor(tvix/store/digests): clippy
useless conversion to the same type: `std::vec::Vec<u8>`

Change-Id: Idcb16679bb2a3350784965d8d9bbb593c760634e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8615
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
066179651c refactor(tvix/store/blobsvc): move from Vec<u8> to B3Digest
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-23 10:48:55 +00:00
Florian Klink
b8ff08b1b0 refactor(tvix/store/directorysvc): move from Vec<u8> to B3Digest
This introduces a new struct, B3Digest, which internally holds a
Vec<u8>, but only allows construction with 32 bytes.

It also implements display, which will print the base64 representation.
This should reduce some boilerplate when parsing Vec<u8>.

Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-23 10:48:55 +00:00
Florian Klink
e31008db14 fix(tvix/store): fix timing sensitivity in gRPC directorysvc test
One test in here assumed the server was fast, but when very busy, a
computer running these tests might not be fast.

Treat both cases that can occur separately.

Change-Id: Iba168ad39f2458c4fb8873211df33beeaff7c6c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8595
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-18 19:38:54 +00:00
Florian Klink
103d2eb909 refactor(tvix/store): bump CalculateNARResponse.nar_size to u64
Change-Id: I5f41981820363a5eb9982c3d19830916207c62cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8591
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
0e7bbf0d01 fix(tvix/store/proto/grpc_blobsvc): clippy
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-18 19:29:53 +00:00
Florian Klink
a0801ae220 refactor(tvix/store/blobsvc/sled): clippy
Change-Id: Icfea8ceb6b827a645c6f10efae741f6fa2114fad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8590
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
2e09e94aad refactor(tvix/store/directorysvc/traverse): clippy, use NamedNode
Also, get rid of the explicit byte comparison here, but unwrap like in
the rest of the codebase. We'll deal with this in a generic manner in
b/267 and b/189.

Change-Id: Ie5f3d27ab35b7e88d67a2796c29cdd7bc7df71f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8589
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
a1324513ad feat(tvix/store/directorysvc): add traverse_to
This walks from a node further down until it reaches the requested path.

Change-Id: I2f9a15a8601db4d06c95d7b47cd6153264e203e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8568
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-18 06:26:55 +00:00
Florian Klink
3a4e29c261 refactor(tvix/store): rename import::{import_path -> ingest_path}
This distinguishes it better from the EvalIO::import_path method.
Also update the docstring to explain what it does (and what it doesn't).

Change-Id: I32a8b2869fa67a894df28532b22bf170961a2abf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8578
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-17 13:06:53 +00:00
Florian Klink
a6c7869393 fix(tvix/store/src/proto/mod.rs): fix error string
This was missed while renaming NixPath to StorePath.

Change-Id: Ibcc929c43b111e4370e8222c1dd86d403548367f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8577
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-17 13:06:52 +00:00
Florian Klink
e815b680c0 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directly
We currently only support querying by the output hash digest.
This makes the interface a bit simpler.

Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-16 10:39:06 +00:00
Florian Klink
a9a2fce27e feat(tvix/store): add GRPCBlobService
This connects to a (remote) tvix-store BlobService over gRPC.

Change-Id: If31f706738a5c3445886c117feca8b61f3203e9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8552
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-14 18:59:55 +00:00
Florian Klink
616fa4476f refactor(tvix/store): remove ChunkService
Whether chunking is involved or not, is an implementation detail of each
Blobstore. Consumers of a whole blob shouldn't need to worry about that.
It currently is not visible in the gRPC interface either. It
shouldn't bleed into everything.

Let the BlobService trait provide `open_read` and `open_write` methods,
which return handles providing io::Read or io::Write, and leave the
details up to the implementation.

This means, our custom BlobReader module can go away, and all the
chunking bits in there, too.

In the future, we might still want to add more chunking-aware syncing,
but as a syncing strategy some stores can expose, not as a fundamental
protocol component.

This currently needs "SyncReadIntoAsyncRead", taken and vendored in from
https://github.com/tokio-rs/tokio/pull/5669.
It provides a AsyncRead for a sync Read, which is necessary to connect
our (sync) BlobReader interface to a GRPC server implementation.

As an alternative, we could also make the BlobReader itself async, and
let consumers of the trait (EvalIO) deal with the async-ness, but this
is less of a change for now.

In terms of vendoring, I initially tried to move our tokio crate to
these commits, but ended up in version incompatibilities, so let's
vendor it in for now.

Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-11 14:27:57 +00:00
Florian Klink
a56b5131c8 chore(tvix): bump cargo dependencies
Change-Id: I6b872a33885f4e29082c554062a60317db754188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8550
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Johannes Kirschbauer <hsjobeki+github@gmail.com>
2023-05-10 06:15:37 +00:00
Florian Klink
a226a3c2c4 fix(tvix/store/nar/renderer): handle digest error
Change-Id: I183580732e1dd33ed079a2593097ec790def0a55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8554
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-10 06:15:37 +00:00
John Ericson
26c68f8e89 refactor(nix-compat): Properly encapsulate store path construction
Before there was code scattered about (e.g. text hashing module and
derivation output computation) constructing store paths from low level
building blocks --- there was some duplication and it was easy to make
nonsense store paths.

Now, we have roughly the same "safe-ish" ways of constructing them as
C++ Nix, and only those are exposed:

- Make text hashed content-addressed store paths

- Make other content-addressed store paths

- Make input-addressed fixed output hashes

Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-04-09 15:12:21 +00:00
Florian Klink
6edc2182d1 docs(tvix/store): update comment regarding reachability check
This might be more understandable.

Change-Id: I91371d304a9a316a2934ce41929998eaef67e11c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:35:39 +00:00
Florian Klink
60751b70eb docs(tvix/store/grpc_directorysvc_wrapper): add TODO
The logic validating connectivity of Directory nodes should be moved
to SimplePutter, and this use whatever DirectoryPutter the store comes
with.

Change-Id: Id68a86a96cc49ff73920017839788859ea9c5161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8358
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:35:39 +00:00
Florian Klink
8263c024c9 refactor(tvix/store/import): use DirectoryPutter in import.rs
This should allow import_path to communicate to a gRPC remote store,
that actually verifies the Directory nodes are interconnected.

Change-Id: Ic5d28c33518f50dedec15f1732d81579a3afaff1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8357
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:26:36 +00:00
Florian Klink
0836450006 feat(tvix/store/directorysvc): add put_multiple_start
This provides a handle to upload multiple proto::Directory as part of
the same closure.

Change-Id: I9213dde257a260c8622239918ea541064b270484
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8356
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-04-07 09:26:36 +00:00
Florian Klink
5bf9134324 refactor(tvix/store/directorysvc): move DirectoryTraverser to utils
Change-Id: Ie60a660e0fda7c80a6c7de20404c1965fe0e0d63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8355
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-31 10:54:48 +00:00
Florian Klink
5f2c2e79e1 refactor(tvix/nix-compat): move build_store_path out of derivation
This doesn't have anything to do with ATerms, we just happen to be using
the aterm representation of a Derivation as contents.

Moving this into store_path/utils.rs makes these things much cleaner -
Have a build_store_path_from_references function, and a
build_store_path_from_fingerprint helper function that makes use of it.

build_store_path_from_references is invoked from the derivation module
which can be used to calculate the derivation path.

In the derivation module, we also invoke
build_store_path_from_fingerprint during the output path calculation.

Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-30 14:03:35 +00:00
Florian Klink
d884890e56 docs(tvix/store/directorysvc/grpc): fix comment
We do compare for equality. This comment probably was when I tried to
compare the `Result<T, E>`, and as `E` doesn't derive PartialEq it was
annoying.

Change-Id: I18bb19528c76af91c9d24d88d55dd46d0c092d20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8354
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 14:37:33 +00:00
Florian Klink
2fe53cce40 feat(tvix/store/directorysvc): add DirectoryService::get_recursive()
This moves the recursive BFS traversal of Directory closures from the
GRPCDirectoryServiceWrapper out into a a DirectoryTraverser struct
implementing Iterator.

It is then used from various implementors of DirectoryService in the
`get_recursive()` method.

This allows distinguishing between recursive requests and non-recursive
requests in the gRPC client trait implementation.

Change-Id: I50bfd4a0d9eb11832847329b78c587ec7c9dc7b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8351
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-27 09:03:34 +00:00
Florian Klink
367a5e9922 feat(tvix/store/directorysvc): add gRPC client
This provides a GRPCDirectoryService struct implementing
DirectoryService, allowing a client to Directory objects from a (remote)
tvix-store.

Remote in this case is anything outside the current process, be it
another process, or an endpoint on the network.

To keep the sync interface in the `DirectoryService` trait, a handle to
some tokio runtime needs to be passed into the constructor, and the two
methods use `self.tokio_handle.spawn` to start an async function, and
`self.tokio_handle.block_on` to wait for its completion.

The client handle, called `grpc_client` itself is easy to clone, and
treats concurrent requests internally. This means, even though we keep
the `DirectoryService` trait sync, there's nothing preventing it from
being used concurrently, let's say from multiple threads.

There's still two limitations for now:

1) The trait doesn't make use of the `recursive` request, which
   currently leads to a N+1 query problem. This can be fixed
   by `GRPCDirectoryService` having a reference to another
   `DirectoryService` acting as the local side.
   I want to wait for general store composition code to pop up before
   manually coding this here.

2) It's currently only possible to put() leaf directory nodes, as the
   request normally requires uploading a whole closure. We might want
   to add another batch function to upload a whole closure, and/or do
   this batching in certain cases. This still needs some more thinking.

Change-Id: I7ffec791610b72c0960cf5307cefbb12ec946dc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8336
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 08:48:41 +00:00
Florian Klink
b919f29752 feat(tvix/store/pathinfosvc): add gRPC client
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-27 08:48:41 +00:00
Florian Klink
93f3964cbc docs(tvix/store/blobreader): fix docstring
Change-Id: Ia8ce1dd1663663903b9f60feada12e6a16fb4af2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8353
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-27 08:44:40 +00:00
Florian Klink
bde10682a5 docs(tvix/store): fix typo in comment
We query the blob service for detailled blob info, not the chunk
service.

Change-Id: I85a6a57b1dae74a950f734be7d4455c5c35ae355
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8348
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 08:44:39 +00:00
sterni
3a74268b6e chore(3p/sources): Bump channels & overlays
* //users/wpcarro/emacs: use top level (ELPA) version of eglot, as it
  was removed from MELPA:

  dc2ead17a8

* //3p/nixpkgs:tdlib: 1.8.11 -> 1.8.12

* //tvix/store: regenerate protos after buf update

Change-Id: I782a8d91fda5ed461788055a3721104e8c032207
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8327
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-21 01:31:48 +00:00
Florian Klink
49cb2a2b1f feat(tvix/store/bin): print store path
After ingestion of the contents into the store, this will use the
NonCachingNARCalculationService to create a NAR stream or the contents
of the path, and use our Derivation output path calculation machinery to
determine the output path (using recursive hashing strategy).

In a real-world scenario, we obviously want to cache these calculations,
but this should be sufficient to tinker around with it.

Change-Id: I9b2e69384414f0be1bdcb5a99a4bfd46e8db9932
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8317
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-19 20:57:27 +00:00
Florian Klink
b025ebb2a1 refactor(tvix/store/nar): pass in &proto::node::Node
Passing in a &proto::node::Node into all this allows us consumers to
keep ownership of the proto::node::Node.

Change-Id: I44882a86c46826b06a8a8a0b24c18adfc7052662
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8316
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-17 07:49:38 +00:00
Florian Klink
985f842e32 chore(tvix/store/grpcblobsvc): clippy lint
Change-Id: Idb78e0417a962599cdfdef5e7346f7fa41e3fa1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8320
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-17 07:49:38 +00:00
Florian Klink
23ddf49229 chore(tvix/store/chunkservice): clippy lint
Change-Id: I2073408fe6e87ffadbeb35ef2f9aaa0316c54966
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8319
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-17 07:49:37 +00:00
Florian Klink
7a966b2540 chore(tvix/store/blobreader): clippy lint
Change-Id: I4c85425782878295277abe6dc91a1b570768f7fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8318
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-17 07:49:37 +00:00
Florian Klink
530cb920b5 refactor(tvix/store/chunksvc): use [u8; 32] instead of Vec<u8>
Change-Id: Ie2b94aa5d69ff2c61fb77e13ae844f81f6270273
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8314
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-16 13:47:32 +00:00
Florian Klink
ee23220564 refactor(tvix/store/directorysvc): use [u8; 32] instead of Vec<u8>
Also, simplify the trait interface, only allowing lookups of Directory
objects by their digest.

Change-Id: I6eec28a8cb0557bed9b69df8b8ff99a5e0f8fe35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8313
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-16 13:47:32 +00:00
Florian Klink
36bed4ce57 chore(tvix/store): drop separate bin package
Apparently, having multiple packages with the same path is a bad thing:

```
The bin target `tvix-store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)` has the same output filename as the lib target `tvix_store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)`.
Colliding filename is: /home/flokli/tvl/tvix/target/doc/tvix_store/index.html
The output filenames should be unique.
This is a known bug where multiple crates with the same name use
the same path; see <https://github.com/rust-lang/cargo/issues/6313>.
```

Change-Id: Ic785c0349070783baf5e8fd23f5fb60603a3c995
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8308
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 22:23:10 +00:00
Florian Klink
21782d24f5 feat(tvix/store): drop BlobWriter
All code initially using this has been replaced by the simpler and more
performant implementation with StreamCDC and read_all_and_chunk.

Change-Id: I08889e9a6984de91c5debcf2b612cb68ae5072d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8265
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 10:05:21 +00:00
Florian Klink
c8bbddd5e5 refactor(tvix/store): use read_all_and_chunk in gRPC blobservice
This was the last piece of code using BlobWriter.

We can also use `read_all_and_chunk`, it's just requires a bit more
plumbing:

 - The data coming from the client (stream) needs to be mapped (we
   extract the .data field).
 - The stream needs to be turned into an (async) reader
 - The reader needs to be made sync, and that code using the sync reader
   needs to be in a `task::spawn_blocking`.

Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 10:05:21 +00:00
Florian Klink
7ffb2676ee refactor(tvix/store): add read_all_and_chunk method
This moves the logic from src/import.rs that

 - reads over the contents of a file
 - chunks them up and uploads individual chunks
 - keeps track of the uploaded chunks in a BlobMeta structure
 - returns the hash of the blob and the BlobMeta structure

… into a generic read_all_and_chunk function in
src/chunkservice/util.rs.

It will work on anything implementing io::Read, not just files, which
will help us in a bit.

Change-Id: I53bf628114b73ee2e515bdae29974571ea2b6f6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8259
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-03-13 08:46:51 +00:00
Florian Klink
2fe7192dbc refactor(tvix/store): use update_hasher in blobwriter
Make use of the helper function here as well.

Change-Id: Ia0afd84eb3903bb897ee6aee884dc291f3e4371c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8258
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-13 08:46:51 +00:00
Florian Klink
62ecfc7001 chore(tvix/store): allow(unused_mut)
When building tvix-store without default features, this variable doesn't
need to be mutable. Silence the warning.

Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-12 16:45:37 +00:00
Florian Klink
e4687765d7 refactor(tvix/store): bump fastcdc dep
This removes the use of Box::new, switching fastcdc to version 3.0.2
with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed.

Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-11 14:12:46 +00:00
Florian Klink
b049b88d2d refactor(tvix/store): factor out hash update into function
We're using this in a bunch of places. Let's move it into a helper
function.

Change-Id: I118fba35f6d343704520ba37280e4ca52a61da44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8251
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
2dc93f8de2 docs(tvix/store): make upload_chunk docs a docstring
Change-Id: I6794e2a69b4907ab8d9ed2d55335c11db2c30c02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8250
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
ead113cdfc feat(tvix/store/import): use StreamCDC instead of blobwriter
This seems to be way faster.

Change-Id: Ica7cee95d108c51fe67365f07366634ddbbfa060
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8246
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-11 12:54:07 +00:00
Florian Klink
ceb9d670bf refactor(tvix/store): rename C to DS, client to directory_service
This matches the semantics in other gRPC wrappers.

Change-Id: I505c901aa77a58deab216316d08542877b0b2c14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8247
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 16:15:48 +00:00
Florian Klink
3506d3bd0e refactor(tvix/store): move upload_chunk out of blobwriter
This is useful not only in blobwriter contexts.

Change-Id: I4c584b5264ff7b4bb3b1a9671affc39e18bf4ccf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8245
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 15:45:45 +00:00
Florian Klink
ac3e470b15 feat(tvix/store): clippy
warning: the following explicit lifetimes could be elided: 'set
   --> store/src/proto/mod.rs:201:1
    |
201 | / fn update_if_lt_prev<'set, 'n>(
202 | |     prev_name: &'set mut &'n str,
203 | |     name: &'n str,
204 | | ) -> Result<(), ValidateDirectoryError> {
    | |_______________________________________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes
    = note: `#[warn(clippy::needless_lifetimes)]` on by default

Change-Id: I053d370a34e5c6721ef86502c7f225c3fbfb8746
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8244
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:23:04 +00:00
Florian Klink
e1783c45a4 refactor(tvix/store/blobwriter): don't clone
This already is a &CS, so we don't need to clone().

Change-Id: I5397a5948ae7fe4781f18df760a79047f83dca01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8243
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 12:23:04 +00:00
Florian Klink
d2b353e516 refactor(tvix/store): clippy
Change-Id: Ib41c766a45e7d5467a4e5e90e1c00b009121959a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8238
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:15:02 +00:00
Florian Klink
6d3f5b7eb4 refactor(tvix/store): impl Default for MemoryBlobService
Change-Id: I78a4cd86cb364b970e99393579808e773db5ceb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8237
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:15:02 +00:00
Florian Klink
b3d6e70909 refactor(tvix/store): impl Default for MemoryChunkService
Change-Id: If1ceb870e9e5d03b651923a3a786350a8663e990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 12:13:01 +00:00
Florian Klink
e716bd0957 refactor(tvix/store): impl Default for MemoryDirectoryService
Change-Id: Ie75a1e67c8c0499ff10f60d90251b768937dad3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8235
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:13:01 +00:00
Florian Klink
8e97c0deb0 refactor(tvix/store): impl Default for MemoryPathInfoService
Change-Id: I07f83f8aaa6312cc37ce7e7ee2b713e60df23782
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8234
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:13:00 +00:00
Florian Klink
510927e43a feat(tvix/store): use rayon to upload chunks concurrently
Look at the data that's written to us, and upload all chunks but the
rest in parallel, using rayon. This required moving `upload_chunk`
outside the struct, and accepting a ChunkService to use for upload
(which it was previously getting from `self.chunk_service`).

This doesn't speed up things too much for now, because things are still
mostly linear.

Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 11:53:29 +00:00
Florian Klink
2ef60282b6 feat(tvix/store/import): make sure entries are sorted
The Directory service does already reject inserting invalid (wrongly
sorted) Directory messages, but our test case didn't provoke it.

Change-Id: I228e201925e8999186659a2d8da0118db184d9ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8167
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
a4d06b68d8 feat(tvix/store): move to daemon subcommand, add import subcommand
This exposes the previous default behavior at the `tvix-store daemon`
subcommand.

It also adds a `tvix-store import` command, which will ingest a given
path into the store.

Change-Id: Ide14f1d409b9364e7f98090690c744326486e470
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8166
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
6f6ddccc92 refactor(tvix/store): move entrypoint to src/bin/tvix-store.rs
Change-Id: Ibb83be75a2be27debd9e85b43c1b824f59e54dab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8165
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
b4a89461f5 feat(tvix/store): use Memory*Service instead of temporary sled
Change-Id: I65d2030aa643b6937dd92baa85db033bdce38b02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8208
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
a4f6c4181a feat(tvix/store): add new_temporary for all Sled services
This provides a service using /dev/shm, that's deleted once the
reference is dropped.

Refactor all tests to use these, which allows getting rid of most
TempDir usage in the tests.

The only place where we still use TempDir is in the importer tests,
which work on a filesystem path.

Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
535e1b15ab fix(tvix/store/proto/grpc_blobservice_wrapper): buffer recv data
While we don't want to keep all of the data in memory, we want to
feed a reasonably-enough buffer to the chunking function, to prevent
unnecessarily trying to chunk over and over again.

Change-Id: I5bbe2d55e8c1c63f8f7ce343889d374b528b559e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8160
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
b29d1ae372 feat(tvix/store): add import::import_path
This imports the contents at a given Path into the tvix store.

It doesn't register the contents at a Path in the store itself, that's up
to the PathInfoService.

Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
52a5181eba refactor(tvix/store/tests/nar_renderer): only upload EMPTY_BLOB
This also uploaded HELLOWORLD_BLOB_CONTENTS before, but it's not
referred from anywhere in the fixture.

Change-Id: I823133afe0f08d18a59e2ac4e4d4bb7d34ce8a2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8158
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
092f8a7df8 refactor(tvix/store/tests): move nar contents to fixtures
Change-Id: I89a700240e7250a4d93eb8c4235e79c29ecb7f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8157
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
e7279b6063 refactor(tvix/store/tests): move fixtures into separate module
Change-Id: I362dbf0899e4dc42114fd2e6a8fa7f537e9ea138
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8156
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
28a862976b refactor(tvix/store/tests): move gen_*_service() into helper
This allows hiding to tests what exact implementation we're using, when
testing things that do something with a store, but don't care what's
used for underlying storage.

Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
0baaabc43e refactor(tvix/store): move blob splitting into a BlobWriter struct
This will moves the chunking-as-we-receive logic that so far only lived
in grpc_blobservice_wrapper.rs into a generic BlobWriter.

Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
d8ab140d25 feat(tvix/store): do not buffer blob data
Use the FastCDC::cut function to ask fastcd for cutting points as we
receive the data. Make sure to keep the last chunk in the temporary
buffer, as we might not actually cut at the end.

Also, use rayon to calculate the blake3 hash if the input data is
> 128KiB.

Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
a40d2dcdcd feat(tvix/store): bump fastcdc, use v2020 version
This switches away from the less canonical "ronomon" version to the
implementation as described in the
[paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et
al., in 2020.

That version uses 64-bit hash values and tends to be faster than both
the ronomon and v2016 versions, and produces the same chunking as the
2016 version.

As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's
the recommended choice.

The crate also gained support for streaming version of chunkers:
https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be
useful.

Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
fbabcf0420 feat(tvix/store): add --json arg
This configures logging as JSON.

Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
7fe7e03df3 chore(tvix/store): remove old implementations
These were implementing against the (more complicated) gRPC interface,
for which we now have a wrapper.

Change-Id: I0a8284493718ab99618a1d21a76df4d173edb899
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8100
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
6b91efa5cf refactor(tvix/store): use GRPC Wrappers and new services
This swaps out the implementation used in the main entrypoint.
We now use the non-gRPC aware Sled*Service, and wrap it with
GRPC*ServiceWrapper to spin up the gRPC server.

Change-Id: I5a1236c0612eee85e49891062040127c8bd95058
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8106
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
cd22a8aa6d feat(tvix/store): add tests for GRPCPathInfoServiceWrapper
Change-Id: I22de33c28c132fc9710ddecd0e9158ee3b3ef94b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8108
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
dd1ee7a2e7 feat(tvix/store): add GRPCPathInfoServiceWrapper
This exposes a proto::pathinfo_service_server::PathInfoService for a
directoryservice::PathInfoService and a way to calculate NARs.

Change-Id: I30cd058562f83d063c78b84976ec97190de49400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8097
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
9b3228959a feat(tvix/store): add pathinfoservice
This adds a PathInfoService trait, and an implementation for it using
sled, and one using a HashMap.

Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
419f7d0f0c feat(tvix/store): add NARCalculationService trait and impl
This adds a NARCalculationService trait, which will take a root node,
and provide a proto::CalculateNarResponse in return.

It also adds a NonCachingNARCalculationService implementation, that will
simply always render the NAR in memory to calculate the size and sha256,
without any caching.

Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
ab02fc668c feat(tvix/store): validate blob size in NARRenderer
Make sure the blob size in the current proto node matches what we get
back from the blob backend.

Change-Id: I939fa18f37c7bc86ada8a495c7be622e69ec47f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8129
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
bc3f71838f refactor(tvix/store): don't use anyhow::Result<()> in tests
It's much easier to just unwrap the TempDir::new() result.

Change-Id: I7b05bc18f3146401e30e1cc2bb412503c5171a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8128
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
df3223fd68 chore(tvix/store): move NAR rendering logic into Renderer struct
This moves the logic rendering NARs to a struct using the
previously introduced, more granular BlobService, ChunkService and
DirectoryService.

Instead of passing them around to the helper functions, they're kept as
members of a struct.

Remove the async invocations in the nar_renderer tests, there's nothing
async in here.

Change-Id: Ic6d24aaad68a1fda46ce29f2cdb5f7b87f481d5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8095
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
cdb9458310 feat(tvix/store): add tests for GRPCDirectoryServiceWrapper
Change-Id: Iccf70664cd5dd8ea22e0b43002d2ff1a9680870a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8102
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
1540c7e276 feat(tvix/store): add GRPCDirectoryServiceWrapper
This exposes a proto::directory_service_server::DirectoryService for a
directoryservice::DirectoryService.

Change-Id: I191a6e44e5bce10d904b5f10f0073d6adaf1f572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8094
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
3af467d7ee feat(tvix/store): add directoryservice
This adds a DirectoryService trait, and an implementation for it using
sled, and one using a HashMap.

Change-Id: Ida61524b2ca949e1b3a78089a5aa7d9f9800c8d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8093
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
35ea0b0d2e feat(tvix/store): add tests for GRPCBlobServiceWrapper
Change-Id: I0c9a2d887b5b94c43afee97a8a1e701cc9f74f23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8098
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
69516f1f68 feat(tvix/store/blobservice): add GRPCBlobServiceWrapper
This takes a BlobService and ChunkService in the constructor, and
provides a [proto::blob_service_server::BlobService] trait for it.

Implementing proto::blob_service_server::BlobService is a lot of surface
to cover, and providing this wrapper will make individual
implementations taking care of how to store chunks or chunking
information much simpler.

Change-Id: Ia7b46484fb3ac9104354d496ff2922dca96ff7b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8092
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
bdf0725def refactor(tvix/store): remove anyhow::Result in directory it test
There's no `?` being used.

Change-Id: Ifbfd88f6056e0064d11458cb335d05b333ec7b6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8127
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
95aec51b2c refactor(tvix/store): remove useless clone in test
Change-Id: If62887cbc3338110cb7c815ec2ee2160fae04353
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8111
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
80f68bf828 chore(tvix/store): move protos into separate mod.rs
This allows adding more stuff into this namespace, from different files.

Also move tests on proto-related code from src/tests to src/proto/tests.

Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
60abca1d8e feat(tvix/store): add blobreader
A BlobReader can be used to read a blob, which might consist out of
multiple chunks. Chunks are fetched from a ChunkService.

Change-Id: I1806225b0052adaa4a2320b79b744e554e524ee3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8088
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 10:58:55 +00:00
Florian Klink
b74ffda583 feat(tvix/store): add blobservice
This adds a BlobService trait, and an implementation for it using sled,
and one using a HashMap.

Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 15:32:47 +00:00
Florian Klink
119aa43171 feat(tvix/store): add chunkservice
This adds the simpler ChunkService trait, and an implementation for it
using sled, and one using a HashMap.

Change-Id: Icb0fdc41b37b44e9e9e4f548d0f4acae1d83b71e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8086
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-06 15:32:47 +00:00
Florian Klink
3f27fe3484 feat(tvix/store): impl From<PoisonError> for Error
Change-Id: Ib61e276b45c0102e383a7e7e641172b151369b03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8207
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 15:32:47 +00:00
Florian Klink
786854713c feat(tvix/store): add errors
Change-Id: I7be0a4eaa29840e1a3329bccb791975de22828ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8085
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-06 15:32:47 +00:00
Florian Klink
6019c75deb docs(tvix/store): add document describing why we don't use git trees
This came up recently again, it makes sense to document the reasoning
behind the decision.

Change-Id: Ic51d5bc7998c70e8b070b6f42877d8e88613935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8223
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-05 18:33:54 +00:00
Florian Klink
db8cebe0b4 docs(tvix/store): move link hrefs to bottom
Change-Id: Id8e32ee4de98fbe48266d417279194cd93e968cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8222
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-05 14:42:05 +00:00
Florian Klink
f2624f1028 refactor(tvix/store): remove needless borrow in build.rs
Change-Id: I2c2a70e5ffbb080702f57218aa9dcfe53d08870f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8110
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-15 18:40:36 +00:00
Florian Klink
cb8466d417 refactor(tvix/store/proto): use .cloned()
Instead of using an explicit closure to clone elements, use .cloned().

Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-15 18:40:34 +00:00
Florian Klink
cfa0755fc4 refactor(tvix/store): avoid casting to the same type
The size field already is u32, we don't need to convert here.

Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-15 18:40:33 +00:00
Florian Klink
87e1934f2d docs(store/protos): update comment on blobstore Read and Put
Further emphasize Read() can be used to ask for blobs OR chunks, and
that clients usually want to stat and then request (smaller) chunks,
rather than reading whole blobs.

Also clarify that the chunking used to send BlobChunks over has nothing
to do with the chunk sizes communicated in a Stat() request.

Change-Id: Ia615d190aae570611de2655b11342a14d0b75976
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-07 14:14:35 +00:00
Florian Klink
0db73cb2bd feat(tvix/store): add write_nar function
This adds a function that consumes a [proto::node::Node] pointing to the
root of a (store) path, and writes the contents in NAR serialization to
the passed [std::io::Write].

We need this in various places:

 - tvix-store's calculate_nar() RPC method needs to render a NAR stream
   to get the nar hash, which is necessary to give things imported in
   the store a "NAR-based" store path.

 - communication with (remote) Nix (via daemon protocol) needs a NAR
   representation.

 - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a
   NAR representation.

Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:28:22 +00:00
Florian Klink
9e809e21cc feat(tvix/store): implement iteration over Directory nodes
This provides a `Directory.nodes()` function, returning an iterator over
all three node types in an ordered fashion.

Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
1ee6bd06e3 feat(tvix/store/proto): implement get_name for node::Node, pub trait
Make the trait public, so consumers can use it.

Also, implement it for node::Node, so we can later use this to access
the name from all three if we don't care about the enum type.

Change-Id: Iae530a16b705493645e61947852c03273876cc55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
c27bacd905 refactor(tvix): introduce nix-compat crate
Move nixbase32 and store_path into this.

This allows //tvix/cli to not pull in //tvix/store for now.

Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 13:26:18 +00:00
Florian Klink
8ec8d03175 fix(tvix/store/nixbase32): fix encoder/decoder
Replace our data_encoding usage with the implementation taken from
https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32

Also uncomment some of the unit tests, and add a regression test for a
NIXBASE32.encode with a 32 bytes sequence.

The previous implementation of NIXBASE32.encode is wrong in that case:

```
❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA=
0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk
❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C
00000000  b3 a2 4d e9 7a 8f db c8  35 b9 83 31 69 50 10 30  |..M.z...5..1iP.0|
00000010  b8 97 70 31 bc b5 4b 3b  3a c1 37 40 f8 46 ab 30  |..p1..K;:.7@.F.0|
00000020
```

Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 11:46:30 +00:00
Florian Klink
85e563c554 refactor(tvix/store/tests): move Directory tests into directory.rs
There's gonna be more tests coming, let's move them into a separate file
everything Directory-related into a separate module.

Change-Id: I78a0f263925528907a22724b028f75cce644b329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-30 09:45:11 +00:00
Florian Klink
a2c33f517c feat(tvix/store/proto): implement get_name for all nodes
Also add a `NamedNode` trait. We'll later use this to access names from
all three individually.

Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 09:45:11 +00:00
Florian Klink
9c18888715 docs(tvix/store/protos): add docstring for Directory::digest()
Change-Id: I361dbca444a267fea28cd212d563ee2d03497c16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7952
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 09:45:11 +00:00
Florian Klink
42d3c06988 docs(tvix/store/protos): fix docstring
Change-Id: Idf835e9648ab6c95c29960f3f8176153d4888b27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7951
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-30 09:45:11 +00:00
Florian Klink
1a84e3e6d8 chore(tvix/store): add fastcdc crate
This is used for content-defined chunking.

Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 15:12:45 +00:00
Florian Klink
90979d39f3 feat(tvix/store/directory): validate Directory and sizes
This calls out to Directory::validate() for all received Directory
messages, and also makes sure the sizes we refer a Directory message as
matches the sizes that have been calculated.

Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 10:48:17 +00:00
Florian Klink
d8e0fa8e5e feat(tvix/store/directory): deduplicate Directory messages
We can omit sending Directory messages to clients that have already been
sent in the same stream.

We can also omit storing a Directory message if we already have it -
they're content-addressed anyways.

Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 10:48:17 +00:00
Florian Klink
e719da53be feat(tvix/store): add SledDirectoryService
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 10:48:17 +00:00
Florian Klink
ab8486e5b8 chore(tvix/store): add tonic-mock
Upstream seems to be dead, so we're using https://github.com/tyrchen/
tonic-mock/pull/3 here.

According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment-
1241164173, we might not need this crate at all, but for now, it gets
the job done and is less code to write in the tests.

Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 09:34:15 +00:00
Florian Klink
a8b13a0b57 refactor(tvix/store): simplify test a bit
Import more things, and use expect_err to unpack the response.

Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 09:34:15 +00:00
Florian Klink
4a256dda09 docs(tvix/store): fix docstring for StorePath::to_absolute_path
Change-Id: I8b36c85fe11e4fb62e5d28f6900cd80d89a5cc41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7854
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
0f7e4bec66 feat(tvix/store): rename and expose StorePath::validate_name
Change-Id: I8a16c214c7c644756d9d54187beba8c80ccfb39c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7853
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Vincent Ambo
43d83274e6 chore(tvix/store): derive Clone for StorePath
Change-Id: Id325afa766b0bd2a3d04cf538f13e77692a22687
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7829
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-14 09:48:37 +00:00
Florian Klink
43f6aec384 feat(tvix/store): implement PathInfoService with sled
This uses [sled](https://github.com/spacejam/sled) to store PathInfo
objects.

Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-12 10:11:57 +00:00
Florian Klink
cfa42fd19a chore(tvix/store): update ParseStorePathError messages
Change-Id: I49799393a1e35b4475566819111beb57a628c555
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7801
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-12 10:11:57 +00:00
Florian Klink
ed3b28aa02 chore(tvix/store): update ValidatePathInfoError messages
Change-Id: I8f16e25d998a74ac110ae99eec0edbbfd720c8dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7800
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-12 10:11:57 +00:00
Vincent Ambo
3e03e59893 refactor(tvix/store): move protobuf build config one level up
This embeds the build config directly at the point where `Cargo.nix`
is imported, making it transparent to library consumers.

Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 17:57:06 +00:00
Florian Klink
144d010515 feat(tvix/store): document StorePath a bit more.
Change-Id: Ifab28d97ddc22a2073c5df5e6e2cefb51b4b9191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7777
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 15:47:46 +00:00
Florian Klink
c89af03a03 refactor(tvix/store): rename NixPath to StorePath
As discussed in #tvl, this is a more common term for it.

Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 15:39:35 +00:00
Florian Klink
f73ab76fce feat(tvix/store): add NixPath::to_absolute_path
This provides a function returning a string starting with the store
path prefix, the counterpart of `from_absolute_path`.

Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 15:39:35 +00:00
Florian Klink
8449f6cd0c feat(tvix/store): implement Nixpath::from_absolute_path
This allows constructing a NixPath from an absolute path. It pops off
the STORE_DIR prefix and the trailing slash and returns an error if it
couldn't be found.

Change-Id: Ib540e353c63cc247ac15e20414b0db2caf695ef4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7751
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 20:23:29 +00:00
Florian Klink
8e2b1de3d5 feat(tvix/store/nixpath): DIGEST_SIZE public, use more consts
Rename PATH_HASH_SIZE to DIGEST_SIZE.

It's a digest, not hash (we don't necessarily have the internal hash
state anymore), and the fact it's about (Nix)Paths is already visible
from the module name.

Also expose ENCODED_DIGEST_SIZE, so we don't need to do the calculation
inside from_string() method, and it becomes more clear this is a
constant.

Change-Id: I0e7577dd7a6e503039037b986313b214e995d826
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7725
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Tested-by: BuildkiteCI
2023-01-03 13:03:07 +00:00
Florian Klink
2df642231c feat(tvix/store/nixpath): expose digest and name fields
These can be accessed directly.

Change-Id: I71dc84f982820d53f319efefbed9b9055034954d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7724
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 13:03:07 +00:00
Florian Klink
0b56d9f21b feat(src/proto): add PathInfo.validate()
This provides validation of PathInfo messages, and ensures the output
hashes are properly parsed from the root node names.

NixPath already has a more extensive test suite for various wrong
NixPaths, so it's omitted from here.

Change-Id: I5d69118df5816daabb521ddb19d178bddd1caacf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7684
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-03 13:03:07 +00:00
Florian Klink
ceb2c0ba89 chore(tvix/store): make importable
This allows other crates to import tvix_store.

Rename the bin crate to tvix-store-bin, to avoid having multiple crates
with the same name (https://github.com/rust-lang/cargo/issues/6313)

Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 13:03:07 +00:00
Florian Klink
d0bbc8c821 chore(tvix/store/nixbase32): address clippy
Change-Id: Ib47a55e39ed752492b9732439de5f327a7fa601e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7723
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 12:06:55 +00:00
Florian Klink
ecd943ea6a chore(tvix/store/main): address clippy
Change-Id: Ie400a7109d3aa6b66153c0ab25a38abf06984cbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7722
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-03 12:06:54 +00:00
Florian Klink
319c03f634 feat(tvix/store): add logging with tracing
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/
tracing.

Annotate all method handlers with an instrument macro, and warn! a
message for them being unimplemented.

Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-30 20:25:09 +00:00
Florian Klink
0bf2b0ef11 feat(tvix/store): implement reflection
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools
like evans automatically discover available services, without having to
specify the path to the .proto files client-side.

It's behind a reflection feature flag, which is enabled by default.

Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-30 20:25:09 +00:00
edef
51243007f6 refactor(tvix/store): make nixbase32 reversal more idiomatic
Change-Id: Ibe550f9573b0f45ee95453b50c7510e49b07c719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7685
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-30 16:40:17 +00:00
Jürgen Hahn
60bdf619a3 feat(tvix/store): refactor digest conversion
This refactors how the original digest type (Vec<u8>) is converted
to [u8; PATH_HASH_SIZE].

Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-12-30 16:02:38 +00:00
Florian Klink
58f5ff2c17 feat(tvix/store): initial dummy implementation
This replaces the hello world example from tvix-store with an actual
gRPC endpoint, implementing all of BlobService, DirectoryService and
PathInfoService.

All RPC methods currently respond with the unimplemented gRPC status.

Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-30 15:21:38 +00:00
Florian Klink
d22a9c8610 chore(tvix/store/protos): more idiomatic go
Pointed out by edef in
https://cl.tvl.fyi/c/depot/+/7648/comment/4551ba4b_e89ade36/#, thanks!

Change-Id: I6b0d317bb0210521622483cdf4cb557bc637a100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7709
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 15:18:02 +00:00
Florian Klink
4036740ca7 feat(tvix/store): parameterise validate_digest error
Similar to cl/7682, we also want to make that error configurable.

Change-Id: I64f1a4570b3d75af4741abe10c2855959766e107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7708
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 14:43:19 +00:00
Jürgen Hahn
5520bf3682 feat(tvix/store): add nixpath
This implements the NixPath structure. NixPath allow to parse a string to a nix path. If the parsing fails, a DecodeError will be raised.

Change-Id: I28363cdcfb27f04bf21a11c0d130b461667e3720
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 14:39:59 +00:00
Florian Klink
357c4d4836 feat(tvix/store): add nixbase32 mod
This implements the nix-specific base32 encoding and decoding, exposing
a subset of the API that the data-encoding crate provides.

Nix uses a custom alphabet, no padding, and encodes bytes in reverse
order. The latter one is the reason we can't just use the data-encoding
crate directly.

Three odd corner case tests ported over from go-nix failed. We opened
b/235 to further investigate.

Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-12-30 13:31:46 +00:00
Florian Klink
5ba47a2bc3 feat(tvix/store): parameterise validate_node_name error
We'll be using validate_node_name in other places in a bit, where
returning a ValidateDirectoryError is not appropriate.

Accept a function mapping a string to error as a second argument, and
pass ValidateDirectoryError::InvalidName at the current call sites.

Change-Id: I45cbb0deb4763061ad912c6b18a112c727795a17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7682
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2022-12-29 21:55:45 +00:00
Florian Klink
e97a429e73 fix(tvix/store): fix typo
Change-Id: I351a2bd4e007443aef1b97f50aecffc095ac60b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7681
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 21:51:43 +00:00
Florian Klink
6ab8320f07 chore(tvix/store): address clippy warnings
The only warnings left are unused warnings, but that'll change once we
have a real implementation, and not just tests.

Change-Id: I28912281b5e66735be37e999cc8ef4b8b09028fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7669
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-29 12:07:57 +00:00
Florian Klink
b0879917d4 chore(tvix/store): trim Cargo.toml
lazy_static is only used in tests, and anyhow isn't used at all (yet).

This can be dropped.

Change-Id: Ic41ff3f9bb93cfa600c3485e85464f78a3976504
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7668
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-29 12:07:57 +00:00
Florian Klink
ff71366f41 chore(tvix/store): move tests into separate file
Move them from the bottom of src/proto.rs to its own src/tests/mod.rs.

Also drop the test_ prefix, this is not golang.

Change-Id: I2e0b6b9812264f3d9721c0766936f08157fadc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7667
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 12:07:57 +00:00
Florian Klink
c4ee942b1c feat(tvix/store/protos): rename Get to Read, add Stat method
Stat exposes metadata about a given blob,
such as more granular chunking, baos.
It implicitly allows checking for existence too, as asking this for a
non-existing Blob will return a Status::not_found grpc error.

The previous version returned a Status::not_found error on the Get
request too, but there was no chance to prevent the server from starting
to stream (except sending an immediate cancellation).

Being able to check whether something exists in a BlobStore helps to
prevent from uploading in first place.

The granular chunking bits are an optional optimization - if the
BlobStore implements no more granular chunking, the Stat response can
simply contain a single chunk.

Read returns a stream of BlobChunk, which is just a stream of bytes -
not necessarily using the chunking that's returned in the reply of a
Stat() call. It can be used to read blobs or chunks.

Change-Id: I4b6030ef184ace5484c84ca273b49d710433731d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7652
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
1c15154b83 feat(tvix/store): make blobstore stream chunks
This changes the RPC methods to return/consume a stream of chunks, instead of a
very big message containing the whole blob, to keep message sizes in manageable
sizes (less than 4MiB).

Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
d973c9772d chore(tvix/store/protos): add PathInfoService::CalculateNAR()
Expose the NAR calculation to a separate `CalculateNAR` method, which
responds with the NAR size and sha256 hash.

Contrary to what cl/7618 and cl/7620 initially did, don't add different
other request types.

In the CalculateNARResponse message, there's now some duplication in the
(optional) `narinfo` field of a PathInfo, but I'm not entirely sure if
we want to drop the fields from there yet.

Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
ce7be00994 chore(tvix/store/protos): drop non-sha256 NAR hashes
Nix actually doesn't support anything else than sha256 NAR hashes, so
there's little reason to communicate anything else in here.

Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I760370bd6cabd02028e001a74c454ef9296eb600
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7619
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 13:55:18 +00:00
Florian Klink
27bfc8cfcd chore(tvix/store/protos): move (Root)Node into separate message
This is gonna get used in another place in a second.

Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I347c11c8d24379628b7ed09d2c90670c576e686a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7617
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 13:55:18 +00:00
Florian Klink
7163d3ad37 feat(tvix/store): implement Directory::validate()
Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 10:58:50 +00:00
Florian Klink
0c7e545fd0 chore(tvix/store): use DUMMY_DIGEST, too
Missed from cl/7649.

Change-Id: Id2d382567f91cba9cdd792fe44bc0a83d040ec69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7656
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 10:58:50 +00:00
Florian Klink
f930c91086 chore(tvix/store): use Directory::default in some more places
Change-Id: Iebad5e0b38f9506933fa855e1bfb2611d27ff572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7655
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 10:58:50 +00:00
Florian Klink
35b18b2fdb chore(tvix/store): use lazy_static and dummy digests
While there's currently nothing in here checking the size of the digest,
we should use something that passes the to-be-introduced validate()
function.

Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 21:31:33 +00:00
Florian Klink
dfd9286f68 feat(tvix/store/protos): implement Directory.Validate()
Validate thecks the Directory message for invalid data, such as:
- violations of name restrictions
- invalid digest lengths
- not properly sorted lists
- duplicate names in the three lists

Change-Id: I8d43a13797793c64097e526ef3bd482c9606c87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7648
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 21:31:33 +00:00
Florian Klink
c3fb6d2218 feat(tvix/store/protos): implement Size() and Digest() for Directory
This adds Size() and Digest() functions for the golang version.

Change-Id: If71445a9bb26100bb4076ac4f5c96945b33919f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7325
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-27 21:31:33 +00:00
Florian Klink
a91e6334ff feat(txix/store/protos): init go.mod, generate .pb.go files
This allows importing the generated .pb.go files into other go projects.

I initially looked at buildGo.protos, but it doesn't work for multi-.proto
files, and actually having LSP support for the generated structs is nice, too.

Change-Id: Idbd448008010790a10a0ea42e4059dbb609eaf1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7322
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
2b9330911a feat(tvix/store/protos): add go_package option
Change-Id: I0898b8a0a78e704219da38e5acaabef1e640d4e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7321
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2022-12-26 11:38:52 +00:00
Florian Klink
e05e8bdd03 docs(tvix/store): add README, document services and store model
These are intended to help digest the protocol definitions for tvix-
store, and how they tie into the whole concept.

Co-Authored-By: Vincent Ambo <mail@tazj.in>
Change-Id: Ic1ba3ba41ef599209453f15d0ac2e07a6144bcca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7439
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
3676ce8d6d feat(tvix/store): use tonic-build for protos
While prost-build already exposes protobuf message types as structs, we
actually need tonic-build too, to be able to get traits for all the RPC
services defined in the proto files.

Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
2e04f68df8 feat(tvix/proto): add rpc_directory.proto
This provides an interface to retrieve and upload single Directory
messages, or a DAG of them.

Change-Id: Id9e7084bd697d843649a122da2c992a3e36d808c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7137
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 14:35:32 +00:00
Florian Klink
870cd533dd feat(tvix/proto): add rpc_pathinfo
This defines a service that can be used to upload and retrieve metadata
of nix paths.

Change-Id: Id86eb531ce4ae316adb15934b0d1386a14ba2132
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 14:35:32 +00:00
Florian Klink
5f9796cf94 feat(tvix/proto): add rpc_blobstore
This defines a service that can be used to get and put content-addressed
chunks of data.

Change-Id: I36cf2278ed1daf71848c04fdfd14450b2268c5de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-15 19:10:26 +00:00
Vincent Ambo
3aca3d3bba refactor(tvix): build Rust projects using crate2nix
Introduces granular dependency builds using crate2nix, bootstrapped
off the generated configuration from the newly introduced
workspace (see cl/7533).

This commit checks in the generated Cargo.nix file which can be
regenerated with a parameterless invocation of `crate2nix generate` in
`//tvix`. I tried generating this in IFD, but it turned out to be
harder than what seemed worthwhile for now.

In this setup, the various build targets for Rust projects end up
being attributes of the imported `Cargo.nix` file at the `tvix.crates`
attribute. These still lack configuration, however, which has been
fixed in the various `default.nix` files of individual projects.

Note that we (temporarily) lose the ability to build tvix-eval's
benchmarks in CI. I haven't figured out what magic incantation summons
them from the void again ...

The `eval-okay-readDir` tests from both test suites have been disabled
because they fail for unknown reasons when run in this new derivation.
Somebody will have to debug it!

Change-Id: I2014614ccb9c8951aedbd71df7966ca191a13695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7538
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-12-15 17:26:45 +00:00
Vincent Ambo
d9d627cdf0 refactor(tvix): share a Cargo.lock file between Rust projects
This relates to the (abandoned) cl/7256.

Introduces a Cargo workspace at //tvix that is primarily intended to
be used as a workaround for the annoying Nix+Rust tooling while having
a consistent set of dependencies.

This is driven in part by a desire to adopt crate2nix and get more
granular Nix builds for Tvix's Rust projects, and in part by a need to
split //tvix/eval into something providing the CLI (REPL etc.), and a
library providing eval, without significantly altering the structure
of build targets.

To accomplish this the workspace has been designed to allow projects
to remain independent build targets. I want to avoid lumping all the
projects together - something like //tvix/eval should always be
independent of other parts of tvix.

A helper function in //tvix/default.nix lets downstream naersk
projects construct a sparse root for the project which combines the
workspace's `Cargo.lock` with the project's own `Cargo.toml`.

Note that cargo commands in the workspace itself require the build
dependencies of _all_ projects to be present, which is currently a bit
annoying to accomplish.

This introduces some breakage:

1. It breaks usage of rust-analyser without being in a shell with the
   dependencies of *all* Tvix projects, as it is not capable of
   respecting only the subset of dependencies for a part of the
   workspace.

2. It is no longer possible to run tests using `cargo test`, as the
   test generation crate we use does not work with workspaces:

   https://github.com/frehberg/test-generator/issues/6

   This still works in the Nix build as we construct a Cargo project
   that looks like it's not in a workspace there. Until somebody fixes
   that crate / writes a new macro / does something else with the test
   suite, the way to run the tests is through the Nix build.

Long-term we'll probably want to get rid of cargo completely, it's
just a big wart and most tooling works without it if correctly
configured, but we don't have time for that now.

Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-15 17:26:45 +00:00
Florian Klink
0a4a0d1c6b feat(tvix/store): add size() and digest() functions for Directory
Change-Id: I3a9e3c77da1d5d7bfb93da0cd7bce8d48dbc8cd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7254
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-04 10:41:39 +00:00
Florian Klink
a97e3726df feat(tvix/store): build rust proto files
This uses prost-build to build the proto files.

Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-04 10:41:39 +00:00
Florian Klink
c432aae9b3 feat(tvix/store): init
Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-04 10:41:39 +00:00
Florian Klink
12413a669f feat(tvix/store): add pathinfo.proto
This adds the PathInfo message, which hosts information about a Nix
Store path, mapping to either of a {Directory,File,Symlink}Node.

Change-Id: I79d871b6fad450d6a4ae4101fb72c51f9a83471f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7132
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-04 10:41:39 +00:00
Florian Klink
ff5e041fdb chore(tvix/store): move castore.proto
This moves the castore.proto file into the //tvix/store/protos directory. Per-
component protos might make more sense, than a "tvix-wide" proto directory.

Change-Id: Ie728210174b041e9285a0e2ac605d715d0f0cbda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7285
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-04 10:41:39 +00:00