Commit graph

506 commits

Author SHA1 Message Date
Florian Klink
64beaab163 refactor(tvix/store/protos): remove unused mustBlobDigest function
Change-Id: I79a73de0cb06abcf16a6579ae298876004ab0ba8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9719
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-13 14:43:26 +00:00
Florian Klink
9b7629826f refactor(tvix/castore): factor out node checks
Implement `validate()` on `node::Node`, and call it from PathInfo's
validate() too. Node-related errors are moved to a ValidateNodeError
error type.

This additionally adds some more validations for symlink targets (they
must not be empty, and not contain null bytes).

Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-12 21:05:48 +00:00
Florian Klink
c9e90b4dd7 refactor(tvix/store/fs): use while let statement
Suggested by clippy.

Change-Id: I686b1abcb40fab42017972a3f814fae29fa70c25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9712
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-12 20:46:24 +00:00
Florian Klink
4da906bf34 refactor(tvix/store/proto): merge two match statements into one
Change-Id: I3daca008dff5527169f5916f4845234e8f3263cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9711
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-12 20:46:24 +00:00
Florian Klink
3dd5ba042b refactor(tvix/store): clippy lint
Change-Id: I03aa5c6cb0167ff6cb7312d81d1e43116fcb6d70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9710
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-12 20:46:24 +00:00
Florian Klink
a9e5a7cec9 feat(tvix/store): enable tls[-roots] feature in tonic
This teaches tonic to speak https, so `grpc+https` becomes a working
connection string.

Change-Id: I9c550ea665b829725b545a164cf6f78587bd5213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9714
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-12 19:18:59 +00:00
edef
be32ab1eb2 fix(tvix/store/fs): fix typo
Change-Id: I132987dda1073dcc0b30e50e26682d6c8724d988
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9689
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-12 09:51:30 +00:00
Florian Klink
e64e97ee23 feat(tvix/store): validate nar_sha256
Change-Id: I4c4dcdb75ea7748f2ab01a0bab218596b90b7b58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9648
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-11 15:29:08 +00:00
Florian Klink
8ced43f3c7 feat(tvix/store/protos): validate NarSha256
Change-Id: I016d1b020b04850f0dca68c2ea96643230dada10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9649
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-11 15:29:08 +00:00
Florian Klink
fb04645b0d docs(tvix/store/protos): explain a bit more what we're doing
Explain a bit better what we do in the codeblock below,
and add a newline to make it clear we don't just document the first
branch.

Change-Id: Ifb142d9984e6d2cbca648525c10298f2fcbdfedf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9645
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
2023-10-11 11:43:29 +00:00
Florian Klink
d94749ac22 refactor(tvix/store/protos): have Export accept root node
We don't need the full PathInfo message, only the root node.

Change-Id: I667045ed766875dfbf8ac126a50b02baa2df67a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9604
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-10 20:22:47 +00:00
Florian Klink
e6ba84ea50 feat(tvix/store/protos): add Export
Export will traverse a given PathInfo structure, and write the contents
in NAR format to the passed Writer.
It uses directoryLookupFn and blobLookupFn to resolve references.

This is being moved over from nar-bridge. We need to keep the code there
around until we can bump go.mod to storev1 with this merged, but the
tests can already be moved entirely.

Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
2023-10-10 18:55:49 +00:00
edef
d23fe6ee20 feat(tvix/store): use tvix_compat::nar::writer::async
Change-Id: Iad36872244df6f2225a2884f6b20cacd8f918b31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9619
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
2023-10-10 17:33:33 +00:00
Florian Klink
d1bccfe82f chore(tvix/*store): use google.golang.org/protobuf
The other one is deprecated, according to
https://pkg.go.dev/github.com/golang/protobuf/proto.

Change-Id: I502d5bfa7431df0dea3d564f1e0ecfea14c90e10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9597
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:07 +00:00
Florian Klink
28cd4b1a2f feat(tvix/nar-bridge): add ToNixNarInfo()
Convenience function, moves all code converting from a PathInfo struct
to to go-nix's NarInfo.

Change-Id: Idf0dcc38675674563f2dfd3286a4a55fa2a24a82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9593
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:02 +00:00
Florian Klink
92481825b3 test(tvix/store/protos): add go tests for Validate()
We already have validation tests for Rust, let's add the missing ones
for golang too.

Change-Id: Iaf3a3e1ee72d5647da3f2aa977d6e0d0379b2ce5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9595
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-09 22:01:59 +00:00
Florian Klink
a464088d2e test(tvix/store): use tokio-retry for exp backoff
Rather than using this loop, use exponential backoff while waiting for
the socket path to be created.

Change-Id: I3056b1525784cd712b1d81f84876c9ca0be10dc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9569
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 22:50:42 +00:00
Florian Klink
c0376995c9 refactor(tvix/store): simplify test_valid_unix_path_ping_pong
We don't need to spawn two tokio runtimes anymore, and can do the URL
parsing at once, too.

Change-Id: I7885a894bb1250cd087d4e1893e3e73b631331da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9567
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 22:47:38 +00:00
Florian Klink
4b27cdbc57 docs(tvix/store/bin): drop outdated TODO
We already use store_path::build_nar_based_store_path as a helper
function down there.

Change-Id: I46842bb0a2f4cd5de7dcfc3c4b6aae399618667a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9571
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 22:41:32 +00:00
Florian Klink
c63c55f052 chore(tvix): bump tokio[-util], tonic, prost
This bumps all these dependencies to their more recent versions.

We have to enable the `macros` tonic feature, and looks like we should
also enable `rt`, not only `rt-multi-thread`.

Change-Id: Icc3600848fca0bacce24f4889d088e75711594ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8984
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 21:53:20 +00:00
Florian Klink
09328c79d5 feat(tvix/store/listener): implement connect_info()
For Unix and TCP streams, we can actually give more info than just ().

Change-Id: I80ca261b9e7a880389ec2029ea5893180314f0b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9561
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 07:44:13 +00:00
Florian Klink
31f28b6105 chore(tvix/store): drop tonic-mock from dependencies
It's not used in here.

Change-Id: If1dbf360ec37b649c27d3a9bf11f7bf0ca947631
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9558
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-08 07:17:54 +00:00
Connor Brewster
9019d8568e fix(tvix/store): Remove virtiofs from default features
vhost-user-backend doesn't support macOS yet, so the virtiofs features
will not work on macOS. This removes it as a default feature which makes
`cargo build` work out of the box on macOS.

The `virtiofs` feature is enabled for Linux when building via Nix, but
if being built by cargo directly, the feature must be enabled via a
cargo flag.

Change-Id: I2aaca9582f8e3dbcf9ee5f1b9831d614909f3799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9555
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-10-07 02:34:00 +00:00
Florian Klink
96c0b3f069 refactor(tvix/store/proto): use NamedNode trait
This saves us writing the name parsing code three times. We can also
delay parsing until we did other (cheaper) checks.

Change-Id: I1abe3f20dba4215b38839cf7466297e028d64656
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9548
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 18:58:55 +00:00
Florian Klink
600815c168 feat(tvix/store/protos): add Validate() method to pathinfo.go
This is very similar to the Rust counterpart.

Change-Id: I40d51aaac3fcf7f52e5896587e561bc2377f6269
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9549
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 18:42:58 +00:00
Florian Klink
d45d6de561 test(tvix/store/pathinfo): add more tests for references
This should cover all error cases produced.

Change-Id: If31816d9b087551d86d7913df55df8f9f44bb554
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9546
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-05 10:25:16 +00:00
Florian Klink
1f03a520a9 feat(tvix/store): add validation for references
This validates the size of reference digests in the PathInfo message,
as well as inside the narinfo submessage. If narinfo is set, they need
to parse to StorePath, and have the same digest there as in the PathInfo
message.

`proto::tests::pathinfo::validate_references` needed to be updated,
because we actually did not populate the proper references before.

Change-Id: I9545b2487aab9fe0d229c26aceba5ddc5e6daafd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9545
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-05 09:52:19 +00:00
Florian Klink
7706a8f224 refactor(tvix/store): mv *Err*::Invalid{Node,}DigestLen
There's other digests in the PathInfo structure, that also might have
wrong digest lengths. Rename this to give some room for them, and update
the error message a bit as we go.

Change-Id: I06562664721156e658f2ed14ba1de907377d284b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9543
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-05 09:52:13 +00:00
Brian Olsen
cfb810d81a fix(tvix/store): Fix FUSE support on MacOS
This partially fixes b/312 and gets FUSE to work again on MacOS.

It is mostly small type changes and an update to fuse-backend-rs because
upstream currently doesn't work with MacFuse. It also sets the default
FUSE thread count on MacOS to 1 because otherwise the mount command will
hang when shutting down as only one thread gets ENODEV and all the others
just keep blocking.

Change-Id: Ifb3c4268caf296c487049c1dc4618acb32497f44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9490
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-02 15:46:47 +00:00
Brian Olsen
5c2cad0ac4 fix(tvix/store): FuseDaemon unmount is blocking async runtime
The unmount method in FuseDaemon calls join on a bunch of threads and that is
a blocking call but it is called from an async context in the tvix-store
binary.

This change wraps the call to unmount in a spawn_blocking.

Change-Id: If89183b4a3f890874e75f5faf90cd24cb18da1e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9489
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-02 12:50:16 +00:00
Florian Klink
89a8306ded fix(tvix/store): rename Read method in Readme
This call was renamed to `Read` a while ago, but I forgot to update
the README.

Change-Id: Ibcb97fe2520a5e9bb8444ec1adade6890dda4a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9476
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-26 15:24:32 +00:00
Florian Klink
d8f5c5b109 fix(tvix/store): allow building with Nix on Darwin
Curently both fuse and virtiofs features fail to build on Darwin.
Disable them for now until fixed.

Change-Id: Ic99fa5e3ebec8a7072ba884101a5c18652ed409c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9473
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-09-26 10:07:41 +00:00
Florian Klink
6b02800e48 chore(tvix/[ca]store): allow building without tonic-reflection
- rename the feature to `tonic-reflection` in both tvix-store and tvix-
   castore.
 - set it to disabled in tvix-castore by default
 - enable it in tvix-store, and pull in tvix-castore with the feature
   enabled.

Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 10:07:40 +00:00
Florian Klink
7062fdc132 chore(tvix/store): only depend on vhost-user-fs if virtiofs enabled
Change-Id: Iaf88d43df27b87548794a8dddc9a1e08279b8ffb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9471
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 10:03:37 +00:00
Florian Klink
124c1c06de chore(tvix/store): move fuse-backend-rs back to upstream
It's in no release yet, but this is the current rev in master.

Change-Id: I9e4bc2fde26e85c4182bd66b3e566c2d957072ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9469
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-09-26 09:51:02 +00:00
Connor Brewster
f8746fcab0 feat(tvix/store): Support listening on UNIX domain sockets
This adds support for listening on UNIX domain sockets via the
tokio-listener crate. The crate will automatically determine whether to
start a TCP or UNIX domain socket server based on the listen address.

Unfortunately, it's not compatible with tonic right out of the box so I
added some wrapper types to implement the necessary traits to make
things work. We should investigate upstreaming a `tonic` option to the
tokio-listener crate which implements the relevant `tonic` traits.

Example:
```
$ tvix-store daemon -l /run/tvix-store.sock
INFO tvix_store: tvix-store listening on /run/tvix-store.sock

$ tvix-store mount -l /mnt/tvix --blob-service-addr grpc+unix:///run/tvix-store.sock --directory-service-addr grpc+unix:///run/tvix-store.sock --path-info-service-addr grpc+unix:///run/tvix-store.sock

$ ls /mnt/tvix
```

Change-Id: I91c4a4b0c5a177b3b90e6c01a4e5d263130e6bdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9429
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-25 17:41:37 +00:00
Connor Brewster
993c505cdb feat(tvix/store/fs): Add support for virtiofs backend
This adds a virtiofs daemon implementation which hooks into the existing
tvix-store filesystem implementation that is used for FUSE.

This allows adding the filesystem to a microvm without having to set up
FUSE inside the guest.

Change-Id: If80c36c9657f2289853e8d9a364bf4f1f7b7559c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9344
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-24 17:05:21 +00:00
Florian Klink
e13c86d916 chore(tvix): re-enable {ca,}store-protos-go
This couldn't be enabled before until go get was able to fetch the
modules again.

Change-Id: I82db7d22e497661ea2a0c8a45c0c865fcc0a1f18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9406
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 17:36:05 +00:00
Florian Klink
32f41458c0 refactor(tvix): move castore into tvix-castore crate
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.

Notable changes:
 - `fixtures` and `utils` had to be moved out of the `test` cfg, so they
   can be imported from tvix-store.
 - Some ad-hoc fixtures in the test were moved to proper fixtures in the
   same step.
 - The protos are now created by a (more static) recipe in the protos/
   directory.

The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)

Bug: https://b.tvl.fyi/issues/301

Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:51:21 +00:00
Florian Klink
d8ef0cfb4a docs(tvix/castore): rename traverse_to -> descend_to
With the move of this code out into castore it has become apparent this
is a general descent inside the castore. Concerns like making sure the
whole Directory closure has been fetched/is fetched initially is nothing
this layer needs to worry about. We can handle this during substitution
of a new PathInfo, once there's store composition.

Closes b/270.

Change-Id: I661ed08e54bc81478e032cfb9abeb23e5b337fbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9373
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:25 +00:00
Florian Klink
7c9a52e3ee chore(tvix): remove some unused crate dependencies
Change-Id: I31b0e7be3a2ebc268c6f4045ba282e8fb1c72a64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9375
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:24 +00:00
Florian Klink
3d064f714c docs(tvix/store/pathinfosvc): fix copypasta error
This is not a simple get and put returning digest.

You currently can get by output hash, and a put gives you back the
(possibly modified) PathInfo message.

Change-Id: I43174d87a764417b8620c4d5bdac058cc25b0373
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9371
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-22 09:14:33 +00:00
Connor Brewster
37a348b4fa refactor(tvix/store): Asyncify PathInfoService and DirectoryService
We've decided to asyncify all of the services to reduce some of the
pains going back and for between sync<->async. The end goal will be for
all the tvix-store internals to be async and then expose a sync
interface for things like tvix eval io.

Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-21 17:58:23 +00:00
Connor Brewster
7e737fde34 refactor(tvix/store/fs): Separate FUSE and filesystem code
In prepration for adding virtiofs support, I thought it would make sense
to split out the filesystem implementation from FUSE itself.

The `fs` module holds the tvix-store filesystem implemetation and the
`fuse` module holds the code to spawn a FUSE daemon backed by multiple
threads.

Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-20 14:27:53 +00:00
Connor Brewster
6b7c936bc5 refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rs
This switches the FUSE implementation from fuser to fuse-backend-rs.
fuse-backend-rs is designed to work with both FUSE and virtiofs.
Virtiofs support will make it possible to plug the tvix-store into a
microvm and have `/nix/store` access without having to setup FUSE inside
the guest.

Additionally fuse-backend-rs has nice support for running multiple FUSE
threads and has some async support.

The goal of this commit is to mechanically switch over to
fuse-backend-rs with minimal changes. I did have to add some locks here
and there because fuse-backend-rs uses `&self` on all methods whereas
fuser uses `&mut self`. `&self` is required for concurrent access to the
FUSE server, so this makes sense.

We can consider switching to concurrent maps and use some other
techniques to reduce lock contention and critical section size.

Issue: https://b.tvl.fyi/issues/305

Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-20 14:21:32 +00:00
Florian Klink
ca25acf11b chore(tvix/store/protos): drop unused fields for now
This wasn't removed yet, and no code is using/populating it so far.

It's confusing, let's update it to the state of things now, and re-
introduce it once we get there.

Change-Id: I68f5ba17a8eee604d8ccd82749da7c8be094cb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9351
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-18 10:33:30 +00:00
Florian Klink
bf2fe88a5c fix(tvix/store/fuse): fix executable bit
We were blindly returning 0o444 for all regular files, but regular files
with executable bit need to be 0o555.

This wasn't spotted because stat'ing executable files was not part of
the test suite, it's now added.

Change-Id: I04c69784053e7e43d838c01bb288f2df48f40b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9345
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 10:33:30 +00:00
Florian Klink
da6cbb4a45 refactor(tvix/store/blobsvc): make BlobStore async
We previously kept the trait of a BlobService sync.

This however had some annoying consequences:

 - It became more and more complicated to track when we're in a context
   with an async runtime in the context or not, producing bugs like
   https://b.tvl.fyi/issues/304
 - The sync trait shielded away async clients from async worloads,
   requiring manual block_on code inside the gRPC client code, and
   spawn_blocking calls in consumers of the trait, even if they were
   async (like the gRPC server)
 - We had to write our own custom glue code (SyncReadIntoAsyncRead)
   to convert a sync io::Read into a tokio::io::AsyncRead, which already
   existed in tokio internally, but upstream ia hesitant to expose.

This now makes the BlobService trait async (via the async_trait macro,
like we already do in various gRPC parts), and replaces the sync readers
and writers with their async counterparts.

Tests interacting with a BlobService now need to have an async runtime
available, the easiest way for this is to mark the test functions
with the tokio::test macro, allowing us to directly .await in the test
function.

In places where we don't have an async runtime available from context
(like tvix-cli), we can pass one down explicitly.

Now that we don't provide a sync interface anymore, the (sync) FUSE
library now holds a pointer to a tokio runtime handle, and needs to at
least have 2 threads available when talking to a blob service (which is
why some of the tests now use the multi_thread flavor).

The FUSE tests got a bit more verbose, as we couldn't use the
setup_and_mount function accepting a callback anymore. We can hopefully
move some of the test fixture setup to rstest in the future to make this
less repetitive.

Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-18 10:33:30 +00:00
Florian Klink
51b8571c48 feat(tvix/store): improve error message formatting
WrongSorting(Vec<u8>) actually encodes the name of a node, so if we can,
we want this to display it as a string (and fall back to the base64
encoding if we can't).

Before:

> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: [108, 111, 99, 97, 108, 101] is not sorted

After:

> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: locale is not sorted

Change-Id: I68420c53a89cb1aa96e4bdce414366cebcb7915f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9350
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-17 13:14:03 +00:00
Florian Klink
54bb63dea3 chore(tvix/store): allow non_snake_case
Some of the tonic-generated types create new warnings.

Change-Id: Ib538bfd7d90d857d80a129fbbdf3a38c89cc04ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9349
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-17 13:14:03 +00:00
Florian Klink
3816eaa5e9 feat(tvix/store): add simple integration test
This imports the docs folder into the tvix store, and ensures it comes
up with the same store path as Nix.

While we validate hashes in general through tvix-cli output path checks
already, it doesn't include the tvix-store CLI itself.

See https://cl.tvl.fyi/c/depot/+/9329/comment/339f0720_524f0104/
for context.

Change-Id: I239ce5b6a07cb962b242142ab716693359b8674c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9338
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 21:00:54 +00:00
Florian Klink
32f2f8b618 feat(tvix/store/import): print store path to stdout
This allows comparing the output in an integration test.

Change-Id: I8bb2254e18e90005a4f1b30fd47ef69642e3732e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9337
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 21:00:54 +00:00
Florian Klink
b52bb6e791 fix(tvix/store): log to stderr, not stdout
This leaves some space in stdout to provide actual meaningful output.

Also, rename print_node to log_node because that's what it does, it's
using the logger to log out more detailed info.

Change-Id: Ic64a6330dbfcdc63eb4198067a5c5e47b841b9a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9336
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-09-15 19:38:23 +00:00
Florian Klink
c77c7ae9a5 fix(tvix/store/blobservice/seeker): fix debug assert
We were asserting absolute_offset > self.pos, but that's not true for
both being zero.

Ramp up the tracing bits a bit, so we actually can see this in the debug
logs.

Change-Id: I21693bcafab227549b19cd6f1215d2f8dee77ecc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9292
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-12 12:15:08 +00:00
Brian McGee
1549f020ee fix(tvix/protos): support []byte node names
The underlying protos were changed to return `[]byte` instead of `string`.

Change-Id: I5d3e5d8de0ed7200325f7ab0d62e3c10d8eb1b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9285
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-09 12:13:02 +00:00
Florian Klink
ec22a80f26 docs(tvix/store/fuse): add more comment to list_root arg
Forgot to squash this into cl/9217.

Change-Id: Id5a87cbe729592f3b53e8c329b6890519f5da319
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9272
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-08 16:13:21 +00:00
Florian Klink
7923cc19f6 refactor(tvix/store): use tokio::task::JoinHandle
This makes the inside code a bit less verbose.

I wasn't able to describe the type of the async move closure itself,
which would allow us to remove the JoinHandle<_> type annotation
entirely.

Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-05 21:13:50 +00:00
Florian Klink
f499d2e031 feat(tvix/store): fix ctrl-c handling on mount command
This enables the tokio `signal` feature, and registers a ctrl_c signal
handler, which will use the unmount handle to unmount in case a ctrl-c
signal is received.

This avoids having disconnected mountpoints when Ctrl-C'ing a
`tvix-store mount` invocation.

In case the filesystem is unmounted externally (via `umount /path/to/
mountpoint`), the future is waiting for the signal is never resolved and
the task is stopped.

Change-Id: I149f705a6cb50188177f2a6c6a5fcd77218e2a3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9218
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-05 21:13:49 +00:00
Florian Klink
f9b5fc49b1 feat(tvix/store/fuse): allow listing
This provides an additional configuration flag to the tvix-store mount
subcommand, and logic in the fuse module to request listing for the
root of the mountpoint.

Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: adisbladis <adisbladis@gmail.com>
Tested-by: BuildkiteCI
2023-09-05 21:13:49 +00:00
Florian Klink
da9d706e0a feat(tvix/store/pathinfosvc): provide listing
This provides an additional method in the PathInfoService trait, as
well as an RPC method on the gRPC layer to list all PathInfo objects in
a PathInfoService.

Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-05 20:46:42 +00:00
Florian Klink
e41b5ae3f0 refactor(tvix/store): infer more types
We don't need to explicitly describe the type of the task itself,
describing the return type of the async closure is sufficient.

Also, use io::Result<_> instead of Result<_, io::Error>.

Change-Id: I9ab3f990eb49929b0aea335b2bb07da392ab631f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9267
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-09-05 20:41:41 +00:00
Florian Klink
7bd3c42c74 chore(tvix/store): drop walkdir workaround for symlinks at root
https://github.com/BurntSushi/walkdir/pull/170 got merged, meaning we
don't need to keep our own logic in here anymore.

Our test cases already cover this.

Change-Id: Ied3043ee651c8aafa10271c1e1ca5d460fb6c0b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9269
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-05 15:19:43 +00:00
Florian Klink
3c340b28bd refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate
This trait is eval-specific, there's no point in dealing with these
things in tvix-store.

This implements the EvalIO interface for a Tvix store.
The proper place for this glue code (for now) is tvix-cli, which knows
about both tvix-store and tvix-eval.

There's one annoyance with this move: The `tvix-store import` subcommand
previously also used the TvixStoreIO implementation (because it
conveniently did what we wanted).
Some of this code had to be duplicated, mostly logic to calculate the
NAR-based output path and create the PathInfo object.

Some, but potentially more of this can be extracted into helper
functions in a shared crate, and then be used from both TvixStoreIO in
tvix-cli as well as the tvix-store CLI entrypoint.

Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-09-04 16:02:12 +00:00
Florian Klink
583cdec3d8 docs(tvix/store): address cargo doc warnings
Fix some broken link references.

Change-Id: I69c9b2b62af35bb777e4df1a01ba3181a368be47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9214
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-03 10:01:40 +00:00
Florian Klink
9bbda4421e docs(tvix/store): update rpc_pathinfo.proto comments
Don't repeat the name of the method in the description, don't repeat
things already described in request message comments.

Change-Id: I180e4792577419050947eea8fea7043861aba463
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9213
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-03 10:01:40 +00:00
Florian Klink
116a137be0 refactor(tvix/store): implement rename for node::Node
This returns a node with a new name.

Change-Id: Iebcab537f8dd63d826b9841d4d0181fcb941afdd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-02 20:08:52 +00:00
Florian Klink
1e9d262ad5 docs(tvix/store): update comment
Since the refactor to use URIs for all three services, this actually
does talk to a daemon by default.

Change-Id: Ied296772b77eef514bfcae0a9dfc50f848a1c2f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9210
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-02 20:03:34 +00:00
Florian Klink
0193f07642 refactor(tvix/nix-compat/nixhash): validate digest lengths
There was a NixHash::new() before, which didn't perform any validation
of the digest length. We had some length validation when parsing nix
hashes or SRI hashes, but some places didn't perform validation and/or
constructed the struct directly.

Replace NixHash::new() with a
`impl TryFrom<(HashAlgo, Vec<u8>)> for NixHash`,  which does do this
validation, and update constructing code to use that, rather than
populating structs directly. In some rare cases where we're sure the
digest length is correct we still populate the struct manually.

Fixes b/291.

Change-Id: I7a323c5b18d94de0ec15e391b3e7586df42f4229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9109
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-08-20 22:19:22 +00:00
Florian Klink
3ffbcc6c8a refactor(tvix/store): cargo clippy
Change-Id: I3a80560d036e7ed08036b5e9f0974080d1a30ded
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9096
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-08-20 21:51:04 +00:00
Vincent Ambo
fcda068235 feat(nix/sparseTree): add optional name argument
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.

I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.

Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-08 20:11:56 +00:00
Florian Klink
432222f098 feat(tvix/store/proto): use Bytes instead of Vec<u8>
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our
bytes field cheaper to clone.

It's a bit annoying to configure due to
https://github.com/hyperium/tonic/issues/908, but the workaround does
get the job done.

Change-Id: I25714600b041bb5432d3adf5859b151e72b12778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-22 09:03:02 +00:00
Florian Klink
a6580748aa feat(tvix/store/digests): use bytes::Bytes instead of Vec<u8>
This will save us some copies, because a clone will simply create an
additional pointer to the same data.

Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-21 19:01:50 +00:00
Florian Klink
72e82ffcb1 refactor(tvix/store): use bytes for node names and symlink targets
Some paths might use names that are not valid UTF-8. We should be able
to represent them.

We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.

While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?

Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-21 19:01:49 +00:00
Florian Klink
638f3e874d feat(tvix/store/fuse): implement open explicitly
This "reverts" commit 9f600de226 (the
initial revert of f5e291cf83).

Now with BlobService returning a BlobReader that implements io::Seek, we
can actually just call blob_reader.seek(io::SeekFrom::Start(offset as
u64)).

This means, we currently will fail to seek backwards inside a file.

Change-Id: I9c19448df6831a3537252f99210374f2126ecfc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8886
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 18:52:47 +00:00
Florian Klink
7613e2e769 feat(tvix/store/blobservice): implement seek
For memory and sled, it's trivial, as we already have a Cursor<Vec<u8>>.
For gRPC, we simply reject going backwards, and skip n bytes for now.

Once the gRPC protocol gets support for offsets and verified streaming,
this can be improved.

Change-Id: I734066a514aed287ea3db64bfb1680911ac1eeb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8885
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-21 18:52:46 +00:00
Florian Klink
42dc18353d feat(tvix/nix-compat): have StorePath accept bytes
The primary constructor for this is now from_bytes, from_string is
simply calling .as_bytes() on the string, passing it along.

The InvalidName error now contains a Vec<u8>, to encode the invalid name
(which might not be a string anymore).

from_absolute_path now accepts a &[u8] (even though we might want to
make this a OSString of some sort).

StorePath::validate_name has been degraded to a pub(crate) function.
It's still used in src/derivation, even though it probably shouldn't at
all - that cleanup is left for cl/8412 though.

Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 18:04:38 +00:00
Florian Klink
5364fcb127 feat(tvix/nix-compat): fold NameError into Error
This being a nested error makes things more complicated than necessary.

Also, this caused BuildStorePathError to only hold NameError,
so refactor these utility functions to either return Error, or
BuildStorePathError.

Change-Id: I046fb403780cc5135df8b8833a291fc2a90fd913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8972
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-21 18:04:38 +00:00
sterni
88b0545908 chore(3p/sources): Bump channels & overlays
* //tvix/store: update code generated from proto files

Change-Id: Ie6aa53e71f07a0a0e637673ba4005ed983db315d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8929
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-08 10:46:31 +00:00
DavHau
e751372f2f docs(tvix/store): various improvements
Improve change some little things I noticed while reading through it.

Change-Id: I033209eece395e5aad4e10825e8dd6c0cfe68191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8725
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-07 09:48:32 +00:00
Florian Klink
c767460440 fix(tvix/store/fuse): fix log level for a debug message
Change-Id: Ib801e46636901553d71455a739aed34e5828ca0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8888
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-02 11:40:29 +00:00
Florian Klink
2a601d26a8 docs(tvix/store/fuse/inodes): clippy lint
Change-Id: I7b1bcb9c0cca76f05271f25912d26b14152fe0c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8887
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-02 11:18:52 +00:00
Florian Klink
6b599b3763 feat(tvix/store/fuse): add test reading large file
Change-Id: Ic705d05909c59c764d68a730169e9cc2b2538d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8847
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 12:35:56 +00:00
Florian Klink
7fbf874f1c feat(tvix/store/blobsvc): add more blobservice tests
Change-Id: I3e27dfb4ce3e52974d7614814abb7b5ae4a37f8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8782
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 12:35:56 +00:00
Florian Klink
3edc580e80 docs(tvix/store/blobservice): update docstring
Change-Id: I6ae693a66530e1ecca57723a97d56b309fa0651a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8881
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 12:30:54 +00:00
Florian Klink
9f600de226 fix(tvix/store/fuse): revert "implement open explicitly"
This reverts commit f5e291cf83.

The offsets are relative to the start of the file, and as long as we
don't have BlobReaders implement seek, this will be very annoying to
deal with.

Change-Id: I05968f7c5c0ec0000597da90f451d6bb650c3e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8882
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-30 12:30:53 +00:00
Florian Klink
6604ce4e51 fix(tvix/store/blobservice): write into hasher from b, not buf
buf contains everything written so far, whereas b is the slice passed in
the current write() call. If we copy from &buf, we end up with the wrong
hash, because we keep writing the wrong data to the hash function.

Change-Id: I768d4645934a6a7d75b9c8eeba35f8f3be5edd26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8880
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 12:30:53 +00:00
Florian Klink
f5e291cf83 feat(tvix/store/fuse): implement open explicitly
This moves from stateless I/O to actually dealing with file handles,
allowing the filesystem to keep reusing existing blobreaders, instead of
opening a new reader on every read() call.

Change-Id: I3fc35c071e4aee1021c8bbd58749d082b0abd188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8834
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 11:59:48 +00:00
Florian Klink
b25d98a84e feat(tvix/store/fuse): initial implementation
This is a first implementation of a FUSE filesystem, mounting tvix-store
to a given location.

This is mostly meant as one additional lens into a store, and could be
used for builds. It's not meant to be used as a general-purpose thing.

It still has some rough edges:

 - It doesn't implement open/close, so it doesn't use file handles.
   Which means, we need to open blobs for partial reads over and over
   again.
 - It doesn't implement seek, as BlobReader doesn't implement seek yet.
 - It doesn't track "lifetimes" of inodes by listening on forget,
   meaning it might hold more data in memory than necessary.
 - As we don't have store composition (and a caching layer) yet,
   operations might be slow.

Change-Id: Ib1812ed761dfaf6aeb548443ae939c87530b7be8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8667
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-20 10:14:00 +00:00
Florian Klink
6656b865b6 docs(tvix/store): correct some cargo doc warnings
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-19 22:54:46 +00:00
Florian Klink
ad687ccaf4 chore(tvix/store/pathinfosvc): clippy
Change-Id: Ied4bed08e989791f832922da8776d2104035e28a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8812
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-19 14:29:05 +00:00
Florian Klink
b10f008b03 chore(tvix/store/directorysvc): clippy
Change-Id: Idf45aaa0f6211ac35a9a41d0f3f60dfbe1009398
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8811
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-19 14:29:05 +00:00
Florian Klink
71093a513a chore(tvix/store/blobsvc): clippy
Change-Id: Ie384bdd27e1e9282ceda83edc74ffaad387f352b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8810
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-19 14:29:04 +00:00
Florian Klink
f087e07ca1 fix(tvix/store/blobsvc): drop stray println! debug statement
Change-Id: Ica073820ea8240d42ce8b979309f881af18a8cde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8781
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-14 23:16:09 +00:00
Florian Klink
c5851fb815 refactor(tvix/store/blobsvc): make from_addr not async
A previous iteration of this code did actually connect (in the gRPC
client), which was why we had this function async.

However, as the connection there is now lazy too, we can drop the
asyncness in this function.

Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 23:16:09 +00:00
Florian Klink
35bff2bda6 refactor(tvix/store/pathinfosvc): add from_addr
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-14 23:15:08 +00:00
Florian Klink
bb7c76739a feat(tvix/store/directorysvc): add from_addr
Add --directory-service-addr arg to tvix-store CLI.

Change-Id: Iea1e6f08f27f7157b21ccf397297c68358bd78a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8743
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 23:15:07 +00:00
Florian Klink
a1acb5bcb3 feat(tvix/store/blobsvc): add from_addr
This allows constructing blob stores with a URL syntax at runtime,
by passing the --blob-service-addr arg.

We probably still want to have some builder pattern here, to allow
additional schemes to be registered.

Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-14 20:44:36 +00:00
Florian Klink
c947a73b8e chore(tvix/store/pathinfosvc): remove GRPCPathInfoService::new
There's very little reason to instantiate a GRPCPathInfoService in a
context where we are not already in a tokio context.

Change-Id: Ib81d649387717cb98de8a8039f92472f727b10c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8755
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 21:33:59 +00:00
Florian Klink
011fbe2214 chore(tvix/store/directorysvc): remove GRPCDirectoryService::new
The only place where we did use new was also already where we've been in
a tokio context, so just using from_client is easier.

Change-Id: I39dbc18f6aaa3abc342409be623395647f968530
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8754
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 21:33:59 +00:00
Florian Klink
37e7742ba9 chore(tvix/store/blobsvc): remove GRPCBlobService::new
There's very little reason to instantiate a GRPCBlobService in a context
where we are not already in a tokio context.

Change-Id: Ic6e18809a9f2a76f1c098ed330118d8dcfba5137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8753
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-12 21:33:59 +00:00
Florian Klink
b5e37869e6 refactor(tvix/store/pathinfosvc): use Arc<dyn …>
This removes the use of generics, like previously done with Blob and
Directory services.

Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 13:27:05 +00:00
Florian Klink
64a4f6185c refactor(tvix/store/blobsvc): drop Result<_,_> around open_write
We never returned Err here anyways, and we can still return an error
during the first (or subsequent) write(s).

Change-Id: I4b4cd3d35f6ea008e9ffe2f7b71bfc9187309e2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8750
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 12:43:34 +00:00
Florian Klink
b49f7cfec6 docs(tvix/store/blobservice): fix docstrings
These are outdated.

Change-Id: I1a1ae130a55847f57a48d5e244e7e029c1ecae7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8749
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 12:43:34 +00:00
Ryan Lahfa
5de04dbc12 feat(tvix/store): increase blob chunk size
From 64 bytes to 100 KBytes.
We need to provide a custom wrapper with a different Default instance.

Change-Id: Id7c6c437b8183b355a9e388f98cef1622b363f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-06-12 12:36:56 +00:00
Florian Klink
09c5ca0a0d refactor(tvix/store/fuse): use Arc<dyn …> instead of generics
Change-Id: I5685379bd6f89d17da6843d31bef4c1fc4dc0a18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8745
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-12 10:28:59 +00:00
Florian Klink
38a7caaada refactor(tvix/store/blobsvc/sled): cargo clippy
Change-Id: I00b8b567509d5e0847270f36dadb5dcb534b9b73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8739
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-12 10:28:59 +00:00
Florian Klink
aa7bdc1199 refactor(tvix/store): use Arc instead of Box
This allows us to blob services without closing them before putting them
in a box.
We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers
require Sync.

Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-12 10:24:24 +00:00
Florian Klink
7725eb53ad refactor(tvix/store): use Box<dyn DirectoryService>
Once we support configuring services at runtime, we don't know what
DirectoryService we're using at compile time.

This also means, we can't explicitly use the is_closed method from
GRPCPutter, without making it part of the DirectoryPutter itself.

Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-12 10:24:24 +00:00
Florian Klink
6f85dbfc06 feat(tvix/store/pathinfosvc): add calculate_nar method
Putting this in the PathInfoService trait makes much more sense, we can
have direct control over where/how to cache the results in the
implementation.

This now requires each PathInfoService to hold pointers to BlobService
and DirectoryService.

Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 10:15:20 +00:00
Florian Klink
8d05c0ceaa refactor(tvix/src/nar): drop NARCalculationService
There's only one way to calculate NAR files, by walking through them.

Things like caching such replies should be done closer to where we use
these, composing NARCalculationService doesn't actually give us much.

Instead, expose two functions, `nar::calculate_size_and_sha256` and
`nar::writer_nar`, the latter writing NAR to a writer, the former using
write_nar to only keeping the NAR size and digest.

Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-12 10:15:20 +00:00
Florian Klink
27ff98000b feat(tvix/store): eliminate generics in BlobStore
To construct various stores at runtime, we need to eliminate associated
types from the BlobService trait, and return Box<dyn …> instead of
specific types.

This also means we can't consume self in the close() method, so
everything we write to is put in an Option<>, and during the first close
we take from there.

Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 10:15:20 +00:00
Florian Klink
b3ca1a78eb feat(tvix/store): add mount command to entrypoint and fuse mod
`tvix-store mount PATH` will mount the tvix-store to the given path.

Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-10 15:33:07 +00:00
Florian Klink
83c1546b9a chore(tvix/store): add fuse feature
This brings in fuse (via the `fuser` crate), and adds pkg-config and
libfuse to the dev shell, so `cargo build` can link against it.

Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-10 14:48:48 +00:00
Florian Klink
d925ec34cd docs(src/tests/nar_renderer): fix comment
This testcase tests a missing blob fails the rendering, the comment has
been copied from elsewhere.

Change-Id: I48fa3fa454e12506590fa14a3591d156bafa8b5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8722
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-09 15:33:14 +00:00
Florian Klink
485fd26334 docs(tvix/store): drop nar-bridge references
The CLs did bitrot a bit, they're based on an older version of the
protocol, and it's unclear if they'll be a separate Go Binary, or just
another HTTP handler inside tvix-store itself, considering we now have
way more NAR juggling code than before.

Change-Id: I3632035cda8d75a8ff23b3132312f0f086d9e02f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8732
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-09 15:33:14 +00:00
Florian Klink
7f5372327c docs(tvix/store): update invocation in README
This has moved to a `daemon` subcommand.

Change-Id: Iae9778d8a59e6bf84555119fabfd62db3917bb62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8731
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-09 15:33:14 +00:00
Florian Klink
b78a9b7fe1 docs(tvix/store/docs): fix grammar
Change-Id: I1d7d306e5d997a7ba47a83d613edc19a4fba1ed4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8721
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-09 09:10:51 +00:00
Florian Klink
2b0b4cadbd feat(tvix/store/bin): use sled for daemon, grpc for import cmd
This now creates different store client, depending on the cli
subcommand.
The `import` command will connect to the gRPC service, and the `daemon`
command will use the sled implementation.

It might make sense to define some URI syntax to make this configurable
by the user, via the CLI.

Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 18:10:29 +00:00
Florian Klink
48b66a8982 feat(tvix/store/bin/import): process all path imports concurrently
Change-Id: I3e1428a4725fc2e552e8f37bc0550121117fcef6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8633
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 10:13:07 +00:00
Florian Klink
dbff1289b8 refactor(tvix/store): drop Clone from BS and DS trait bound
We don't need to be able to clone these services in here.

Change-Id: Ifb69450f7ebdc8364cbf9cdfb6464f8560440e4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8645
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-30 10:13:07 +00:00
Florian Klink
a4d0a80466 refactor(tvix/store/bin): instantiating TvixStoreIO once
Instead of instantiating it once in every loop iteration, put it in an
Arc, and clone that before passing it to the spawned task.

Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 12:35:59 +00:00
Florian Klink
d25962b9a4 refactor(tvix/eval): stop borrowing &mut self
This does undo cl/8571.

Change-Id: Ib14b4e7404f906e346304b6113860ae811afc94a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8631
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 11:11:59 +00:00
Florian Klink
fcfbcf9cfa refactor(tvix/store): drop mut self borrow in ingest_path
With traverse_to not requiring a &mut anymore, we can drop the &mut self
in all these function signatures.

Change-Id: I22105376b625cb281c39e92d3206df8a6ce97a88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 11:11:58 +00:00
Florian Klink
ca06f7061c refactor(tvix/store/directorysvc): don't borrow self mutable
Change-Id: I97f183e1ef3b1209a8f05e05e152f70d1f7a9596
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8628
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 10:48:52 +00:00
Florian Klink
dbb4d5e5b2 refactor(tvix/store/bin): use std::io
Change-Id: I3f3dc9732d90790d92268c04c75eccbe92e7e05b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8634
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-25 10:48:51 +00:00
Florian Klink
92b6d15da3 fix(tvix/store/bin): use spawn_blocking to call import_path
This operation is blocking, so it should be run inside a blocking tokio
task. Tokio panics if it detects a blocking operation inside a non-
blocking task, so cl/8619 would cause it to panic (as the GRPC clients
use spawn_blocking under the hood).

As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614),
we create a new instance of TvixStoreIO inside each loop iteration.

Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-25 05:06:24 +00:00
Florian Klink
ad6b6b14db feat(tvix/store/pathinfosvc): derive Clone
This allows closing a TvixStoreIO if it contains a SledPathInfoService.

Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-24 16:04:08 +00:00
Florian Klink
5774117d5e feat(tvix/store): implement TvixStoreIO
This providesEvalIO, asking given PathInfoService, DirectoryService
and BlobService.

Change-Id: I32f210f5a7aa8173ad9a7d53e8a5ac03619f527a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8561
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 09:25:02 +00:00
Florian Klink
7bd9df0133 feat(tvix/store/pathinfosvc): GRPCPathInfoService::from_client
Change-Id: I4a4f6f713dd76bf95f393c094c0eb9b6d15c5436
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8613
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:37 +00:00
Florian Klink
8d392221c6 feat(tvix/store/directorysvc): GRPCDirectoryService::from_client
Change-Id: Ifa274a380683b01f0f24cd9ff1f50d22f13b38bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8612
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:36 +00:00
Florian Klink
f4a7b325e9 feat(tvix/store/blobsvc): add constructors
Change-Id: I55e06bf4e8a11dc2caf92c597558f1b820b42566
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8610
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-24 09:01:36 +00:00
Florian Klink
2d214eacc5 fix(tvix/store/directorysvc): stop renaming spans
We have code.namespace as a field, set to
tvix_store::directoryservice::sled, so there's no need to repeat the
name here.

Change-Id: Ic1aa8a2b24de439c6a189966bd773e9acf49d1e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8623
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 09:01:35 +00:00
Florian Klink
457fd4c325 feat(tvix/store/nar): add GRPCNARCalculationService
This asks a remote tvix-store for the nar size and digest of a given
root node.

Change-Id: If9f916d9bfc5f8dc3166e2c6c1671c0f0124d1c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8611
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-24 09:01:35 +00:00
Florian Klink
79bbf25cb2 refactor(tvix/store/nar): clippy
Change-Id: I0d864fd1b9248fe4d5c88e3d8fa396ea759fea09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8616
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
4dd954f8f5 refactor(tvix/store/directorysvc): clippy
Change-Id: I30f7e98fe79ba22d218b4aabaef88c84b6085d82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8617
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
e51d85e078 refactor(tvix/store/digests): clippy
useless conversion to the same type: `std::vec::Vec<u8>`

Change-Id: Idcb16679bb2a3350784965d8d9bbb593c760634e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8615
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 12:28:34 +00:00
Florian Klink
066179651c refactor(tvix/store/blobsvc): move from Vec<u8> to B3Digest
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-23 10:48:55 +00:00
Florian Klink
b8ff08b1b0 refactor(tvix/store/directorysvc): move from Vec<u8> to B3Digest
This introduces a new struct, B3Digest, which internally holds a
Vec<u8>, but only allows construction with 32 bytes.

It also implements display, which will print the base64 representation.
This should reduce some boilerplate when parsing Vec<u8>.

Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-23 10:48:55 +00:00
Florian Klink
e31008db14 fix(tvix/store): fix timing sensitivity in gRPC directorysvc test
One test in here assumed the server was fast, but when very busy, a
computer running these tests might not be fast.

Treat both cases that can occur separately.

Change-Id: Iba168ad39f2458c4fb8873211df33beeaff7c6c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8595
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-18 19:38:54 +00:00
Florian Klink
103d2eb909 refactor(tvix/store): bump CalculateNARResponse.nar_size to u64
Change-Id: I5f41981820363a5eb9982c3d19830916207c62cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8591
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
0e7bbf0d01 fix(tvix/store/proto/grpc_blobsvc): clippy
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-18 19:29:53 +00:00
Florian Klink
a0801ae220 refactor(tvix/store/blobsvc/sled): clippy
Change-Id: Icfea8ceb6b827a645c6f10efae741f6fa2114fad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8590
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
2e09e94aad refactor(tvix/store/directorysvc/traverse): clippy, use NamedNode
Also, get rid of the explicit byte comparison here, but unwrap like in
the rest of the codebase. We'll deal with this in a generic manner in
b/267 and b/189.

Change-Id: Ie5f3d27ab35b7e88d67a2796c29cdd7bc7df71f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8589
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
a1324513ad feat(tvix/store/directorysvc): add traverse_to
This walks from a node further down until it reaches the requested path.

Change-Id: I2f9a15a8601db4d06c95d7b47cd6153264e203e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8568
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-18 06:26:55 +00:00
Florian Klink
3a4e29c261 refactor(tvix/store): rename import::{import_path -> ingest_path}
This distinguishes it better from the EvalIO::import_path method.
Also update the docstring to explain what it does (and what it doesn't).

Change-Id: I32a8b2869fa67a894df28532b22bf170961a2abf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8578
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-05-17 13:06:53 +00:00
Florian Klink
a6c7869393 fix(tvix/store/src/proto/mod.rs): fix error string
This was missed while renaming NixPath to StorePath.

Change-Id: Ibcc929c43b111e4370e8222c1dd86d403548367f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8577
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-17 13:06:52 +00:00
Florian Klink
e815b680c0 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directly
We currently only support querying by the output hash digest.
This makes the interface a bit simpler.

Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-16 10:39:06 +00:00
Florian Klink
a9a2fce27e feat(tvix/store): add GRPCBlobService
This connects to a (remote) tvix-store BlobService over gRPC.

Change-Id: If31f706738a5c3445886c117feca8b61f3203e9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8552
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-14 18:59:55 +00:00
Florian Klink
616fa4476f refactor(tvix/store): remove ChunkService
Whether chunking is involved or not, is an implementation detail of each
Blobstore. Consumers of a whole blob shouldn't need to worry about that.
It currently is not visible in the gRPC interface either. It
shouldn't bleed into everything.

Let the BlobService trait provide `open_read` and `open_write` methods,
which return handles providing io::Read or io::Write, and leave the
details up to the implementation.

This means, our custom BlobReader module can go away, and all the
chunking bits in there, too.

In the future, we might still want to add more chunking-aware syncing,
but as a syncing strategy some stores can expose, not as a fundamental
protocol component.

This currently needs "SyncReadIntoAsyncRead", taken and vendored in from
https://github.com/tokio-rs/tokio/pull/5669.
It provides a AsyncRead for a sync Read, which is necessary to connect
our (sync) BlobReader interface to a GRPC server implementation.

As an alternative, we could also make the BlobReader itself async, and
let consumers of the trait (EvalIO) deal with the async-ness, but this
is less of a change for now.

In terms of vendoring, I initially tried to move our tokio crate to
these commits, but ended up in version incompatibilities, so let's
vendor it in for now.

Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-11 14:27:57 +00:00
Florian Klink
a56b5131c8 chore(tvix): bump cargo dependencies
Change-Id: I6b872a33885f4e29082c554062a60317db754188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8550
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Johannes Kirschbauer <hsjobeki+github@gmail.com>
2023-05-10 06:15:37 +00:00
Florian Klink
a226a3c2c4 fix(tvix/store/nar/renderer): handle digest error
Change-Id: I183580732e1dd33ed079a2593097ec790def0a55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8554
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-10 06:15:37 +00:00
John Ericson
26c68f8e89 refactor(nix-compat): Properly encapsulate store path construction
Before there was code scattered about (e.g. text hashing module and
derivation output computation) constructing store paths from low level
building blocks --- there was some duplication and it was easy to make
nonsense store paths.

Now, we have roughly the same "safe-ish" ways of constructing them as
C++ Nix, and only those are exposed:

- Make text hashed content-addressed store paths

- Make other content-addressed store paths

- Make input-addressed fixed output hashes

Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-04-09 15:12:21 +00:00
Florian Klink
6edc2182d1 docs(tvix/store): update comment regarding reachability check
This might be more understandable.

Change-Id: I91371d304a9a316a2934ce41929998eaef67e11c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:35:39 +00:00
Florian Klink
60751b70eb docs(tvix/store/grpc_directorysvc_wrapper): add TODO
The logic validating connectivity of Directory nodes should be moved
to SimplePutter, and this use whatever DirectoryPutter the store comes
with.

Change-Id: Id68a86a96cc49ff73920017839788859ea9c5161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8358
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:35:39 +00:00
Florian Klink
8263c024c9 refactor(tvix/store/import): use DirectoryPutter in import.rs
This should allow import_path to communicate to a gRPC remote store,
that actually verifies the Directory nodes are interconnected.

Change-Id: Ic5d28c33518f50dedec15f1732d81579a3afaff1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8357
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-07 09:26:36 +00:00
Florian Klink
0836450006 feat(tvix/store/directorysvc): add put_multiple_start
This provides a handle to upload multiple proto::Directory as part of
the same closure.

Change-Id: I9213dde257a260c8622239918ea541064b270484
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8356
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-04-07 09:26:36 +00:00
Florian Klink
5bf9134324 refactor(tvix/store/directorysvc): move DirectoryTraverser to utils
Change-Id: Ie60a660e0fda7c80a6c7de20404c1965fe0e0d63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8355
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-31 10:54:48 +00:00
Florian Klink
5f2c2e79e1 refactor(tvix/nix-compat): move build_store_path out of derivation
This doesn't have anything to do with ATerms, we just happen to be using
the aterm representation of a Derivation as contents.

Moving this into store_path/utils.rs makes these things much cleaner -
Have a build_store_path_from_references function, and a
build_store_path_from_fingerprint helper function that makes use of it.

build_store_path_from_references is invoked from the derivation module
which can be used to calculate the derivation path.

In the derivation module, we also invoke
build_store_path_from_fingerprint during the output path calculation.

Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-30 14:03:35 +00:00
Florian Klink
d884890e56 docs(tvix/store/directorysvc/grpc): fix comment
We do compare for equality. This comment probably was when I tried to
compare the `Result<T, E>`, and as `E` doesn't derive PartialEq it was
annoying.

Change-Id: I18bb19528c76af91c9d24d88d55dd46d0c092d20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8354
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 14:37:33 +00:00
Florian Klink
2fe53cce40 feat(tvix/store/directorysvc): add DirectoryService::get_recursive()
This moves the recursive BFS traversal of Directory closures from the
GRPCDirectoryServiceWrapper out into a a DirectoryTraverser struct
implementing Iterator.

It is then used from various implementors of DirectoryService in the
`get_recursive()` method.

This allows distinguishing between recursive requests and non-recursive
requests in the gRPC client trait implementation.

Change-Id: I50bfd4a0d9eb11832847329b78c587ec7c9dc7b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8351
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-27 09:03:34 +00:00
Florian Klink
367a5e9922 feat(tvix/store/directorysvc): add gRPC client
This provides a GRPCDirectoryService struct implementing
DirectoryService, allowing a client to Directory objects from a (remote)
tvix-store.

Remote in this case is anything outside the current process, be it
another process, or an endpoint on the network.

To keep the sync interface in the `DirectoryService` trait, a handle to
some tokio runtime needs to be passed into the constructor, and the two
methods use `self.tokio_handle.spawn` to start an async function, and
`self.tokio_handle.block_on` to wait for its completion.

The client handle, called `grpc_client` itself is easy to clone, and
treats concurrent requests internally. This means, even though we keep
the `DirectoryService` trait sync, there's nothing preventing it from
being used concurrently, let's say from multiple threads.

There's still two limitations for now:

1) The trait doesn't make use of the `recursive` request, which
   currently leads to a N+1 query problem. This can be fixed
   by `GRPCDirectoryService` having a reference to another
   `DirectoryService` acting as the local side.
   I want to wait for general store composition code to pop up before
   manually coding this here.

2) It's currently only possible to put() leaf directory nodes, as the
   request normally requires uploading a whole closure. We might want
   to add another batch function to upload a whole closure, and/or do
   this batching in certain cases. This still needs some more thinking.

Change-Id: I7ffec791610b72c0960cf5307cefbb12ec946dc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8336
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 08:48:41 +00:00
Florian Klink
b919f29752 feat(tvix/store/pathinfosvc): add gRPC client
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-27 08:48:41 +00:00
Florian Klink
93f3964cbc docs(tvix/store/blobreader): fix docstring
Change-Id: Ia8ce1dd1663663903b9f60feada12e6a16fb4af2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8353
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-27 08:44:40 +00:00
Florian Klink
bde10682a5 docs(tvix/store): fix typo in comment
We query the blob service for detailled blob info, not the chunk
service.

Change-Id: I85a6a57b1dae74a950f734be7d4455c5c35ae355
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8348
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 08:44:39 +00:00
sterni
3a74268b6e chore(3p/sources): Bump channels & overlays
* //users/wpcarro/emacs: use top level (ELPA) version of eglot, as it
  was removed from MELPA:

  dc2ead17a8

* //3p/nixpkgs:tdlib: 1.8.11 -> 1.8.12

* //tvix/store: regenerate protos after buf update

Change-Id: I782a8d91fda5ed461788055a3721104e8c032207
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8327
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-21 01:31:48 +00:00
Florian Klink
49cb2a2b1f feat(tvix/store/bin): print store path
After ingestion of the contents into the store, this will use the
NonCachingNARCalculationService to create a NAR stream or the contents
of the path, and use our Derivation output path calculation machinery to
determine the output path (using recursive hashing strategy).

In a real-world scenario, we obviously want to cache these calculations,
but this should be sufficient to tinker around with it.

Change-Id: I9b2e69384414f0be1bdcb5a99a4bfd46e8db9932
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8317
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-19 20:57:27 +00:00
Florian Klink
b025ebb2a1 refactor(tvix/store/nar): pass in &proto::node::Node
Passing in a &proto::node::Node into all this allows us consumers to
keep ownership of the proto::node::Node.

Change-Id: I44882a86c46826b06a8a8a0b24c18adfc7052662
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8316
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-17 07:49:38 +00:00
Florian Klink
985f842e32 chore(tvix/store/grpcblobsvc): clippy lint
Change-Id: Idb78e0417a962599cdfdef5e7346f7fa41e3fa1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8320
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-17 07:49:38 +00:00
Florian Klink
23ddf49229 chore(tvix/store/chunkservice): clippy lint
Change-Id: I2073408fe6e87ffadbeb35ef2f9aaa0316c54966
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8319
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-17 07:49:37 +00:00
Florian Klink
7a966b2540 chore(tvix/store/blobreader): clippy lint
Change-Id: I4c85425782878295277abe6dc91a1b570768f7fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8318
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-17 07:49:37 +00:00
Florian Klink
530cb920b5 refactor(tvix/store/chunksvc): use [u8; 32] instead of Vec<u8>
Change-Id: Ie2b94aa5d69ff2c61fb77e13ae844f81f6270273
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8314
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-16 13:47:32 +00:00
Florian Klink
ee23220564 refactor(tvix/store/directorysvc): use [u8; 32] instead of Vec<u8>
Also, simplify the trait interface, only allowing lookups of Directory
objects by their digest.

Change-Id: I6eec28a8cb0557bed9b69df8b8ff99a5e0f8fe35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8313
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-16 13:47:32 +00:00
Florian Klink
36bed4ce57 chore(tvix/store): drop separate bin package
Apparently, having multiple packages with the same path is a bad thing:

```
The bin target `tvix-store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)` has the same output filename as the lib target `tvix_store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)`.
Colliding filename is: /home/flokli/tvl/tvix/target/doc/tvix_store/index.html
The output filenames should be unique.
This is a known bug where multiple crates with the same name use
the same path; see <https://github.com/rust-lang/cargo/issues/6313>.
```

Change-Id: Ic785c0349070783baf5e8fd23f5fb60603a3c995
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8308
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 22:23:10 +00:00
Florian Klink
21782d24f5 feat(tvix/store): drop BlobWriter
All code initially using this has been replaced by the simpler and more
performant implementation with StreamCDC and read_all_and_chunk.

Change-Id: I08889e9a6984de91c5debcf2b612cb68ae5072d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8265
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-13 10:05:21 +00:00
Florian Klink
c8bbddd5e5 refactor(tvix/store): use read_all_and_chunk in gRPC blobservice
This was the last piece of code using BlobWriter.

We can also use `read_all_and_chunk`, it's just requires a bit more
plumbing:

 - The data coming from the client (stream) needs to be mapped (we
   extract the .data field).
 - The stream needs to be turned into an (async) reader
 - The reader needs to be made sync, and that code using the sync reader
   needs to be in a `task::spawn_blocking`.

Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 10:05:21 +00:00
Florian Klink
7ffb2676ee refactor(tvix/store): add read_all_and_chunk method
This moves the logic from src/import.rs that

 - reads over the contents of a file
 - chunks them up and uploads individual chunks
 - keeps track of the uploaded chunks in a BlobMeta structure
 - returns the hash of the blob and the BlobMeta structure

… into a generic read_all_and_chunk function in
src/chunkservice/util.rs.

It will work on anything implementing io::Read, not just files, which
will help us in a bit.

Change-Id: I53bf628114b73ee2e515bdae29974571ea2b6f6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8259
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-03-13 08:46:51 +00:00
Florian Klink
2fe7192dbc refactor(tvix/store): use update_hasher in blobwriter
Make use of the helper function here as well.

Change-Id: Ia0afd84eb3903bb897ee6aee884dc291f3e4371c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8258
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-13 08:46:51 +00:00
Florian Klink
62ecfc7001 chore(tvix/store): allow(unused_mut)
When building tvix-store without default features, this variable doesn't
need to be mutable. Silence the warning.

Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-12 16:45:37 +00:00
Florian Klink
e4687765d7 refactor(tvix/store): bump fastcdc dep
This removes the use of Box::new, switching fastcdc to version 3.0.2
with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed.

Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-11 14:12:46 +00:00
Florian Klink
b049b88d2d refactor(tvix/store): factor out hash update into function
We're using this in a bunch of places. Let's move it into a helper
function.

Change-Id: I118fba35f6d343704520ba37280e4ca52a61da44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8251
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
2dc93f8de2 docs(tvix/store): make upload_chunk docs a docstring
Change-Id: I6794e2a69b4907ab8d9ed2d55335c11db2c30c02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8250
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
ead113cdfc feat(tvix/store/import): use StreamCDC instead of blobwriter
This seems to be way faster.

Change-Id: Ica7cee95d108c51fe67365f07366634ddbbfa060
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8246
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-11 12:54:07 +00:00
Florian Klink
ceb9d670bf refactor(tvix/store): rename C to DS, client to directory_service
This matches the semantics in other gRPC wrappers.

Change-Id: I505c901aa77a58deab216316d08542877b0b2c14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8247
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 16:15:48 +00:00
Florian Klink
3506d3bd0e refactor(tvix/store): move upload_chunk out of blobwriter
This is useful not only in blobwriter contexts.

Change-Id: I4c584b5264ff7b4bb3b1a9671affc39e18bf4ccf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8245
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 15:45:45 +00:00
Florian Klink
ac3e470b15 feat(tvix/store): clippy
warning: the following explicit lifetimes could be elided: 'set
   --> store/src/proto/mod.rs:201:1
    |
201 | / fn update_if_lt_prev<'set, 'n>(
202 | |     prev_name: &'set mut &'n str,
203 | |     name: &'n str,
204 | | ) -> Result<(), ValidateDirectoryError> {
    | |_______________________________________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes
    = note: `#[warn(clippy::needless_lifetimes)]` on by default

Change-Id: I053d370a34e5c6721ef86502c7f225c3fbfb8746
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8244
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:23:04 +00:00
Florian Klink
e1783c45a4 refactor(tvix/store/blobwriter): don't clone
This already is a &CS, so we don't need to clone().

Change-Id: I5397a5948ae7fe4781f18df760a79047f83dca01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8243
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 12:23:04 +00:00
Florian Klink
d2b353e516 refactor(tvix/store): clippy
Change-Id: Ib41c766a45e7d5467a4e5e90e1c00b009121959a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8238
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:15:02 +00:00
Florian Klink
6d3f5b7eb4 refactor(tvix/store): impl Default for MemoryBlobService
Change-Id: I78a4cd86cb364b970e99393579808e773db5ceb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8237
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:15:02 +00:00
Florian Klink
b3d6e70909 refactor(tvix/store): impl Default for MemoryChunkService
Change-Id: If1ceb870e9e5d03b651923a3a786350a8663e990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 12:13:01 +00:00
Florian Klink
e716bd0957 refactor(tvix/store): impl Default for MemoryDirectoryService
Change-Id: Ie75a1e67c8c0499ff10f60d90251b768937dad3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8235
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:13:01 +00:00
Florian Klink
8e97c0deb0 refactor(tvix/store): impl Default for MemoryPathInfoService
Change-Id: I07f83f8aaa6312cc37ce7e7ee2b713e60df23782
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8234
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:13:00 +00:00
Florian Klink
510927e43a feat(tvix/store): use rayon to upload chunks concurrently
Look at the data that's written to us, and upload all chunks but the
rest in parallel, using rayon. This required moving `upload_chunk`
outside the struct, and accepting a ChunkService to use for upload
(which it was previously getting from `self.chunk_service`).

This doesn't speed up things too much for now, because things are still
mostly linear.

Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 11:53:29 +00:00
Florian Klink
2ef60282b6 feat(tvix/store/import): make sure entries are sorted
The Directory service does already reject inserting invalid (wrongly
sorted) Directory messages, but our test case didn't provoke it.

Change-Id: I228e201925e8999186659a2d8da0118db184d9ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8167
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
a4d06b68d8 feat(tvix/store): move to daemon subcommand, add import subcommand
This exposes the previous default behavior at the `tvix-store daemon`
subcommand.

It also adds a `tvix-store import` command, which will ingest a given
path into the store.

Change-Id: Ide14f1d409b9364e7f98090690c744326486e470
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8166
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
6f6ddccc92 refactor(tvix/store): move entrypoint to src/bin/tvix-store.rs
Change-Id: Ibb83be75a2be27debd9e85b43c1b824f59e54dab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8165
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
b4a89461f5 feat(tvix/store): use Memory*Service instead of temporary sled
Change-Id: I65d2030aa643b6937dd92baa85db033bdce38b02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8208
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
a4f6c4181a feat(tvix/store): add new_temporary for all Sled services
This provides a service using /dev/shm, that's deleted once the
reference is dropped.

Refactor all tests to use these, which allows getting rid of most
TempDir usage in the tests.

The only place where we still use TempDir is in the importer tests,
which work on a filesystem path.

Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
535e1b15ab fix(tvix/store/proto/grpc_blobservice_wrapper): buffer recv data
While we don't want to keep all of the data in memory, we want to
feed a reasonably-enough buffer to the chunking function, to prevent
unnecessarily trying to chunk over and over again.

Change-Id: I5bbe2d55e8c1c63f8f7ce343889d374b528b559e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8160
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
b29d1ae372 feat(tvix/store): add import::import_path
This imports the contents at a given Path into the tvix store.

It doesn't register the contents at a Path in the store itself, that's up
to the PathInfoService.

Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
52a5181eba refactor(tvix/store/tests/nar_renderer): only upload EMPTY_BLOB
This also uploaded HELLOWORLD_BLOB_CONTENTS before, but it's not
referred from anywhere in the fixture.

Change-Id: I823133afe0f08d18a59e2ac4e4d4bb7d34ce8a2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8158
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
092f8a7df8 refactor(tvix/store/tests): move nar contents to fixtures
Change-Id: I89a700240e7250a4d93eb8c4235e79c29ecb7f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8157
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
e7279b6063 refactor(tvix/store/tests): move fixtures into separate module
Change-Id: I362dbf0899e4dc42114fd2e6a8fa7f537e9ea138
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8156
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
28a862976b refactor(tvix/store/tests): move gen_*_service() into helper
This allows hiding to tests what exact implementation we're using, when
testing things that do something with a store, but don't care what's
used for underlying storage.

Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
0baaabc43e refactor(tvix/store): move blob splitting into a BlobWriter struct
This will moves the chunking-as-we-receive logic that so far only lived
in grpc_blobservice_wrapper.rs into a generic BlobWriter.

Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
d8ab140d25 feat(tvix/store): do not buffer blob data
Use the FastCDC::cut function to ask fastcd for cutting points as we
receive the data. Make sure to keep the last chunk in the temporary
buffer, as we might not actually cut at the end.

Also, use rayon to calculate the blake3 hash if the input data is
> 128KiB.

Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
a40d2dcdcd feat(tvix/store): bump fastcdc, use v2020 version
This switches away from the less canonical "ronomon" version to the
implementation as described in the
[paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et
al., in 2020.

That version uses 64-bit hash values and tends to be faster than both
the ronomon and v2016 versions, and produces the same chunking as the
2016 version.

As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's
the recommended choice.

The crate also gained support for streaming version of chunkers:
https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be
useful.

Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
fbabcf0420 feat(tvix/store): add --json arg
This configures logging as JSON.

Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
7fe7e03df3 chore(tvix/store): remove old implementations
These were implementing against the (more complicated) gRPC interface,
for which we now have a wrapper.

Change-Id: I0a8284493718ab99618a1d21a76df4d173edb899
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8100
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
6b91efa5cf refactor(tvix/store): use GRPC Wrappers and new services
This swaps out the implementation used in the main entrypoint.
We now use the non-gRPC aware Sled*Service, and wrap it with
GRPC*ServiceWrapper to spin up the gRPC server.

Change-Id: I5a1236c0612eee85e49891062040127c8bd95058
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8106
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
cd22a8aa6d feat(tvix/store): add tests for GRPCPathInfoServiceWrapper
Change-Id: I22de33c28c132fc9710ddecd0e9158ee3b3ef94b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8108
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
dd1ee7a2e7 feat(tvix/store): add GRPCPathInfoServiceWrapper
This exposes a proto::pathinfo_service_server::PathInfoService for a
directoryservice::PathInfoService and a way to calculate NARs.

Change-Id: I30cd058562f83d063c78b84976ec97190de49400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8097
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
9b3228959a feat(tvix/store): add pathinfoservice
This adds a PathInfoService trait, and an implementation for it using
sled, and one using a HashMap.

Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
419f7d0f0c feat(tvix/store): add NARCalculationService trait and impl
This adds a NARCalculationService trait, which will take a root node,
and provide a proto::CalculateNarResponse in return.

It also adds a NonCachingNARCalculationService implementation, that will
simply always render the NAR in memory to calculate the size and sha256,
without any caching.

Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
ab02fc668c feat(tvix/store): validate blob size in NARRenderer
Make sure the blob size in the current proto node matches what we get
back from the blob backend.

Change-Id: I939fa18f37c7bc86ada8a495c7be622e69ec47f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8129
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
bc3f71838f refactor(tvix/store): don't use anyhow::Result<()> in tests
It's much easier to just unwrap the TempDir::new() result.

Change-Id: I7b05bc18f3146401e30e1cc2bb412503c5171a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8128
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
df3223fd68 chore(tvix/store): move NAR rendering logic into Renderer struct
This moves the logic rendering NARs to a struct using the
previously introduced, more granular BlobService, ChunkService and
DirectoryService.

Instead of passing them around to the helper functions, they're kept as
members of a struct.

Remove the async invocations in the nar_renderer tests, there's nothing
async in here.

Change-Id: Ic6d24aaad68a1fda46ce29f2cdb5f7b87f481d5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8095
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
cdb9458310 feat(tvix/store): add tests for GRPCDirectoryServiceWrapper
Change-Id: Iccf70664cd5dd8ea22e0b43002d2ff1a9680870a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8102
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
1540c7e276 feat(tvix/store): add GRPCDirectoryServiceWrapper
This exposes a proto::directory_service_server::DirectoryService for a
directoryservice::DirectoryService.

Change-Id: I191a6e44e5bce10d904b5f10f0073d6adaf1f572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8094
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
3af467d7ee feat(tvix/store): add directoryservice
This adds a DirectoryService trait, and an implementation for it using
sled, and one using a HashMap.

Change-Id: Ida61524b2ca949e1b3a78089a5aa7d9f9800c8d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8093
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
35ea0b0d2e feat(tvix/store): add tests for GRPCBlobServiceWrapper
Change-Id: I0c9a2d887b5b94c43afee97a8a1e701cc9f74f23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8098
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
69516f1f68 feat(tvix/store/blobservice): add GRPCBlobServiceWrapper
This takes a BlobService and ChunkService in the constructor, and
provides a [proto::blob_service_server::BlobService] trait for it.

Implementing proto::blob_service_server::BlobService is a lot of surface
to cover, and providing this wrapper will make individual
implementations taking care of how to store chunks or chunking
information much simpler.

Change-Id: Ia7b46484fb3ac9104354d496ff2922dca96ff7b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8092
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
bdf0725def refactor(tvix/store): remove anyhow::Result in directory it test
There's no `?` being used.

Change-Id: Ifbfd88f6056e0064d11458cb335d05b333ec7b6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8127
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
95aec51b2c refactor(tvix/store): remove useless clone in test
Change-Id: If62887cbc3338110cb7c815ec2ee2160fae04353
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8111
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
80f68bf828 chore(tvix/store): move protos into separate mod.rs
This allows adding more stuff into this namespace, from different files.

Also move tests on proto-related code from src/tests to src/proto/tests.

Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
60abca1d8e feat(tvix/store): add blobreader
A BlobReader can be used to read a blob, which might consist out of
multiple chunks. Chunks are fetched from a ChunkService.

Change-Id: I1806225b0052adaa4a2320b79b744e554e524ee3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8088
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 10:58:55 +00:00
Florian Klink
b74ffda583 feat(tvix/store): add blobservice
This adds a BlobService trait, and an implementation for it using sled,
and one using a HashMap.

Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 15:32:47 +00:00
Florian Klink
119aa43171 feat(tvix/store): add chunkservice
This adds the simpler ChunkService trait, and an implementation for it
using sled, and one using a HashMap.

Change-Id: Icb0fdc41b37b44e9e9e4f548d0f4acae1d83b71e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8086
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-06 15:32:47 +00:00
Florian Klink
3f27fe3484 feat(tvix/store): impl From<PoisonError> for Error
Change-Id: Ib61e276b45c0102e383a7e7e641172b151369b03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8207
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 15:32:47 +00:00
Florian Klink
786854713c feat(tvix/store): add errors
Change-Id: I7be0a4eaa29840e1a3329bccb791975de22828ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8085
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-06 15:32:47 +00:00
Florian Klink
6019c75deb docs(tvix/store): add document describing why we don't use git trees
This came up recently again, it makes sense to document the reasoning
behind the decision.

Change-Id: Ic51d5bc7998c70e8b070b6f42877d8e88613935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8223
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-05 18:33:54 +00:00
Florian Klink
db8cebe0b4 docs(tvix/store): move link hrefs to bottom
Change-Id: Id8e32ee4de98fbe48266d417279194cd93e968cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8222
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-05 14:42:05 +00:00
Florian Klink
f2624f1028 refactor(tvix/store): remove needless borrow in build.rs
Change-Id: I2c2a70e5ffbb080702f57218aa9dcfe53d08870f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8110
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-15 18:40:36 +00:00
Florian Klink
cb8466d417 refactor(tvix/store/proto): use .cloned()
Instead of using an explicit closure to clone elements, use .cloned().

Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-15 18:40:34 +00:00
Florian Klink
cfa0755fc4 refactor(tvix/store): avoid casting to the same type
The size field already is u32, we don't need to convert here.

Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-15 18:40:33 +00:00
Florian Klink
87e1934f2d docs(store/protos): update comment on blobstore Read and Put
Further emphasize Read() can be used to ask for blobs OR chunks, and
that clients usually want to stat and then request (smaller) chunks,
rather than reading whole blobs.

Also clarify that the chunking used to send BlobChunks over has nothing
to do with the chunk sizes communicated in a Stat() request.

Change-Id: Ia615d190aae570611de2655b11342a14d0b75976
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-07 14:14:35 +00:00
Florian Klink
0db73cb2bd feat(tvix/store): add write_nar function
This adds a function that consumes a [proto::node::Node] pointing to the
root of a (store) path, and writes the contents in NAR serialization to
the passed [std::io::Write].

We need this in various places:

 - tvix-store's calculate_nar() RPC method needs to render a NAR stream
   to get the nar hash, which is necessary to give things imported in
   the store a "NAR-based" store path.

 - communication with (remote) Nix (via daemon protocol) needs a NAR
   representation.

 - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a
   NAR representation.

Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:28:22 +00:00
Florian Klink
9e809e21cc feat(tvix/store): implement iteration over Directory nodes
This provides a `Directory.nodes()` function, returning an iterator over
all three node types in an ordered fashion.

Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
1ee6bd06e3 feat(tvix/store/proto): implement get_name for node::Node, pub trait
Make the trait public, so consumers can use it.

Also, implement it for node::Node, so we can later use this to access
the name from all three if we don't care about the enum type.

Change-Id: Iae530a16b705493645e61947852c03273876cc55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
c27bacd905 refactor(tvix): introduce nix-compat crate
Move nixbase32 and store_path into this.

This allows //tvix/cli to not pull in //tvix/store for now.

Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 13:26:18 +00:00
Florian Klink
8ec8d03175 fix(tvix/store/nixbase32): fix encoder/decoder
Replace our data_encoding usage with the implementation taken from
https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32

Also uncomment some of the unit tests, and add a regression test for a
NIXBASE32.encode with a 32 bytes sequence.

The previous implementation of NIXBASE32.encode is wrong in that case:

```
❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA=
0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk
❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C
00000000  b3 a2 4d e9 7a 8f db c8  35 b9 83 31 69 50 10 30  |..M.z...5..1iP.0|
00000010  b8 97 70 31 bc b5 4b 3b  3a c1 37 40 f8 46 ab 30  |..p1..K;:.7@.F.0|
00000020
```

Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 11:46:30 +00:00
Florian Klink
85e563c554 refactor(tvix/store/tests): move Directory tests into directory.rs
There's gonna be more tests coming, let's move them into a separate file
everything Directory-related into a separate module.

Change-Id: I78a0f263925528907a22724b028f75cce644b329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-30 09:45:11 +00:00
Florian Klink
a2c33f517c feat(tvix/store/proto): implement get_name for all nodes
Also add a `NamedNode` trait. We'll later use this to access names from
all three individually.

Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 09:45:11 +00:00
Florian Klink
9c18888715 docs(tvix/store/protos): add docstring for Directory::digest()
Change-Id: I361dbca444a267fea28cd212d563ee2d03497c16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7952
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 09:45:11 +00:00
Florian Klink
42d3c06988 docs(tvix/store/protos): fix docstring
Change-Id: Idf835e9648ab6c95c29960f3f8176153d4888b27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7951
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-30 09:45:11 +00:00
Florian Klink
1a84e3e6d8 chore(tvix/store): add fastcdc crate
This is used for content-defined chunking.

Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 15:12:45 +00:00
Florian Klink
90979d39f3 feat(tvix/store/directory): validate Directory and sizes
This calls out to Directory::validate() for all received Directory
messages, and also makes sure the sizes we refer a Directory message as
matches the sizes that have been calculated.

Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 10:48:17 +00:00
Florian Klink
d8e0fa8e5e feat(tvix/store/directory): deduplicate Directory messages
We can omit sending Directory messages to clients that have already been
sent in the same stream.

We can also omit storing a Directory message if we already have it -
they're content-addressed anyways.

Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 10:48:17 +00:00
Florian Klink
e719da53be feat(tvix/store): add SledDirectoryService
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 10:48:17 +00:00
Florian Klink
ab8486e5b8 chore(tvix/store): add tonic-mock
Upstream seems to be dead, so we're using https://github.com/tyrchen/
tonic-mock/pull/3 here.

According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment-
1241164173, we might not need this crate at all, but for now, it gets
the job done and is less code to write in the tests.

Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 09:34:15 +00:00
Florian Klink
a8b13a0b57 refactor(tvix/store): simplify test a bit
Import more things, and use expect_err to unpack the response.

Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 09:34:15 +00:00