Commit graph

270 commits

Author SHA1 Message Date
Connor Brewster
993c505cdb feat(tvix/store/fs): Add support for virtiofs backend
This adds a virtiofs daemon implementation which hooks into the existing
tvix-store filesystem implementation that is used for FUSE.

This allows adding the filesystem to a microvm without having to set up
FUSE inside the guest.

Change-Id: If80c36c9657f2289853e8d9a364bf4f1f7b7559c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9344
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-24 17:05:21 +00:00
Florian Klink
e13c86d916 chore(tvix): re-enable {ca,}store-protos-go
This couldn't be enabled before until go get was able to fetch the
modules again.

Change-Id: I82db7d22e497661ea2a0c8a45c0c865fcc0a1f18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9406
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 17:36:05 +00:00
Florian Klink
32f41458c0 refactor(tvix): move castore into tvix-castore crate
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.

Notable changes:
 - `fixtures` and `utils` had to be moved out of the `test` cfg, so they
   can be imported from tvix-store.
 - Some ad-hoc fixtures in the test were moved to proper fixtures in the
   same step.
 - The protos are now created by a (more static) recipe in the protos/
   directory.

The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)

Bug: https://b.tvl.fyi/issues/301

Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:51:21 +00:00
Florian Klink
d8ef0cfb4a docs(tvix/castore): rename traverse_to -> descend_to
With the move of this code out into castore it has become apparent this
is a general descent inside the castore. Concerns like making sure the
whole Directory closure has been fetched/is fetched initially is nothing
this layer needs to worry about. We can handle this during substitution
of a new PathInfo, once there's store composition.

Closes b/270.

Change-Id: I661ed08e54bc81478e032cfb9abeb23e5b337fbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9373
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:25 +00:00
Florian Klink
7c9a52e3ee chore(tvix): remove some unused crate dependencies
Change-Id: I31b0e7be3a2ebc268c6f4045ba282e8fb1c72a64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9375
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:24 +00:00
Florian Klink
3d064f714c docs(tvix/store/pathinfosvc): fix copypasta error
This is not a simple get and put returning digest.

You currently can get by output hash, and a put gives you back the
(possibly modified) PathInfo message.

Change-Id: I43174d87a764417b8620c4d5bdac058cc25b0373
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9371
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-22 09:14:33 +00:00
Connor Brewster
37a348b4fa refactor(tvix/store): Asyncify PathInfoService and DirectoryService
We've decided to asyncify all of the services to reduce some of the
pains going back and for between sync<->async. The end goal will be for
all the tvix-store internals to be async and then expose a sync
interface for things like tvix eval io.

Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-21 17:58:23 +00:00
Connor Brewster
7e737fde34 refactor(tvix/store/fs): Separate FUSE and filesystem code
In prepration for adding virtiofs support, I thought it would make sense
to split out the filesystem implementation from FUSE itself.

The `fs` module holds the tvix-store filesystem implemetation and the
`fuse` module holds the code to spawn a FUSE daemon backed by multiple
threads.

Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-20 14:27:53 +00:00
Connor Brewster
6b7c936bc5 refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rs
This switches the FUSE implementation from fuser to fuse-backend-rs.
fuse-backend-rs is designed to work with both FUSE and virtiofs.
Virtiofs support will make it possible to plug the tvix-store into a
microvm and have `/nix/store` access without having to setup FUSE inside
the guest.

Additionally fuse-backend-rs has nice support for running multiple FUSE
threads and has some async support.

The goal of this commit is to mechanically switch over to
fuse-backend-rs with minimal changes. I did have to add some locks here
and there because fuse-backend-rs uses `&self` on all methods whereas
fuser uses `&mut self`. `&self` is required for concurrent access to the
FUSE server, so this makes sense.

We can consider switching to concurrent maps and use some other
techniques to reduce lock contention and critical section size.

Issue: https://b.tvl.fyi/issues/305

Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-20 14:21:32 +00:00
Florian Klink
ca25acf11b chore(tvix/store/protos): drop unused fields for now
This wasn't removed yet, and no code is using/populating it so far.

It's confusing, let's update it to the state of things now, and re-
introduce it once we get there.

Change-Id: I68f5ba17a8eee604d8ccd82749da7c8be094cb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9351
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-18 10:33:30 +00:00
Florian Klink
bf2fe88a5c fix(tvix/store/fuse): fix executable bit
We were blindly returning 0o444 for all regular files, but regular files
with executable bit need to be 0o555.

This wasn't spotted because stat'ing executable files was not part of
the test suite, it's now added.

Change-Id: I04c69784053e7e43d838c01bb288f2df48f40b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9345
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 10:33:30 +00:00
Florian Klink
da6cbb4a45 refactor(tvix/store/blobsvc): make BlobStore async
We previously kept the trait of a BlobService sync.

This however had some annoying consequences:

 - It became more and more complicated to track when we're in a context
   with an async runtime in the context or not, producing bugs like
   https://b.tvl.fyi/issues/304
 - The sync trait shielded away async clients from async worloads,
   requiring manual block_on code inside the gRPC client code, and
   spawn_blocking calls in consumers of the trait, even if they were
   async (like the gRPC server)
 - We had to write our own custom glue code (SyncReadIntoAsyncRead)
   to convert a sync io::Read into a tokio::io::AsyncRead, which already
   existed in tokio internally, but upstream ia hesitant to expose.

This now makes the BlobService trait async (via the async_trait macro,
like we already do in various gRPC parts), and replaces the sync readers
and writers with their async counterparts.

Tests interacting with a BlobService now need to have an async runtime
available, the easiest way for this is to mark the test functions
with the tokio::test macro, allowing us to directly .await in the test
function.

In places where we don't have an async runtime available from context
(like tvix-cli), we can pass one down explicitly.

Now that we don't provide a sync interface anymore, the (sync) FUSE
library now holds a pointer to a tokio runtime handle, and needs to at
least have 2 threads available when talking to a blob service (which is
why some of the tests now use the multi_thread flavor).

The FUSE tests got a bit more verbose, as we couldn't use the
setup_and_mount function accepting a callback anymore. We can hopefully
move some of the test fixture setup to rstest in the future to make this
less repetitive.

Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-18 10:33:30 +00:00
Florian Klink
51b8571c48 feat(tvix/store): improve error message formatting
WrongSorting(Vec<u8>) actually encodes the name of a node, so if we can,
we want this to display it as a string (and fall back to the base64
encoding if we can't).

Before:

> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: [108, 111, 99, 97, 108, 101] is not sorted

After:

> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: locale is not sorted

Change-Id: I68420c53a89cb1aa96e4bdce414366cebcb7915f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9350
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-17 13:14:03 +00:00
Florian Klink
54bb63dea3 chore(tvix/store): allow non_snake_case
Some of the tonic-generated types create new warnings.

Change-Id: Ib538bfd7d90d857d80a129fbbdf3a38c89cc04ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9349
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-17 13:14:03 +00:00
Florian Klink
3816eaa5e9 feat(tvix/store): add simple integration test
This imports the docs folder into the tvix store, and ensures it comes
up with the same store path as Nix.

While we validate hashes in general through tvix-cli output path checks
already, it doesn't include the tvix-store CLI itself.

See https://cl.tvl.fyi/c/depot/+/9329/comment/339f0720_524f0104/
for context.

Change-Id: I239ce5b6a07cb962b242142ab716693359b8674c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9338
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 21:00:54 +00:00
Florian Klink
32f2f8b618 feat(tvix/store/import): print store path to stdout
This allows comparing the output in an integration test.

Change-Id: I8bb2254e18e90005a4f1b30fd47ef69642e3732e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9337
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 21:00:54 +00:00
Florian Klink
b52bb6e791 fix(tvix/store): log to stderr, not stdout
This leaves some space in stdout to provide actual meaningful output.

Also, rename print_node to log_node because that's what it does, it's
using the logger to log out more detailed info.

Change-Id: Ic64a6330dbfcdc63eb4198067a5c5e47b841b9a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9336
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-09-15 19:38:23 +00:00
Florian Klink
c77c7ae9a5 fix(tvix/store/blobservice/seeker): fix debug assert
We were asserting absolute_offset > self.pos, but that's not true for
both being zero.

Ramp up the tracing bits a bit, so we actually can see this in the debug
logs.

Change-Id: I21693bcafab227549b19cd6f1215d2f8dee77ecc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9292
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-12 12:15:08 +00:00
Brian McGee
1549f020ee fix(tvix/protos): support []byte node names
The underlying protos were changed to return `[]byte` instead of `string`.

Change-Id: I5d3e5d8de0ed7200325f7ab0d62e3c10d8eb1b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9285
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-09 12:13:02 +00:00
Florian Klink
ec22a80f26 docs(tvix/store/fuse): add more comment to list_root arg
Forgot to squash this into cl/9217.

Change-Id: Id5a87cbe729592f3b53e8c329b6890519f5da319
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9272
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-08 16:13:21 +00:00
Florian Klink
7923cc19f6 refactor(tvix/store): use tokio::task::JoinHandle
This makes the inside code a bit less verbose.

I wasn't able to describe the type of the async move closure itself,
which would allow us to remove the JoinHandle<_> type annotation
entirely.

Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-05 21:13:50 +00:00
Florian Klink
f499d2e031 feat(tvix/store): fix ctrl-c handling on mount command
This enables the tokio `signal` feature, and registers a ctrl_c signal
handler, which will use the unmount handle to unmount in case a ctrl-c
signal is received.

This avoids having disconnected mountpoints when Ctrl-C'ing a
`tvix-store mount` invocation.

In case the filesystem is unmounted externally (via `umount /path/to/
mountpoint`), the future is waiting for the signal is never resolved and
the task is stopped.

Change-Id: I149f705a6cb50188177f2a6c6a5fcd77218e2a3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9218
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-05 21:13:49 +00:00
Florian Klink
f9b5fc49b1 feat(tvix/store/fuse): allow listing
This provides an additional configuration flag to the tvix-store mount
subcommand, and logic in the fuse module to request listing for the
root of the mountpoint.

Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: adisbladis <adisbladis@gmail.com>
Tested-by: BuildkiteCI
2023-09-05 21:13:49 +00:00
Florian Klink
da9d706e0a feat(tvix/store/pathinfosvc): provide listing
This provides an additional method in the PathInfoService trait, as
well as an RPC method on the gRPC layer to list all PathInfo objects in
a PathInfoService.

Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-05 20:46:42 +00:00
Florian Klink
e41b5ae3f0 refactor(tvix/store): infer more types
We don't need to explicitly describe the type of the task itself,
describing the return type of the async closure is sufficient.

Also, use io::Result<_> instead of Result<_, io::Error>.

Change-Id: I9ab3f990eb49929b0aea335b2bb07da392ab631f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9267
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-09-05 20:41:41 +00:00
Florian Klink
7bd3c42c74 chore(tvix/store): drop walkdir workaround for symlinks at root
https://github.com/BurntSushi/walkdir/pull/170 got merged, meaning we
don't need to keep our own logic in here anymore.

Our test cases already cover this.

Change-Id: Ied3043ee651c8aafa10271c1e1ca5d460fb6c0b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9269
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-05 15:19:43 +00:00
Florian Klink
3c340b28bd refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate
This trait is eval-specific, there's no point in dealing with these
things in tvix-store.

This implements the EvalIO interface for a Tvix store.
The proper place for this glue code (for now) is tvix-cli, which knows
about both tvix-store and tvix-eval.

There's one annoyance with this move: The `tvix-store import` subcommand
previously also used the TvixStoreIO implementation (because it
conveniently did what we wanted).
Some of this code had to be duplicated, mostly logic to calculate the
NAR-based output path and create the PathInfo object.

Some, but potentially more of this can be extracted into helper
functions in a shared crate, and then be used from both TvixStoreIO in
tvix-cli as well as the tvix-store CLI entrypoint.

Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-09-04 16:02:12 +00:00
Florian Klink
583cdec3d8 docs(tvix/store): address cargo doc warnings
Fix some broken link references.

Change-Id: I69c9b2b62af35bb777e4df1a01ba3181a368be47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9214
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-03 10:01:40 +00:00
Florian Klink
9bbda4421e docs(tvix/store): update rpc_pathinfo.proto comments
Don't repeat the name of the method in the description, don't repeat
things already described in request message comments.

Change-Id: I180e4792577419050947eea8fea7043861aba463
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9213
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-03 10:01:40 +00:00
Florian Klink
116a137be0 refactor(tvix/store): implement rename for node::Node
This returns a node with a new name.

Change-Id: Iebcab537f8dd63d826b9841d4d0181fcb941afdd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-02 20:08:52 +00:00
Florian Klink
1e9d262ad5 docs(tvix/store): update comment
Since the refactor to use URIs for all three services, this actually
does talk to a daemon by default.

Change-Id: Ied296772b77eef514bfcae0a9dfc50f848a1c2f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9210
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-02 20:03:34 +00:00
Florian Klink
0193f07642 refactor(tvix/nix-compat/nixhash): validate digest lengths
There was a NixHash::new() before, which didn't perform any validation
of the digest length. We had some length validation when parsing nix
hashes or SRI hashes, but some places didn't perform validation and/or
constructed the struct directly.

Replace NixHash::new() with a
`impl TryFrom<(HashAlgo, Vec<u8>)> for NixHash`,  which does do this
validation, and update constructing code to use that, rather than
populating structs directly. In some rare cases where we're sure the
digest length is correct we still populate the struct manually.

Fixes b/291.

Change-Id: I7a323c5b18d94de0ec15e391b3e7586df42f4229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9109
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-08-20 22:19:22 +00:00
Florian Klink
3ffbcc6c8a refactor(tvix/store): cargo clippy
Change-Id: I3a80560d036e7ed08036b5e9f0974080d1a30ded
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9096
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-08-20 21:51:04 +00:00
Vincent Ambo
fcda068235 feat(nix/sparseTree): add optional name argument
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.

I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.

Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-08 20:11:56 +00:00
Florian Klink
432222f098 feat(tvix/store/proto): use Bytes instead of Vec<u8>
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our
bytes field cheaper to clone.

It's a bit annoying to configure due to
https://github.com/hyperium/tonic/issues/908, but the workaround does
get the job done.

Change-Id: I25714600b041bb5432d3adf5859b151e72b12778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-22 09:03:02 +00:00
Florian Klink
a6580748aa feat(tvix/store/digests): use bytes::Bytes instead of Vec<u8>
This will save us some copies, because a clone will simply create an
additional pointer to the same data.

Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-21 19:01:50 +00:00
Florian Klink
72e82ffcb1 refactor(tvix/store): use bytes for node names and symlink targets
Some paths might use names that are not valid UTF-8. We should be able
to represent them.

We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.

While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?

Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-21 19:01:49 +00:00
Florian Klink
638f3e874d feat(tvix/store/fuse): implement open explicitly
This "reverts" commit 9f600de226 (the
initial revert of f5e291cf83).

Now with BlobService returning a BlobReader that implements io::Seek, we
can actually just call blob_reader.seek(io::SeekFrom::Start(offset as
u64)).

This means, we currently will fail to seek backwards inside a file.

Change-Id: I9c19448df6831a3537252f99210374f2126ecfc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8886
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 18:52:47 +00:00
Florian Klink
7613e2e769 feat(tvix/store/blobservice): implement seek
For memory and sled, it's trivial, as we already have a Cursor<Vec<u8>>.
For gRPC, we simply reject going backwards, and skip n bytes for now.

Once the gRPC protocol gets support for offsets and verified streaming,
this can be improved.

Change-Id: I734066a514aed287ea3db64bfb1680911ac1eeb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8885
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-21 18:52:46 +00:00
Florian Klink
42dc18353d feat(tvix/nix-compat): have StorePath accept bytes
The primary constructor for this is now from_bytes, from_string is
simply calling .as_bytes() on the string, passing it along.

The InvalidName error now contains a Vec<u8>, to encode the invalid name
(which might not be a string anymore).

from_absolute_path now accepts a &[u8] (even though we might want to
make this a OSString of some sort).

StorePath::validate_name has been degraded to a pub(crate) function.
It's still used in src/derivation, even though it probably shouldn't at
all - that cleanup is left for cl/8412 though.

Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 18:04:38 +00:00
Florian Klink
5364fcb127 feat(tvix/nix-compat): fold NameError into Error
This being a nested error makes things more complicated than necessary.

Also, this caused BuildStorePathError to only hold NameError,
so refactor these utility functions to either return Error, or
BuildStorePathError.

Change-Id: I046fb403780cc5135df8b8833a291fc2a90fd913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8972
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-21 18:04:38 +00:00
sterni
88b0545908 chore(3p/sources): Bump channels & overlays
* //tvix/store: update code generated from proto files

Change-Id: Ie6aa53e71f07a0a0e637673ba4005ed983db315d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8929
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-08 10:46:31 +00:00
DavHau
e751372f2f docs(tvix/store): various improvements
Improve change some little things I noticed while reading through it.

Change-Id: I033209eece395e5aad4e10825e8dd6c0cfe68191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8725
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-07 09:48:32 +00:00
Florian Klink
c767460440 fix(tvix/store/fuse): fix log level for a debug message
Change-Id: Ib801e46636901553d71455a739aed34e5828ca0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8888
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-02 11:40:29 +00:00
Florian Klink
2a601d26a8 docs(tvix/store/fuse/inodes): clippy lint
Change-Id: I7b1bcb9c0cca76f05271f25912d26b14152fe0c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8887
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-02 11:18:52 +00:00
Florian Klink
6b599b3763 feat(tvix/store/fuse): add test reading large file
Change-Id: Ic705d05909c59c764d68a730169e9cc2b2538d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8847
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 12:35:56 +00:00
Florian Klink
7fbf874f1c feat(tvix/store/blobsvc): add more blobservice tests
Change-Id: I3e27dfb4ce3e52974d7614814abb7b5ae4a37f8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8782
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 12:35:56 +00:00
Florian Klink
3edc580e80 docs(tvix/store/blobservice): update docstring
Change-Id: I6ae693a66530e1ecca57723a97d56b309fa0651a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8881
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 12:30:54 +00:00
Florian Klink
9f600de226 fix(tvix/store/fuse): revert "implement open explicitly"
This reverts commit f5e291cf83.

The offsets are relative to the start of the file, and as long as we
don't have BlobReaders implement seek, this will be very annoying to
deal with.

Change-Id: I05968f7c5c0ec0000597da90f451d6bb650c3e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8882
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-30 12:30:53 +00:00
Florian Klink
6604ce4e51 fix(tvix/store/blobservice): write into hasher from b, not buf
buf contains everything written so far, whereas b is the slice passed in
the current write() call. If we copy from &buf, we end up with the wrong
hash, because we keep writing the wrong data to the hash function.

Change-Id: I768d4645934a6a7d75b9c8eeba35f8f3be5edd26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8880
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 12:30:53 +00:00