Commit graph

309 commits

Author SHA1 Message Date
Florian Klink
2fa5e71d5e fix(tvix/store): restore v1alpha reflection endpoint
tonic-reflection 0.12.x moved from the v1alpha to v1 of the reflection
protocol.

However, most clients, like Postman, Kreya and evans don't support that
one yet.

Bump tonic-reflection to 0.12.2, which re-introduces v1alpha support
alongside the v1 version of it, registering both services.

This fixes the example documented in tvix/store/README.md, it was
previously failing as evans couldn't find the v1alpha reflection
service.

See https://github.com/hyperium/tonic/pull/1888 for details.

Change-Id: I55438877317f82dc39face13afeb9594cda07a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12353
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-27 10:34:43 +00:00
Ben Webb
565c0fd24c fix(tvix/castore): u32 -> u64 in DirectoryError::SizeOverflow message
Fix a discrepancy in the error message for DirectoryError::SizeOverflow.

The message indicates that the SizeOverflow error occurs when total size
exceeds u32::MAX, but that's not true. All size fields within the
castore's internal Directory ADT are u64, and the SizeOverflow error is
only returned after a call to the checked_add implementation on u64.

See tvix/castore/nodes/directory.rs +111
and tvix/castore/nodes/directory.rs +88
as of this commit.

Change-Id: I74d161ea8927362e1cb601ba163489aa96fb91b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12259
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-21 22:05:57 +00:00
Ilan Joselevich
73e16c1855 feat(tvix): drop usage of sparseTree in favor of lib.sourceByRegex
We can avoid depending on things outside //tvix by just using a similar
util from nixpkgs.

Change-Id: I9ea3e1f0a8a059ea10caaec173569ba9f316aec6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12247
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 16:56:19 +00:00
Florian Klink
e086c76ee9 refactor(tvix/castore): have SymlinkTarget-specific errors
Don't use ValidateNodeError, but SymlinkTargetError.

Also, add checks for too long symlink targets.

Change-Id: I4b533325d494232ff9d0b3f4f695f5a1a0a36199
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12230
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-18 17:22:21 +00:00
Florian Klink
56fa533e43 refactor(tvix/castore): have PathComponent-specific errors
Don't use DirectoryError, but PathComponentError.

Also add checks for too long path components.

Change-Id: Ia9deb9dd0351138baadb2e9c9454c3e019d5a45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12229
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-08-18 17:22:21 +00:00
Florian Klink
0cfe2aaf6a feat(tvix/castore/proto): add owned conv to castore::Directory
Replace the hand-rolled code comparing names with a try_fold. Also, make
it slightly stricter here, detecting duplicates in the same fields
earlier.

Change-Id: I9c560838ece88c3b8b339249a8ecbf3b05969538
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12226
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-08-18 17:22:21 +00:00
Florian Klink
96832c0411 feat(tvix/castore): add Directory::try_from_iter()
This provides a batched variant to construct a Directory, which reuses
the previously calculated size.

Checking and inserting code is factored out into a check_insert_node
function, taking the current size as a parameter and returning the new
size.

Change-Id: Ia6c2970a0c12181b7c40e63cf7ce8c93298ea37c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12225
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-08-18 16:49:58 +00:00
Florian Klink
5d01892274 refactor(tvix/castore): add name back to edge weights
Make this a proper struct with named fields. We apparently never access
B3Digest in there, so it can be removed.

Change-Id: Ifc07310393e1afb0a835778eae137a19b54070b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12224
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-17 15:59:10 +00:00
Florian Klink
21ceef4934 refactor(tvix/castore): add into_nodes(), implement consuming proto conv
Provide a into_nodes() function on a Directory, which consumes self and
returns owned PathComponent and Node.

Use it to provide a proper conversion from Directory to the proto
variant that doesn't clone.

There's no need for the one taking only &Directory, we don't use it
anywhere, and once someone needs that they might as well clone Directory
before converting it.

Update all other users of the `.nodes()` function to use `.into_nodes()`
where applicable, and avoid some more cloning there.

Change-Id: Id4577b9eb173c012e225337458898d3937112bcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12218
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 15:59:10 +00:00
Florian Klink
5ec93b57e6 refactor(tvix/castore): add PathComponent type for checked components
This encodes a verified component on the type level. Internally, it
contains a bytes::Bytes.

The castore Path/PathBuf component() and file_name() methods now
return this type, the old ones returning bytes were renamed to
component_bytes() and component_file_name() respectively.

We can drop the directory_reject_invalid_name test - it's not possible
anymore to pass an invalid name to Directories::add.
Invalid names in the Directory proto are still being tested to be
rejected in the validate_invalid_names tests.

Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-17 15:59:10 +00:00
Florian Klink
8ea7d2b60e refactor(tvix/castore): drop {Directory,File,Symlink}Node
Add a `SymlinkTarget` type to represent validated symlink targets.
With this, no invalid states are representable, so we can make `Node` be
just an enum of all three kind of types, and allow access to these
fields directly.

Change-Id: I20bdd480c8d5e64a827649f303c97023b7e390f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12216
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-17 09:46:01 +00:00
Florian Klink
49b173786c refactor(tvix/castore): remove name from Nodes
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.

This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.

It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).

The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.

The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.

The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.

Some further cleanups:

The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)

There's some leftover cleanups to do:
 - There should be a from_(sorted_)iter and into_iter in Directory, so
   we can construct and deconstruct in one go.
   That should also enable us to implement conversions from and to the
   proto representation that moves, rather than clones.

 - The BuildRequest and PathInfo structs are still proto-based, so we
   still do a bunch of conversions back and forth there (and have some
   ugly expect there). There's not much point for error handling here,
   this will be moved to stricter types in a followup CL.

Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 09:45:58 +00:00
Florian Klink
c7845f3c88 refactor(tvix/castore): move *Node and Directory to crate root
*Node and Directory are types of the tvix-castore model, not the tvix
DirectoryService model. A DirectoryService only happens to send
Directories.

Move types into individual files in a nodes/ subdirectory, as it's
gotten too cluttered in a single file, and (re-)export all types from
the crate root.

This has the effect that we now cannot poke at private fields directly
from other files inside `crate::directoryservice` (as it's not all in
the same file anymore), but that's a good thing, it now forces us to go
through the proper accessors.

For the same reasons, we currently also need to introduce the `rename`
functions on each *Node directly.

A followup is gonna move the names out of the individual enum kinds, so
we can better represent "unnamed nodes".

Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-13 18:39:49 +00:00
Yureka
3ca0b53840 refactor(tvix/castore): use Directory struct separate from proto one
This uses our own data type to deal with Directories in the castore model.

It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:

 - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
 - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.

Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-13 12:17:01 +00:00
Ilan Joselevich
41dc9ee6a2 feat(tvix/castore): add RedbDirectoryService
This provides a DirectoryService implementation which uses
redb (https://github.com/cberner/redb) as the database. It provides both
in-memory and persistent on-filesystem implementations.

Change-Id: Id8f7c812e2cf401cccd1c382b19907b17a6887bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12038
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-08-01 13:20:01 +00:00
Aspen Smith
756539a596 chore(3p/sources): Bump channels & overlays (2024-07-28)
* Treewide: re-run depotfmt

* //third_party/nixpkgs:html5validator: build with Python 3.11,
  dependency openstackdocstheme doesn't support 3.12

* //users/sterni/machines/ingeborg: adapt to poorly handled fcgiwrap
  module API change: https://github.com/NixOS/nixpkgs/pull/318599

* //tvix/*-go: regenerate protobuf files

* //third_party/nixpkgs:treefmt: Remove patch for merged pull request

* //users/flokli/ipu6-softisp: rebase, drop upstreamed kernel patches

Change-Id: Ie4e0df007c287e8cd6207683a9a25838aa5bd39a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11971
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-01 10:06:33 +00:00
Yureka
5114504b67 test(tvix/castore/dirsvc): check for a pitfall with deduplicated dirs
Change-Id: I3cff2c2e8b2c8a2fd8d8074647d0d99a1db8e693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12034
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-25 09:52:29 +00:00
Yureka
6a988a1598 fix(tvix/castore/GRPCDirectorySvc): fix a bug in the get_recursive fn
When retrieving a closure with get_recursive, the following could happen in the GRPC client:
- The first reference to the deduplicated directory is added to expected_directory_digests
- The deduplicated directory is obtained removed from expected_directory_digests
- The second reference to the deduplicated directory is added to expected_directory_digests
- The deduplicated directory has already been sent, but is still in the
  expected_directory_digests. It looks to the GRPC client like the
  closure is incomplete and the stream ended prematurely.

Change-Id: Ic62bca12e7f8fb85af5fa4dacd199f0f3b8eea8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12033
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-25 09:52:29 +00:00
Yureka
e97202e54a refactor(directoryservice): get rid of NaiveSeeker helper
Use the ChunkedReader in CombinedBlobService instead which also supports seeking.

Change-Id: I681331a80763172c27e55362b7044fe81aaa323b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12031
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-23 20:53:08 +00:00
Yureka
94a0e21e68 fix(tvix/directoryservice): clarify get_recursive not found
Change-Id: I47f21b4b7db9304eadb04094d41bf04d443fcc3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12025
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-23 14:04:23 +00:00
Ilan Joselevich
9f10a71ec5 feat(tvix/store): add redb PathInfoService
This provides a PathInfoService implementation using redb
(https://github.com/cberner/redb) as the underlying storage engine.

Both an in-memory variant, as well as a filesystem one is provided,
similar how it's done with the sled implementation.

Supersedes: https://cl.tvl.fyi/c/depot/+/11692
Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc
Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-22 18:49:04 +00:00
Yureka
8342f70c60 feat(tvix/composition): improve error message for unknown tag
Change-Id: I61a31488de17725ae4311f5f5bf8e02edb752cf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12017
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-22 13:36:08 +00:00
Yureka
e408783bac test(tvix/composition): add recursion and concurrent tests
Change-Id: Ic10773b08d940b45c3067bd514c3080cadac9606
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12003
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 21:18:17 +00:00
Yureka
eea9f3d9ee docs(tvix/composition): add comment about stack
Change-Id: I3888b5034c53728e2f9cfe24213f8854eb38bfe1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12005
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 18:37:50 +00:00
Yureka
76394daca3 feat(tvix/composition): downcast boxed error if possible
We still have the unique store name to identify which instantiation caused the error. For recursion errors, the full chain is still retained inside the CompositionError.

Change-Id: Iaddcece445a5df331e578d7c69d710db3d5f8dcd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12002
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 17:03:27 +00:00
Yureka
a805a80b48 fix(tvix/composition): include typeid in recursion check
Change-Id: Icc279d41a4980d4b57acbb4243bbd509039b753f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12000
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 13:20:00 +00:00
Yureka
ca8e2b9fbf chore(tvix): upgrade to tonic 0.12 / hyper 1.0
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 11:01:38 +00:00
Yureka
8b77c7fcd7 refactor(tvix/store): use composition in tvix_store crate
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-20 19:37:27 +00:00
Florian Klink
0244ae6eaf feat(tvix/castore/protos): return root node after validation
This allows avoiding a `.node.unwrap()`` after validation.

Change-Id: Ieef1ffebab16cdca94c979ca6831a7ab4f6007da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11989
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-20 16:35:07 +00:00
Yureka
4801d6bf85 feat(tvix): different service types in one composition
This will be necessary for the PathInfoService composition, as some
PathInfoService implementations require a BlobService & DirectoryService
to ingest into.

Using the Extend trait for creating compositions allows extending the same
composition with configs of various types e.g. BlobStore, DirectoryStore
Generics are moved from the Composition struct to the functions.The storage of
the InstantiatonStates uses the TypeId in the key and a Box<dyn Any> in the
value, which is downcasted to InstantiatonState<T>.

Change-Id: I2af11f26c535029adfb1c62905e0e7c4aaed7b51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11980
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
2024-07-19 15:13:37 +00:00
Yureka
7ccdf6dad5 test(tvix/castore/ObjectStoreBlobService): test correct chunking
Change-Id: I4a8f5cda2705560a6d52a2eb2be0dde35b784c8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11982
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-19 12:57:41 +00:00
Yureka
3c160731ed fix(tvix/castore/blob/object_store): empty chunk list for single chunk
Change-Id: Ie3609ac6184ecead11c4e03073433c26ce129848
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11981
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-19 10:55:47 +00:00
Yureka
168e4fda59 refactor(tvix): use composition & registry for from_addr
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
2024-07-18 19:19:26 +00:00
Yureka
79317be214 feat(tvix/castore): add sled to composition registry
Change-Id: I03fa8dfabcee14c5f657380f86bb1a7aa00e08ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11977
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-18 19:19:10 +00:00
Yureka
b8415a94d9 feat(tvix/composition): allow creating a blank CompositionContext
this is useful when oneshot-instantiating a store from a single config

Change-Id: I08538fdee1d0bb26b3ae2da7d3b2339b2e93bc0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11975
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-18 18:47:42 +00:00
Yureka
1a6b6e3ef3 feat(tvix/castore): add composition module
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748
Signed-off-by: Yureka <tvl@yuka.dev>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-18 10:42:11 +00:00
Ilan Joselevich
4f7b19701c fix(tvix/{store,castore,build}): Compile tonic w/o config in build.rs
Previously we had to make a mutable Config instance and set bytes and
other values in it because they were not exposed to the builder pattern
(https://github.com/hyperium/tonic/issues/908) but now they are, so we
just set them through the builder.

Change-Id: I8904c6b93f09173b56586024b1ced59d622bce66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11966
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 13:24:22 +00:00
Florian Klink
d17c3d96b6 refactor(tvix): point SSL_CERT_FILE to /dev/null
reqwest wants to be able to read a file of trust roots when constructed,
but as it doesn't actually do any HTTPS connections inside the nix
build, an empty list of trust roots is totally sufficient.

Thankfully /dev/null provides such a file.

Change-Id: I9bd1619b2c9f8ff2a6640d2ac410d4de5b20c2ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-07-07 15:11:29 +00:00
Florian Klink
f6c759de58 chore(tvix): bump bigtable_rs to latest release
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a
while, no need to use our own checkout here.

Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-07-01 15:53:53 +00:00
Florian Klink
afbe995aa9 chore(tvix): bump to data-encoding 2.6.0
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-01 06:57:01 +00:00
Florian Klink
bba64d8762 fix(tvix/castore/object_store): make query pairs object_store opts
We previously called ObjectStoreBlobService::parse_url, which passes an
empty list of options when constructing the ObjectStore.

This is most likely not what we want. The more reasonable thing to do is
pass along the query string (pairs) as options to
`object_store::parse_url_opts`, and remove them from the plain URL we
pass to object_store itself.

Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-29 19:23:57 +00:00
Florian Klink
d0ef6a50df refactor(tvix/castore/blobsvc): inline things a bit
Get rid of the `let grpc_client` and `let resp` in some cases.

Change-Id: Idc1c0f566a3b1b48da62e6f1977b07620656b16c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11884
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 17:51:33 +00:00
Florian Klink
10c7701ac1 docs(tvix/castore): drop wrong comment
This doesnt panic anymore.

Change-Id: I1987f8b0640b1e5fe33abcbf46164449c766fd0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11883
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-06-29 17:51:33 +00:00
Florian Klink
7f625e2fc1 refactor(tvix/*store): remove some trait bounds
We don't need to require these things for these impl blocks yet.

Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Simon Hauser
639a00e2ab feat(tvix/tracing): gRPC trace context propagation
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.

The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.

This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.

Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-20 19:21:01 +00:00
Simon Hauser
2b20d8d82d feat(tvix/castore): instrument tokio task with current span
By default tokio::spawn does not instrument the spawned task with the
current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do
this manually for all tokio::spawn functions in functions that are
instrumented.

Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 15:48:14 +00:00
Florian Klink
28b692fd50 feat(tvix/tvix-store): improve progress bars
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.

Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.

Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.

Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 12:57:34 +00:00
Florian Klink
7e42b4f314 fix(tvix/store/bin): fix shutdown behaviour for FUSE
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.

So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.

Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.

Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.

Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.

Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-06-16 12:02:11 +00:00
Yureka
452299dcd2 feat(castore/directory): add cache combinator
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 08:42:53 +00:00
Yureka
30e72d2d52 refactor(castore/directory/objectstore): use new order validation logic
Use the new helper introduced in CL 11708 instead of rolling our own.

Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 08:40:20 +00:00