mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/tvix/store/Cargo.toml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

97 lines
2.2 KiB
TOML
Raw Normal View History

feat(tvix/store): init Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-11 23:48:24 +01:00
[package]
chore(tvix/store): drop separate bin package Apparently, having multiple packages with the same path is a bad thing: ``` The bin target `tvix-store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)` has the same output filename as the lib target `tvix_store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)`. Colliding filename is: /home/flokli/tvl/tvix/target/doc/tvix_store/index.html The output filenames should be unique. This is a known bug where multiple crates with the same name use the same path; see <https://github.com/rust-lang/cargo/issues/6313>. ``` Change-Id: Ic785c0349070783baf5e8fd23f5fb60603a3c995 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8308 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 22:36:10 +01:00
name = "tvix-store"
feat(tvix/store): init Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-11 23:48:24 +01:00
version = "0.1.0"
edition = "2021"
[dependencies]
feat(tvix/store): implement Directory::validate() Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 18:10:46 +01:00
anyhow = "1.0.68"
refactor(tvix/store): Asyncify PathInfoService and DirectoryService We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-19 18:46:41 +02:00
async-stream = "0.3.5"
feat(tvix/store): add size() and digest() functions for Directory Change-Id: I3a9e3c77da1d5d7bfb93da0cd7bce8d48dbc8cd4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7254 Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-12 00:40:09 +01:00
blake3 = { version = "1.3.1", features = ["rayon", "std"] }
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
bytes = "1.4.0"
feat(tvix/store): add NARCalculationService trait and impl This adds a NARCalculationService trait, which will take a root node, and provide a proto::CalculateNarResponse in return. It also adds a NonCachingNARCalculationService implementation, that will simply always render the NAR in memory to calculate the size and sha256, without any caching. Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-02-13 16:47:22 +01:00
clap = { version = "4.0", features = ["derive", "env"] }
count-write = "0.1.0"
feat(tvix/store): add nixbase32 mod This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 22:47:02 +01:00
data-encoding = "2.3.3"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
futures = "0.3.28"
feat(tvix/store): add nixbase32 mod This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 22:47:02 +01:00
lazy_static = "1.4.0"
feat(tvix/store): use rayon to upload chunks concurrently Look at the data that's written to us, and upload all chunks but the rest in parallel, using rayon. This required moving `upload_chunk` outside the struct, and accepting a ChunkService to use for upload (which it was previously getting from `self.chunk_service`). This doesn't speed up things too much for now, because things are still mostly linear. Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-01 18:30:54 +01:00
nix-compat = { path = "../nix-compat" }
refactor(tvix/store): Asyncify PathInfoService and DirectoryService We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-19 18:46:41 +02:00
parking_lot = "0.12.1"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
pin-project-lite = "0.2.13"
feat(tvix/store): build rust proto files This uses prost-build to build the proto files. Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-12 00:07:48 +01:00
prost = "0.11.2"
feat(tvix/store): add NARCalculationService trait and impl This adds a NARCalculationService trait, which will take a root node, and provide a proto::CalculateNarResponse in return. It also adds a NonCachingNARCalculationService implementation, that will simply always render the NAR in memory to calculate the size and sha256, without any caching. Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-02-13 16:47:22 +01:00
sha2 = "0.10.6"
feat(tvix/store): implement PathInfoService with sled This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 21:39:28 +01:00
sled = { version = "0.34.7", features = ["compression"] }
feat(tvix/store): implement Directory::validate() Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 18:10:46 +01:00
thiserror = "1.0.38"
refactor(tvix/store): Asyncify PathInfoService and DirectoryService We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-19 18:46:41 +02:00
tokio-stream = { version = "0.1.14", features = ["fs"] }
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
tokio-util = { version = "0.7.8", features = ["io", "io-util"] }
refactor(tvix/store/blobsvc): make BlobStore async We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-13 14:20:21 +02:00
tokio = { version = "1.28.0", features = ["fs", "net", "rt-multi-thread", "signal"] }
feat(tvix/store): use tonic-build for protos While prost-build already exposes protobuf message types as structs, we actually need tonic-build too, to be able to get traits for all the RPC services defined in the proto files. Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-13 00:23:14 +01:00
tonic = "0.8.2"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
tower = "0.4.13"
feat(tvix/store): add logging with tracing This uses [tracing](https://github.com/tokio-rs/tracing) for logs/ tracing. Annotate all method handlers with an instrument macro, and warn! a message for them being unimplemented. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 17:17:53 +01:00
tracing = "0.1.37"
feat(tvix/store): add --json arg This configures logging as JSON. Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-02-16 16:49:22 +01:00
tracing-subscriber = { version = "0.3.16", features = ["json"] }
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
tvix-castore = { path = "../castore" }
feat(tvix/store/blobsvc): add from_addr This allows constructing blob stores with a URL syntax at runtime, by passing the --blob-service-addr arg. We probably still want to have some builder pattern here, to allow additional schemes to be registered. Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-09 18:07:00 +02:00
url = "2.4.0"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
walkdir = "2.4.0"
feat(tvix/store): Support listening on UNIX domain sockets This adds support for listening on UNIX domain sockets via the tokio-listener crate. The crate will automatically determine whether to start a TCP or UNIX domain socket server based on the listen address. Unfortunately, it's not compatible with tonic right out of the box so I added some wrapper types to implement the necessary traits to make things work. We should investigate upstreaming a `tonic` option to the tokio-listener crate which implements the relevant `tonic` traits. Example: ``` $ tvix-store daemon -l /run/tvix-store.sock INFO tvix_store: tvix-store listening on /run/tvix-store.sock $ tvix-store mount -l /mnt/tvix --blob-service-addr grpc+unix:///run/tvix-store.sock --directory-service-addr grpc+unix:///run/tvix-store.sock --path-info-service-addr grpc+unix:///run/tvix-store.sock $ ls /mnt/tvix ``` Change-Id: I91c4a4b0c5a177b3b90e6c01a4e5d263130e6bdb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9429 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-23 02:45:01 +02:00
tokio-listener = { version = "0.2.1" }
feat(tvix/store): build rust proto files This uses prost-build to build the proto files. Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-12 00:07:48 +01:00
refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rs This switches the FUSE implementation from fuser to fuse-backend-rs. fuse-backend-rs is designed to work with both FUSE and virtiofs. Virtiofs support will make it possible to plug the tvix-store into a microvm and have `/nix/store` access without having to setup FUSE inside the guest. Additionally fuse-backend-rs has nice support for running multiple FUSE threads and has some async support. The goal of this commit is to mechanically switch over to fuse-backend-rs with minimal changes. I did have to add some locks here and there because fuse-backend-rs uses `&self` on all methods whereas fuser uses `&mut self`. `&self` is required for concurrent access to the FUSE server, so this makes sense. We can consider switching to concurrent maps and use some other techniques to reduce lock contention and critical section size. Issue: https://b.tvl.fyi/issues/305 Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-16 17:20:34 +02:00
[dependencies.fuse-backend-rs]
chore(tvix/store): add fuse feature This brings in fuse (via the `fuser` crate), and adds pkg-config and libfuse to the dev shell, so `cargo build` can link against it. Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-28 09:22:08 +02:00
optional = true
fix(tvix/store): Fix FUSE support on MacOS This partially fixes b/312 and gets FUSE to work again on MacOS. It is mostly small type changes and an update to fuse-backend-rs because upstream currently doesn't work with MacFuse. It also sets the default FUSE thread count on MacOS to 1 because otherwise the mount command will hang when shutting down as only one thread gets ENODEV and all the others just keep blocking. Change-Id: Ifb3c4268caf296c487049c1dc4618acb32497f44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9490 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de>
2023-09-29 18:50:50 +02:00
# TODO: Switch back to upstream version once https://github.com/cloud-hypervisor/fuse-backend-rs/pull/157 lands.
git = "https://github.com/griff/fuse-backend-rs"
branch = "macfuse-fix"
feat(tvix/store/fs): Add support for virtiofs backend This adds a virtiofs daemon implementation which hooks into the existing tvix-store filesystem implementation that is used for FUSE. This allows adding the filesystem to a microvm without having to set up FUSE inside the guest. Change-Id: If80c36c9657f2289853e8d9a364bf4f1f7b7559c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9344 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-16 20:58:52 +02:00
[dependencies.vhost]
optional = true
version = "0.6"
[dependencies.vhost-user-backend]
optional = true
version = "0.8"
[dependencies.virtio-queue]
optional = true
version = "0.7"
[dependencies.vm-memory]
optional = true
version = "0.10"
[dependencies.vmm-sys-util]
optional = true
version = "0.11"
[dependencies.virtio-bindings]
optional = true
version = "0.2.1"
chore(tvix/store): add fuse feature This brings in fuse (via the `fuser` crate), and adds pkg-config and libfuse to the dev shell, so `cargo build` can link against it. Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-28 09:22:08 +02:00
feat(tvix/store): implement reflection This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-26 02:14:02 +01:00
[dependencies.tonic-reflection]
optional = true
version = "0.5.0"
chore(tvix/store): add fuse feature This brings in fuse (via the `fuser` crate), and adds pkg-config and libfuse to the dev shell, so `cargo build` can link against it. Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-28 09:22:08 +02:00
[dependencies.libc]
optional = true
version = "0.2.144"
feat(tvix/store): build rust proto files This uses prost-build to build the proto files. Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-12 00:07:48 +01:00
[build-dependencies]
prost-build = "0.11.2"
feat(tvix/store): use tonic-build for protos While prost-build already exposes protobuf message types as structs, we actually need tonic-build too, to be able to get traits for all the RPC services defined in the proto files. Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-13 00:23:14 +01:00
tonic-build = "0.8.2"
feat(tvix/store): add nixbase32 mod This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 22:47:02 +01:00
[dev-dependencies]
test-case = "2.2.2"
feat(tvix/store): implement PathInfoService with sled This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 21:39:28 +01:00
tempfile = "3.3.0"
feat(tvix/store): implement reflection This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-26 02:14:02 +01:00
[features]
fix(tvix/store): Remove virtiofs from default features vhost-user-backend doesn't support macOS yet, so the virtiofs features will not work on macOS. This removes it as a default feature which makes `cargo build` work out of the box on macOS. The `virtiofs` feature is enabled for Linux when building via Nix, but if being built by cargo directly, the feature must be enabled via a cargo flag. Change-Id: I2aaca9582f8e3dbcf9ee5f1b9831d614909f3799 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9555 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-10-07 04:17:35 +02:00
default = ["fuse", "tonic-reflection"]
refactor(tvix/store/fs): Separate FUSE and filesystem code In prepration for adding virtiofs support, I thought it would make sense to split out the filesystem implementation from FUSE itself. The `fs` module holds the tvix-store filesystem implemetation and the `fuse` module holds the code to spawn a FUSE daemon backed by multiple threads. Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-16 20:54:10 +02:00
fs = ["dep:libc", "dep:fuse-backend-rs"]
chore(tvix/store): only depend on vhost-user-fs if virtiofs enabled Change-Id: Iaf88d43df27b87548794a8dddc9a1e08279b8ffb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9471 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 09:18:59 +02:00
virtiofs = [
"fs",
"dep:vhost",
"dep:vhost-user-backend",
"dep:virtio-queue",
"dep:vm-memory",
"dep:vmm-sys-util",
"dep:virtio-bindings",
"fuse-backend-rs?/vhost-user-fs", # impl FsCacheReqHandler for SlaveFsCacheReq
"fuse-backend-rs?/virtiofs",
]
refactor(tvix/store/fs): Separate FUSE and filesystem code In prepration for adding virtiofs support, I thought it would make sense to split out the filesystem implementation from FUSE itself. The `fs` module holds the tvix-store filesystem implemetation and the `fuse` module holds the code to spawn a FUSE daemon backed by multiple threads. Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-16 20:54:10 +02:00
fuse = ["fs"]
chore(tvix/[ca]store): allow building without tonic-reflection - rename the feature to `tonic-reflection` in both tvix-store and tvix- castore. - set it to disabled in tvix-castore by default - enable it in tvix-store, and pull in tvix-castore with the feature enabled. Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 09:30:35 +02:00
tonic-reflection = ["dep:tonic-reflection", "tvix-castore/tonic-reflection"]
Reference in a new issue Copy permalink