mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/tvix/castore/Cargo.toml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

102 lines
2.1 KiB
TOML
Raw Normal View History

refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
[package]
name = "tvix-castore"
version = "0.1.0"
edition = "2021"
[dependencies]
async-stream = "0.3.5"
feat(tvix/castore/digests): impl From digest::Output<_> for B3Digest This allows calling .into() to get a B3Digest. Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 13:40:54 +01:00
async-tempfile = "0.4.0"
blake3 = { version = "1.3.1", features = ["rayon", "std", "traits-preview"] }
bstr = "1.6.0"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
bytes = "1.4.0"
data-encoding = "2.3.3"
feat(tvix/castore/digests): impl From digest::Output<_> for B3Digest This allows calling .into() to get a B3Digest. Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-03 13:40:54 +01:00
digest = "0.10.7"
feat(tvix/castore/blobsvc): add object storage implementation This uses the `object_store` crate to expose a tvix-castore BlobService backed by object storage. It's using FastCDC to chunk blobs into smaller chunks when writing to it. These are exposed at the .chunks() method. Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-03-01 17:00:53 +01:00
fastcdc = { version = "3.1.0", features = ["tokio"] }
chore(tvix): switch to upstream futures 0.3.30 The bugs have been fixed, https://github.com/rust-lang/futures-rs/pull/2801 and https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended up in that release. Change-Id: Iefd990d2d1719b884504093343e54e9c5258e2e2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10414 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-24 21:56:48 +01:00
futures = "0.3.30"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
lazy_static = "1.4.0"
refactor(tvix/castore): introduce "cloud" feature flag This controls whether tvix-castore has support for various cloud backends or not. Use this to control the set of feature flags for the object_store backend, and only enable the aws, azure and gcp ones if it's set. In the future this can be used to enable/disable other cloud backends too. Without feature flags, `object_store` already supports the `InMemory` and `LocalFilesystem` backends, and we also want to unconditionally enable the `http` one. Make sure at least the construction of these services is covered in the tests. Similarly, the tvix-store crate, which provides the tvix-store CLI has a `cloud` feature flag too (defaulting to enabled). Change-Id: I9fb9c87b740e7dc83f8ff7a0862905d036d513f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11204 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-19 10:46:12 +01:00
object_store = { version = "0.9.1", features = ["http"] }
chore(tvix): move store/fs to castore/fs With the recent introduction of the RootNodes trait, there's nothing in the fs module pulling in tvix-store dependencies, so it can live in tvix-castore. This allows other crates to make use of TvixStoreFS, without having to pull in tvix-store. For example, a tvix-build using a fuse mountpoint at /nix/store doesn't need a PathInfoService to hold the root nodes that should be present, but just a list. tvix-store now has a pathinfoservice/fs module, which contains the necessary glue logic to implement the RootNodes trait for a PathInfoService. To satisfy Rust orphan rules for trait implementations, we had to add a small wrapper struct. It's mostly hidden away by the make_fs helper function returning a TvixStoreFs. It can't be entirely private, as its still leaking into the concrete type of TvixStoreFS. tvix-store still has `fuse` and `virtiofs` features, but they now simply enable these features in the `tvix-castore` crate they depend on. The tests for the fuse functionality stay in tvix-store for now, as they populate the root nodes through a PathInfoService. Once above mentioned "list of root nodes" implementation exists, we might want to shuffle this around one more time. Fixes b/341. Change-Id: I989f664827a5a361b23b34368d242d10c157c756 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-16 23:16:22 +01:00
parking_lot = "0.12.1"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
pin-project-lite = "0.2.13"
chore(tvix): bump tokio[-util], tonic, prost This bumps all these dependencies to their more recent versions. We have to enable the `macros` tonic feature, and looks like we should also enable `rt`, not only `rt-multi-thread`. Change-Id: Icc3600848fca0bacce24f4889d088e75711594ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8984 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 11:47:33 +02:00
prost = "0.12.1"
chore(tvix): drop sled compression support It's been a while since the last sled release, and that one binds to a pretty old version of zstd, requiring workarounds like cl/10090. Upstream sled main branch currently has zstd halfway patched out (it's a no-op, but the feature flag and options are still there), and it's in that state for a year. Rather than maintaining our own fork of sled, let's just stop using the compression feature in sled, dropping the version pin to zstd that way, removing the need for cl/10090. This doesn't mean we won't reintroduce per-blob compression - but we probably just won't let sled take care of the compression, but do it ourselves - which is necessary for more chunked blob storage anyways. Even though we do drop the feature flag, we still need to explicitly use use_compression(false). Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-11 20:25:34 +01:00
sled = { version = "0.34.7" }
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
thiserror = "1.0.38"
tokio-stream = { version = "0.1.14", features = ["fs", "net"] }
chore(tvix): bump tokio[-util], tonic, prost This bumps all these dependencies to their more recent versions. We have to enable the `macros` tonic feature, and looks like we should also enable `rt`, not only `rt-multi-thread`. Change-Id: Icc3600848fca0bacce24f4889d088e75711594ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8984 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 11:47:33 +02:00
tokio-util = { version = "0.7.9", features = ["io", "io-util"] }
tokio = { version = "1.32.0", features = ["fs", "macros", "net", "rt", "rt-multi-thread", "signal"] }
chore(tvix): bump tonic to 0.11.0 This bumps tonic and surrounding crates to 0.11.x. We added support for tonic 0.11.x into tokio-listener (https://github.com/vi/tokio-listener/pull/4), so that's bumped as well. Change-Id: Icfade5894403228299836fefb21b2f9ae59dbebb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11156 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-15 21:53:43 +01:00
tonic = "0.11.0"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
tower = "0.4.13"
tracing = "0.1.37"
url = "2.4.0"
walkdir = "2.4.0"
feat(tvix/castore/blobsvc): add object storage implementation This uses the `object_store` crate to expose a tvix-castore BlobService backed by object storage. It's using FastCDC to chunk blobs into smaller chunks when writing to it. These are exposed at the .chunks() method. Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-03-01 17:00:53 +01:00
zstd = "0.13.0"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
chore(tvix): move store/fs to castore/fs With the recent introduction of the RootNodes trait, there's nothing in the fs module pulling in tvix-store dependencies, so it can live in tvix-castore. This allows other crates to make use of TvixStoreFS, without having to pull in tvix-store. For example, a tvix-build using a fuse mountpoint at /nix/store doesn't need a PathInfoService to hold the root nodes that should be present, but just a list. tvix-store now has a pathinfoservice/fs module, which contains the necessary glue logic to implement the RootNodes trait for a PathInfoService. To satisfy Rust orphan rules for trait implementations, we had to add a small wrapper struct. It's mostly hidden away by the make_fs helper function returning a TvixStoreFs. It can't be entirely private, as its still leaking into the concrete type of TvixStoreFS. tvix-store still has `fuse` and `virtiofs` features, but they now simply enable these features in the `tvix-castore` crate they depend on. The tests for the fuse functionality stay in tvix-store for now, as they populate the root nodes through a PathInfoService. Once above mentioned "list of root nodes" implementation exists, we might want to shuffle this around one more time. Fixes b/341. Change-Id: I989f664827a5a361b23b34368d242d10c157c756 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-16 23:16:22 +01:00
[dependencies.fuse-backend-rs]
optional = true
version = "0.11.0"
[dependencies.libc]
optional = true
version = "0.2.144"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
[dependencies.tonic-reflection]
optional = true
chore(tvix): bump tonic to 0.11.0 This bumps tonic and surrounding crates to 0.11.x. We added support for tonic 0.11.x into tokio-listener (https://github.com/vi/tokio-listener/pull/4), so that's bumped as well. Change-Id: Icfade5894403228299836fefb21b2f9ae59dbebb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11156 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-15 21:53:43 +01:00
version = "0.11.0"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
chore(tvix): move store/fs to castore/fs With the recent introduction of the RootNodes trait, there's nothing in the fs module pulling in tvix-store dependencies, so it can live in tvix-castore. This allows other crates to make use of TvixStoreFS, without having to pull in tvix-store. For example, a tvix-build using a fuse mountpoint at /nix/store doesn't need a PathInfoService to hold the root nodes that should be present, but just a list. tvix-store now has a pathinfoservice/fs module, which contains the necessary glue logic to implement the RootNodes trait for a PathInfoService. To satisfy Rust orphan rules for trait implementations, we had to add a small wrapper struct. It's mostly hidden away by the make_fs helper function returning a TvixStoreFs. It can't be entirely private, as its still leaking into the concrete type of TvixStoreFS. tvix-store still has `fuse` and `virtiofs` features, but they now simply enable these features in the `tvix-castore` crate they depend on. The tests for the fuse functionality stay in tvix-store for now, as they populate the root nodes through a PathInfoService. Once above mentioned "list of root nodes" implementation exists, we might want to shuffle this around one more time. Fixes b/341. Change-Id: I989f664827a5a361b23b34368d242d10c157c756 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-16 23:16:22 +01:00
[dependencies.vhost]
optional = true
version = "0.6"
[dependencies.vhost-user-backend]
optional = true
version = "0.8"
[dependencies.virtio-queue]
optional = true
version = "0.7"
[dependencies.vm-memory]
optional = true
version = "0.10"
[dependencies.vmm-sys-util]
optional = true
version = "0.11"
[dependencies.virtio-bindings]
optional = true
version = "0.2.1"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
[build-dependencies]
chore(tvix): bump tokio[-util], tonic, prost This bumps all these dependencies to their more recent versions. We have to enable the `macros` tonic feature, and looks like we should also enable `rt`, not only `rt-multi-thread`. Change-Id: Icc3600848fca0bacce24f4889d088e75711594ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8984 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 11:47:33 +02:00
prost-build = "0.12.1"
chore(tvix): bump tonic to 0.11.0 This bumps tonic and surrounding crates to 0.11.x. We added support for tonic 0.11.x into tokio-listener (https://github.com/vi/tokio-listener/pull/4), so that's bumped as well. Change-Id: Icfade5894403228299836fefb21b2f9ae59dbebb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11156 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-15 21:53:43 +01:00
tonic-build = "0.11.0"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
[dev-dependencies]
feat(tvix/castore): add rstest-based DirectoryService tests This creates test scenarios (using the DirectoryService trait) that we want all DirectoryService implementations to pass. Some of these tests are ported from proto::tests::grpc_directoryservice, which tested this on the gRPC interface (rather than the trait), some others ensure certain behaviour for which we only recently introduced general checking logic (through ClosureValidator). We also borrow some code related to setting up a gRPC DirectoryService client (connecting to a server exposing a in-memory DiretoryService) from castore::utils, this will be deleted once it's all ported over. Change-Id: I6810215a76101f908e2aaecafa803c70d85bc552 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11247 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-23 21:49:49 +01:00
rstest = "0.18.2"
chore(tvix): bump test-case dep to 3.3.1 Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-01-05 15:12:35 +01:00
test-case = "3.3.1"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
tempfile = "3.3.0"
test(tvix/castore): use tokio-retry for exp backoff Rather than using this loop, use exponential backoff while waiting for the socket path to be created. Change-Id: I18706a64ce06f8916a07892dfbcd409ac5b3bff1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9568 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-08 13:33:43 +02:00
tokio-retry = "0.3.0"
refactor(tvix): condense long bytestrings Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 03:12:26 +02:00
hex-literal = "0.4.1"
feat(tvix/castore): add rstest-based DirectoryService tests This creates test scenarios (using the DirectoryService trait) that we want all DirectoryService implementations to pass. Some of these tests are ported from proto::tests::grpc_directoryservice, which tested this on the gRPC interface (rather than the trait), some others ensure certain behaviour for which we only recently introduced general checking logic (through ClosureValidator). We also borrow some code related to setting up a gRPC DirectoryService client (connecting to a server exposing a in-memory DiretoryService) from castore::utils, this will be deleted once it's all ported over. Change-Id: I6810215a76101f908e2aaecafa803c70d85bc552 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11247 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-23 21:49:49 +01:00
rstest_reuse = "0.6.0"
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
[features]
chore(tvix/[ca]store): allow building without tonic-reflection - rename the feature to `tonic-reflection` in both tvix-store and tvix- castore. - set it to disabled in tvix-castore by default - enable it in tvix-store, and pull in tvix-castore with the feature enabled. Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 09:30:35 +02:00
default = []
refactor(tvix/castore): introduce "cloud" feature flag This controls whether tvix-castore has support for various cloud backends or not. Use this to control the set of feature flags for the object_store backend, and only enable the aws, azure and gcp ones if it's set. In the future this can be used to enable/disable other cloud backends too. Without feature flags, `object_store` already supports the `InMemory` and `LocalFilesystem` backends, and we also want to unconditionally enable the `http` one. Make sure at least the construction of these services is covered in the tests. Similarly, the tvix-store crate, which provides the tvix-store CLI has a `cloud` feature flag too (defaulting to enabled). Change-Id: I9fb9c87b740e7dc83f8ff7a0862905d036d513f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11204 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-19 10:46:12 +01:00
cloud = [
"object_store/aws",
"object_store/azure",
"object_store/gcp",
]
chore(tvix): move store/fs to castore/fs With the recent introduction of the RootNodes trait, there's nothing in the fs module pulling in tvix-store dependencies, so it can live in tvix-castore. This allows other crates to make use of TvixStoreFS, without having to pull in tvix-store. For example, a tvix-build using a fuse mountpoint at /nix/store doesn't need a PathInfoService to hold the root nodes that should be present, but just a list. tvix-store now has a pathinfoservice/fs module, which contains the necessary glue logic to implement the RootNodes trait for a PathInfoService. To satisfy Rust orphan rules for trait implementations, we had to add a small wrapper struct. It's mostly hidden away by the make_fs helper function returning a TvixStoreFs. It can't be entirely private, as its still leaking into the concrete type of TvixStoreFS. tvix-store still has `fuse` and `virtiofs` features, but they now simply enable these features in the `tvix-castore` crate they depend on. The tests for the fuse functionality stay in tvix-store for now, as they populate the root nodes through a PathInfoService. Once above mentioned "list of root nodes" implementation exists, we might want to shuffle this around one more time. Fixes b/341. Change-Id: I989f664827a5a361b23b34368d242d10c157c756 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-16 23:16:22 +01:00
fs = ["dep:libc", "dep:fuse-backend-rs"]
virtiofs = [
"fs",
"dep:vhost",
"dep:vhost-user-backend",
"dep:virtio-queue",
"dep:vm-memory",
"dep:vmm-sys-util",
"dep:virtio-bindings",
"fuse-backend-rs?/vhost-user-fs", # impl FsCacheReqHandler for SlaveFsCacheReq
"fuse-backend-rs?/virtiofs",
]
fuse = ["fs"]
test(tvix/castore): use tokio-retry for exp backoff Rather than using this loop, use exponential backoff while waiting for the socket path to be created. Change-Id: I18706a64ce06f8916a07892dfbcd409ac5b3bff1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9568 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-08 13:33:43 +02:00
tonic-reflection = ["dep:tonic-reflection"]
Reference in a new issue Copy permalink