mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/tvix/store/Cargo.toml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

82 lines
2.9 KiB
TOML
Raw Normal View History

feat(tvix/store): init Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-11 23:48:24 +01:00
[package]
chore(tvix/store): drop separate bin package Apparently, having multiple packages with the same path is a bad thing: ``` The bin target `tvix-store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)` has the same output filename as the lib target `tvix_store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)`. Colliding filename is: /home/flokli/tvl/tvix/target/doc/tvix_store/index.html The output filenames should be unique. This is a known bug where multiple crates with the same name use the same path; see <https://github.com/rust-lang/cargo/issues/6313>. ``` Change-Id: Ic785c0349070783baf5e8fd23f5fb60603a3c995 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8308 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 22:36:10 +01:00
name = "tvix-store"
feat(tvix/store): init Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-11 23:48:24 +01:00
version = "0.1.0"
edition = "2021"
[dependencies]
chore(tvix): Migrate members to inherit deps from workspace From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 23:54:46 +02:00
anyhow = { workspace = true }
async-compression = { workspace = true, features = ["tokio", "bzip2", "gzip", "xz", "zstd"] }
async-stream = { workspace = true }
blake3 = { workspace = true, features = ["rayon", "std"] }
bstr = { workspace = true }
bytes = { workspace = true }
clap = { workspace = true, features = ["derive", "env"] }
count-write = { workspace = true }
data-encoding = { workspace = true }
feat(tvix/store): Add a signing PathInfoService - Add a new PathInfoService implementation that wraps transparently around another except that it dynamically signs all the incoming path-infos with the provided signer. - Add a ServiceBuilder for this PathInfoService that provides a SigningPathInfoService with a keyfile signer Change-Id: I845ddfdf01d14c503c796b2b80c720dab98be091
2024-07-22 15:09:10 +02:00
ed25519 = { workspace = true }
ed25519-dalek = { workspace = true }
chore(tvix): Migrate members to inherit deps from workspace From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 23:54:46 +02:00
futures = { workspace = true }
lazy_static = { workspace = true }
feat(tvix/store): use tvix_compat::nar::writer::async Change-Id: Iad36872244df6f2225a2884f6b20cacd8f918b31 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9619 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu>
2023-10-09 23:06:02 +02:00
nix-compat = { path = "../nix-compat", features = ["async"] }
chore(tvix): Migrate members to inherit deps from workspace From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 23:54:46 +02:00
pin-project-lite = { workspace = true }
prost = { workspace = true }
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
serde_with = { workspace = true }
serde_qs = { workspace = true }
sha2 = { workspace = true }
sled = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["fs", "macros", "net", "rt", "rt-multi-thread", "signal"] }
tokio-listener = { workspace = true, features = ["clap", "multi-listener", "sd_listen", "tonic012"] }
tokio-stream = { workspace = true, features = ["fs"] }
tokio-util = { workspace = true, features = ["io", "io-util", "compat"] }
tonic = { workspace = true, features = ["tls", "tls-roots"] }
tower = { workspace = true }
tower-http = { workspace = true, features = ["trace"] }
refactor(tvix): move castore into tvix-castore crate This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 21:32:44 +02:00
tvix-castore = { path = "../castore" }
chore(tvix): Migrate members to inherit deps from workspace From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 23:54:46 +02:00
url = { workspace = true }
walkdir = { workspace = true }
reqwest = { workspace = true, features = ["rustls-tls-native-roots", "stream"] }
reqwest-middleware = { workspace = true }
lru = { workspace = true }
parking_lot = { workspace = true }
feat(tvix/tracing): http trace propagation Introduces a helper function within tvix-tracing that returns a reqwest tracing middleware that will ingest the traceparent if otlp is enabled. It is feature flagged in tvix-tracing so not every consumer of that library automatically has reqwest in its dependencies. Tested using netcat to verify that the `traceparent` header is there if otlp is enabled and missing if otlp feature is disabled. Change-Id: I5abccae777b725f5ff7382e3686165383c477a39 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-06-27 10:27:29 +02:00
tvix-tracing = { path = "../tracing", features = ["tonic", "reqwest"] }
chore(tvix): Migrate members to inherit deps from workspace From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 23:54:46 +02:00
tracing = { workspace = true }
tracing-indicatif = { workspace = true }
hyper-util = { workspace = true }
chore(tvix): Bump versions of all compatible deps via cargo upgrade This updates all the dependencies and their "minimum" versions in Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s `cargo upgrade` command that will eventually be merged into `cargo update`. Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-28 23:42:29 +02:00
toml = { version = "0.8.19", optional = true }
chore(tvix): Migrate members to inherit deps from workspace From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 23:54:46 +02:00
tonic-health = { workspace = true }
redb = { workspace = true }
mimalloc = { workspace = true }
tonic-reflection = { workspace = true, optional = true }
bigtable_rs = { workspace = true, optional = true }
feat(tvix/store): add bigtable pathinfoservice backend Put behind the "cloud" backend, like in the `tvix-castore` crate. Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-05 15:54:50 +02:00
feat(tvix/store): build rust proto files This uses prost-build to build the proto files. Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-12 00:07:48 +01:00
[build-dependencies]
chore(tvix): Migrate members to inherit deps from workspace From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 23:54:46 +02:00
prost-build = { workspace = true }
tonic-build = { workspace = true }
feat(tvix/store): add nixbase32 mod This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 22:47:02 +01:00
[dev-dependencies]
chore(tvix): Migrate members to inherit deps from workspace From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-28 23:54:46 +02:00
async-process = { workspace = true }
rstest = { workspace = true }
rstest_reuse = { workspace = true }
tempfile = { workspace = true }
tokio-retry = { workspace = true }
feat(tvix/store): implement reflection This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-26 02:14:02 +01:00
[features]
refactor(tvix/castore): introduce "cloud" feature flag This controls whether tvix-castore has support for various cloud backends or not. Use this to control the set of feature flags for the object_store backend, and only enable the aws, azure and gcp ones if it's set. In the future this can be used to enable/disable other cloud backends too. Without feature flags, `object_store` already supports the `InMemory` and `LocalFilesystem` backends, and we also want to unconditionally enable the `http` one. Make sure at least the construction of these services is covered in the tests. Similarly, the tvix-store crate, which provides the tvix-store CLI has a `cloud` feature flag too (defaulting to enabled). Change-Id: I9fb9c87b740e7dc83f8ff7a0862905d036d513f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11204 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-19 10:46:12 +01:00
default = ["cloud", "fuse", "otlp", "tonic-reflection"]
feat(tvix/store): add bigtable pathinfoservice backend Put behind the "cloud" backend, like in the `tvix-castore` crate. Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-05 15:54:50 +02:00
cloud = [
"dep:bigtable_rs",
"tvix-castore/cloud"
]
chore(tvix): move store/fs to castore/fs With the recent introduction of the RootNodes trait, there's nothing in the fs module pulling in tvix-store dependencies, so it can live in tvix-castore. This allows other crates to make use of TvixStoreFS, without having to pull in tvix-store. For example, a tvix-build using a fuse mountpoint at /nix/store doesn't need a PathInfoService to hold the root nodes that should be present, but just a list. tvix-store now has a pathinfoservice/fs module, which contains the necessary glue logic to implement the RootNodes trait for a PathInfoService. To satisfy Rust orphan rules for trait implementations, we had to add a small wrapper struct. It's mostly hidden away by the make_fs helper function returning a TvixStoreFs. It can't be entirely private, as its still leaking into the concrete type of TvixStoreFS. tvix-store still has `fuse` and `virtiofs` features, but they now simply enable these features in the `tvix-castore` crate they depend on. The tests for the fuse functionality stay in tvix-store for now, as they populate the root nodes through a PathInfoService. Once above mentioned "list of root nodes" implementation exists, we might want to shuffle this around one more time. Fixes b/341. Change-Id: I989f664827a5a361b23b34368d242d10c157c756 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-16 23:16:22 +01:00
fuse = ["tvix-castore/fuse"]
feat(tvix/tracing): introduce common tvix-tracing crate Introduce a new common crate that contains tracing boilerplate which then can be used in the cli, tvix-store and tvix-build crates. It has otlp as an optional feature, which is currently only used by tvix-store. Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-06 15:44:11 +02:00
otlp = ["tvix-tracing/otlp"]
chore(tvix/[ca]store): allow building without tonic-reflection - rename the feature to `tonic-reflection` in both tvix-store and tvix- castore. - set it to disabled in tvix-castore by default - enable it in tvix-store, and pull in tvix-castore with the feature enabled. Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 09:30:35 +02:00
tonic-reflection = ["dep:tonic-reflection", "tvix-castore/tonic-reflection"]
feat(tvix/cli,store): add tracy feature If compiled with this features, this emits packets compatible with the [Tracy](https://github.com/wolfpld/tracy) format. Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-14 13:17:37 +02:00
tracy = ["tvix-tracing/tracy"]
feat(tvix/store): add opentelemetry This is behind the otlp feature flag (currently enabled by default). By default, this will try to push traces to a OTLP collector running at https://localhost:4317. You can make one available by running: ``` docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6831:6831/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 --rm \ jaegertracing/all-in-one:1.42 ``` Started like that, jaeger brings a web interface at http://localhost:16686/search As documented in https://docs.rs/opentelemetry-otlp/latest/opentelemetry_otlp/, you can point this to another location by setting `OTEL_EXPORTER_OTLP_ENDPOINT`. Change-Id: Id1dca367d70027b2ea98bb70bcf99a68363ec2be Reviewed-on: https://cl.tvl.fyi/c/depot/+/8194 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: aaqaishtyaq <aaqaishtyaq@gmail.com>
2023-03-01 00:37:08 +01:00
virtiofs = ["tvix-castore/virtiofs"]
fix(store): add toml dependency for feature xp-store-composition Fixes 'use of undeclared crate or module ' with --features xp-store-composition Change-Id: I44dce86e656094d180b91a00f385f685d21f3fbd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12021 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 17:01:07 +02:00
xp-store-composition = ["toml"]
test(tvix/[ca]store): enable bigtable only with "integration" feature The emulator and bigtable client are quite big. Remove them from the default //tvix:shell. Put the tests behind a `integration` feature flag, and add a variant with that enabled to CI, and drop the bigtable tools from //tvix:shell. Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-02 23:09:46 +02:00
# Whether to run the integration tests.
# Requires the following packages in $PATH:
# cbtemulator, google-cloud-bigtable-tool
integration = []
Reference in a new issue Copy permalink