mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/tvix/store/Cargo.toml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

44 lines
1 KiB
TOML
Raw Normal View History

feat(tvix/store): init Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-11 23:48:24 +01:00
[package]
chore(tvix/store): make importable This allows other crates to import tvix_store. Rename the bin crate to tvix-store-bin, to avoid having multiple crates with the same name (https://github.com/rust-lang/cargo/issues/6313) Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-02 14:37:08 +01:00
name = "tvix-store-bin"
feat(tvix/store): init Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-11 23:48:24 +01:00
version = "0.1.0"
edition = "2021"
chore(tvix/store): make importable This allows other crates to import tvix_store. Rename the bin crate to tvix-store-bin, to avoid having multiple crates with the same name (https://github.com/rust-lang/cargo/issues/6313) Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-02 14:37:08 +01:00
[lib]
name = "tvix_store"
feat(tvix/store): init Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-11 23:48:24 +01:00
[dependencies]
feat(tvix/store): implement Directory::validate() Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 18:10:46 +01:00
anyhow = "1.0.68"
feat(tvix/store): add size() and digest() functions for Directory Change-Id: I3a9e3c77da1d5d7bfb93da0cd7bce8d48dbc8cd4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7254 Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-12 00:40:09 +01:00
blake3 = { version = "1.3.1", features = ["rayon", "std"] }
feat(tvix/store): add NARCalculationService trait and impl This adds a NARCalculationService trait, which will take a root node, and provide a proto::CalculateNarResponse in return. It also adds a NonCachingNARCalculationService implementation, that will simply always render the NAR in memory to calculate the size and sha256, without any caching. Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-02-13 16:47:22 +01:00
clap = { version = "4.0", features = ["derive", "env"] }
count-write = "0.1.0"
feat(tvix/store): add nixbase32 mod This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 22:47:02 +01:00
data-encoding = "2.3.3"
feat(tvix/store): bump fastcdc, use v2020 version This switches away from the less canonical "ronomon" version to the implementation as described in the [paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et al., in 2020. That version uses 64-bit hash values and tends to be faster than both the ronomon and v2016 versions, and produces the same chunking as the 2016 version. As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's the recommended choice. The crate also gained support for streaming version of chunkers: https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be useful. Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-02-17 18:15:21 +01:00
fastcdc = "3.0.0"
feat(tvix/store): add nixbase32 mod This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 22:47:02 +01:00
lazy_static = "1.4.0"
feat(tvix/store): build rust proto files This uses prost-build to build the proto files. Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-12 00:07:48 +01:00
prost = "0.11.2"
feat(tvix/store): add NARCalculationService trait and impl This adds a NARCalculationService trait, which will take a root node, and provide a proto::CalculateNarResponse in return. It also adds a NonCachingNARCalculationService implementation, that will simply always render the NAR in memory to calculate the size and sha256, without any caching. Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-02-13 16:47:22 +01:00
sha2 = "0.10.6"
feat(tvix/store): implement PathInfoService with sled This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 21:39:28 +01:00
sled = { version = "0.34.7", features = ["compression"] }
refactor(tvix): introduce nix-compat crate Move nixbase32 and store_path into this. This allows //tvix/cli to not pull in //tvix/store for now. Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 12:18:03 +01:00
nix-compat = { path = "../nix-compat" }
feat(tvix/store): implement Directory::validate() Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 18:10:46 +01:00
thiserror = "1.0.38"
feat(tvix/store): initial dummy implementation This replaces the hello world example from tvix-store with an actual gRPC endpoint, implementing all of BlobService, DirectoryService and PathInfoService. All RPC methods currently respond with the unimplemented gRPC status. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 16:40:28 +01:00
tokio-stream = "0.1.11"
refactor(tvix): introduce nix-compat crate Move nixbase32 and store_path into this. This allows //tvix/cli to not pull in //tvix/store for now. Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 12:18:03 +01:00
tokio = { version = "1.23.0", features = ["rt-multi-thread"] }
feat(tvix/store): use tonic-build for protos While prost-build already exposes protobuf message types as structs, we actually need tonic-build too, to be able to get traits for all the RPC services defined in the proto files. Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-13 00:23:14 +01:00
tonic = "0.8.2"
feat(tvix/store): add logging with tracing This uses [tracing](https://github.com/tokio-rs/tracing) for logs/ tracing. Annotate all method handlers with an instrument macro, and warn! a message for them being unimplemented. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 17:17:53 +01:00
tracing = "0.1.37"
feat(tvix/store): add --json arg This configures logging as JSON. Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-02-16 16:49:22 +01:00
tracing-subscriber = { version = "0.3.16", features = ["json"] }
feat(tvix/store): build rust proto files This uses prost-build to build the proto files. Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-12 00:07:48 +01:00
feat(tvix/store): implement reflection This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-26 02:14:02 +01:00
[dependencies.tonic-reflection]
optional = true
version = "0.5.0"
feat(tvix/store): build rust proto files This uses prost-build to build the proto files. Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-12 00:07:48 +01:00
[build-dependencies]
prost-build = "0.11.2"
feat(tvix/store): use tonic-build for protos While prost-build already exposes protobuf message types as structs, we actually need tonic-build too, to be able to get traits for all the RPC services defined in the proto files. Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-13 00:23:14 +01:00
tonic-build = "0.8.2"
feat(tvix/store): add nixbase32 mod This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 22:47:02 +01:00
[dev-dependencies]
test-case = "2.2.2"
feat(tvix/store): implement PathInfoService with sled This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 21:39:28 +01:00
tempfile = "3.3.0"
chore(tvix/store): add tonic-mock Upstream seems to be dead, so we're using https://github.com/tyrchen/ tonic-mock/pull/3 here. According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment- 1241164173, we might not need this crate at all, but for now, it gets the job done and is less code to write in the tests. Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-18 20:46:33 +01:00
tonic-mock = { git = "https://github.com/brainrake/tonic-mock", branch = "bump-dependencies" }
feat(tvix/store): implement reflection This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-26 02:14:02 +01:00
[features]
chore(tvix/store): add fastcdc crate This is used for content-defined chunking. Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 14:18:51 +01:00
default = ["reflection"]
reflection = ["tonic-reflection"]
Reference in a new issue Copy permalink