We don't actually care if it's a BTreeMap of strings, bstrings or any of
that sort.
The only thing we want to be able to do is get a reference to the bytes
from the keys and values.
Change-Id: I21b85811a9ea47fe06afa3108836ef9295e5d89b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9737
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
We use test_resources and globbing for some of the test cases, so adding
additional files in there will also create new test cases, which we
don't always want.
Move it down one level to make some more space.
Change-Id: I619867dc80a4ced59d45096d0703678663b559cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9729
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Limit the amount of memory consumed on the stack for NixHash. Sha512
isn't used that often, so it's fine if we heap-allocate it.
Change-Id: I4a9eecd20c6184610124dc130c41bfa5d0dc04c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9726
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tonic support was added to tokio-listener upstream which removes the
need for use to have tonic compatibility wrapper types around it.
See: https://github.com/vi/tokio-listener/pull/2
Fixes b/311
Change-Id: I04a2dbb3bc3c8bfe9339583c0b46070c7ec97811
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9721
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is of known length.
Change-Id: Iba48ccc486d5bf9e38ec1a2da6e7b80997d2c6ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9723
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This updates the code to make use of the new Validate() function defined
on a Node.
Change-Id: I9b6ed694661f41e700f19cc78d53d2224b61852d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9718
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This uses the newly introduced StorePath message type to add a Deriver
field to the PathInfo message.
Support for validation is added to both the golang and rust
implementation. This includes extending unit tests.
Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This encodes a store path a bit more concise, which is used in the
Deriver field.
Change-Id: Ibfb54d3b206917e51970d1d5fe94fcedb901704b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9646
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
To make this easier, move it outside of TvixStoreFs, and accept the
DirectoryService as a function argument, so we don't need to worry about
the lifetime of self.
This also aligns with how we spawn async tasks inside the rest of
TvixStoreFs.
Change-Id: I3b95072209d32039f05aed122240f2d6db7ad172
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9713
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This moves the repetitive code to parse a URL and create a channel
connected to it into `tvix_castore::channel::from_url`.
Part of b/308
Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Move the channel creation depending on the string-based URL into its
own block.
Change-Id: I546b769acd2296b548eb966b62c495f910266df5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We had to have these all while the traits where sync, and there was a
lot of spawning and moving.
Most of this can now be removed in favor of some inline `.clone()`.
Change-Id: Id5466c32a403100bc3347866b3172e06a792e311
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Similar to cl/9715, this makes the validation checks more granular,
introducing a Validate on all *Node.
A check for symlink targets is added too.
Once merged, it can also be used from tvix/store/protos.
Change-Id: I0909a89fadcd74b74ef0c9a8a1f22658fccc83b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9716
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We can just check the digest length to be correct, and move the symlink
target checks to a single line.
Change-Id: I41d2e3a50e7990ef6c04f02acd754b1e17b43e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9717
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Implement `validate()` on `node::Node`, and call it from PathInfo's
validate() too. Node-related errors are moved to a ValidateNodeError
error type.
This additionally adds some more validations for symlink targets (they
must not be empty, and not contain null bytes).
Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
This teaches tonic to speak https, so `grpc+https` becomes a working
connection string.
Change-Id: I9c550ea665b829725b545a164cf6f78587bd5213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9714
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
Bump code.tvl.fyi/tvix/store/protos past cl/9649, where Validate()
already ensures the NarSha256 has the correct size.
Change-Id: I774668822f4d9dbd4dea47dde6e4745dc95e8e7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9665
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
As correctly mentioned in
https://cl.tvl.fyi/c/depot/+/9652/comment/03b9b96e_bbb337fd/,
we shouldn't be using these magic constants, but pull them from where
they're defined.
This already is a dependency of go-nix, and pkg/pathinfosvc/server.go,
so no changes in go.mod.
Change-Id: I0cc41ce040fcbddf4b6171417bc9b0de55af4991
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9653
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Explain a bit better what we do in the codeblock below,
and add a newline to make it clear we don't just document the first
branch.
Change-Id: Ifb142d9984e6d2cbca648525c10298f2fcbdfedf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9645
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
We have nixhash.FromHashTypeAndDigest now.
Also, run Validate() on the PathInfo received from the remote
PathInfoService.
Change-Id: I14db0d9356c539c084afc9dd712314b56da2587e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9652
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
… and nar size / sha256 digest.
Instead of producing sparse PathInfo messages when NARs are sent to
nar-bridge, the nar-bridge http server now keeps a lookup table
(narsha256) -> (rootNode, narSize)
This removes a whole bunch of noise, because we don't need to keep
sparse fields around.
A convenience function
`GenPathInfo(rootNode *castorev1pb.Node, narInfo *narinfo.NarInfo)` is
added, which is used to produce PathInfo messages, either when receiving
a NAR file over http and uploading it to a remote PathInfoService, or to
synthesize the PathInfoMessage to return to the client, if nar-bridge is
acting as a PathInfoService for a remove Nix HTTP Binary cache.
Change-Id: Ibba1ab6238a050816c4fab29cb21ae88877d8613
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9651
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Bumps the go module past cl/9604 and update the consumer side.
Change-Id: Id44245017f1dc2f8aac28051cdbb45b83bdc5be3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9650
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
Much friendlier than either Base64 or raw byte slices.
Change-Id: I9b4cdd57c83ddc76c0be8103da4320207657a72b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9622
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We use checked arithmetic for computing the total size, and verify
that size is in-bounds in Directory::validate.
If an out-of-bounds size makes it to the "unchecked" size method,
we either panic (in debug mode), or silently saturate to u32::MAX.
No new panic sites are added, since overflows in debug mode already
panic at the language level.
Change-Id: I95b8c066a42614fa447f08b4f8fe74e16fbe8bf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9616
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Behave like `cargo test` rather than `cargo test --release`.
Change-Id: Ie013d04ac68d7dec2a3b870fa9f0060a70a9635d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9621
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We don't need the full PathInfo message, only the root node.
Change-Id: I667045ed766875dfbf8ac126a50b02baa2df67a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9604
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
This removes the Export method in nar-bridge, and updates all users to
the version now in storev1pb.
It moves the roundtrip test to the importer crate, and some of the
utility functions into a separate util_test.go file.
Change-Id: I81d9e0b35dfd78ef1042bed307281eecd2aaa2a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9603
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
Export will traverse a given PathInfo structure, and write the contents
in NAR format to the passed Writer.
It uses directoryLookupFn and blobLookupFn to resolve references.
This is being moved over from nar-bridge. We need to keep the code there
around until we can bump go.mod to storev1 with this merged, but the
tests can already be moved entirely.
Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
We can use the helper to rename the node.
Change-Id: Id8defea7e5ebbd43d7b7a9b2992c62084e1828ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9601
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
This is preparation for adding an async port.
Change-Id: Id638ec1f6f46e2f3935448184eed51e2233263fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9618
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Consistent error messages, and slightly nicer code layout. We avoid
printing the input data, since we primarily want to point out the
specific violated invariant. In the one place where we do want to,
we use BStr's Debug implementation, since byte slices don't print
nicely.
Change-Id: I3a9a0c37be270ea5f16cf124922c254608fb849e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9617
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
