The `if let` wasn't matching `outputHashAlgo` being unset, and didn't
populate it in that case.
Port the remaining commented-out testcases over to nix-lang based tests.
Change-Id: I140b5643b9ed9d29f9522ec65d98d0b12262d728
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9825
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Get some test coverage with some calls to builtins.derivation.
The expected output paths were calculated with Nix.
Change-Id: I2ce205ea5244e8ef939d9cacb033283fc6f15d17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9817
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Have a private `eval` function in the test module that returns an
EvaluationResult, and migrate the existing tests over to use it, rather
than repeating itself.
Change-Id: I879987700c8507248c644ef03b62a8cb8e308139
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9816
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
If the output was fixed, we broke out of the for loop too early, before
actually validating individual outputs.
Change-Id: I2259697dfa2a157764358f6d326a1f7f6610647c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9815
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This specific struct is only used to represent content-addressed paths
(in case a Derivation has a fixed-output hash, for example).
Rename `Output`'s `hash_with_mode` to `ca_hash`.
We now also include `CAHash::Text`, and update the `validate` function
of the `Output` struct to reject text hashes there.
This allows cleaning up the various output path calculation functions
inside nix-compat/src/store_path/utils.rs, as they can now match on
the type.
`make_type` is renamed to `make_references_string`,
`build_regular_ca_path` is renamed to `build_ca_path`, and
`build_text_path` has a disclaimer added, because you might not actually
want to use it.
Change-Id: I674d065f2ed5c804012ddfed56e161ac49d23931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9814
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This adds support to represent the `CA` field found in some .narinfo
files. As `deriver`, it's also a subfield of the `narinfo` field.
Extending nix-compat with a more accessible data structure that can
take care of formatting, as well as writing validation functions in Rust
+ Golang, and integrating it into nar-bridge is something for a followup
CL.
Change-Id: I71e9c30957bcd03051a491aa54d7baac25b6dd2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9795
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This information is present in the .narinfo files, it should have gone
there.
Change-Id: Ib43d0cf30c2795bf1fe77c46646174353ade0458
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9794
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
It is *eventually* followed by a Node, but there is some stuff in
between.
Change-Id: Ie7c7b462828bd3e066f4a7e774895f30b82763ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9768
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Equivalent to the existing code, but a little less cryptic.
Change-Id: Ib9b2f9aedddc84d0e79840bba4cce01f92d9bc56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
As seen in cl/9761, we only use put() and close() in some cases.
Make sure this is documented, so we don't end up having implementations
not able to deal with that.
Change-Id: I406ad23f2f4c9604eca731d898ae41a4663fb846
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9786
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
`code.tvl.fyi/tvix/store/protos` now points to a directory that only
contains the `.proto` files, while all golang tooling and .pb.go files
live in tvix/store-go.
As discussed in
https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount
of people currently using this is still small, so rename the go.mod now,
while it doesn't yet hurt.
Also, use code.tvl.fyi/tvix/castore-go instead of code.tvl.fyi/tvix/
castore/protos, to make use of cl/9791.
Change-Id: I9ea89957d7c29dfae4c893b9aae8ac8a0bad2d8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9792
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
`code.tvl.fyi/tvix/castore/protos` now points to a directory that only
contains the `.proto` files, while all golang tooling and .pb.go files
live in tvix/castore-go.
As discussed in
https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount
of people currently using this is still small, so rename the go.mod now,
while it doesn't yet hurt.
Change-Id: Ib3c6a2dac2923b3806ebb05be00af66d0da9f698
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9791
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This keeps things isolated a bit more.
Change-Id: I437f2f63ee1567c0cbc02298514ad4d89f1dce85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9790
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Similar to the castore-go CL before, this also updates the store-go
bindings to the new layout.
Change-Id: Id73d7ad43f7d70171ab021728e303300c5db71f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9788
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Have `tvix/castore/protos` only contain the protos, no go noise.
Make the `.pb.go` file generation a pure Nix build
at `//tvix/castore/protos:go-bindings`, and have a script at
`//tvix:castore-go-generate` (TBD) that copies the results to
`tvix/castore-go`.
`//tvix:castore-go`, with sources in `tvix/castore-go` now contains the
tooling around the generated bindings, and the generated bindings
themselves (So go mod replace workflows still work).
An additional CI step is added from there to ensure idempotenty of
the .pb.go files.
The code.tvl.fyi webserver config is updated to the new source code
path. I'm still unsure if we want to also update the go.mod name. While
being a backwards-incompatible change, it'll probbaly make it easier
where to find these files, and the amount of external consumers is still
low enough.
Part of b/323.
Change-Id: I2edadd118c22ec08e57c693f6cc2ef3261c62489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
After finishing the ingestion, the directory putter was not being
closed. This caused a race where the root directory node was accessed
before the directory node had been flushed to the server.
This patch makes it so we close the putter before returning the root
node which should ensure that the root node exists on the directory
service server before the `ingest_path` function returns.
Fixes b/326
Change-Id: Id16cf46bc48962121dde76d3c9c23a845d87d0f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9761
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
That's needed to build nar-bridge, as well as things in tvix/*store/
protos.
Change-Id: If5f71b6c528a27689faa3c5ab3489cc528f9e6e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9759
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We need to invoke protoc at build time (only).
Change-Id: Ib4c101d2ccdbdbb078725fc98c30d6c00086667d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9754
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
On Darwin, some crates producing binaries need to be able to link
against security.
Change-Id: I5bdd69247c12729b9efd5c4f18527d361ef99e87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9758
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Also, make some of them buildInputs, not nativeBuildInputs.
We likely need to link against libiconv, so it shouldn't be a native
build input.
Change-Id: I757bc7f5d8c864ecaa3db26988d00b496c085c8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9757
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
It looks like we need Security Framework in the shell to be able to
imperatively run `cargo build`.
Change-Id: Ia5df9052bafca3610d4ed235b180b812d33d0d15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9756
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
The previous CLs did already absorb all the logic into the common tests,
no need to write this here again.
Change-Id: I7ba84ba86d5445ed247e5d11d5e59b7fa815670e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9732
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Due to the lack of a ATerm parser, we were previously loading the JSON
fixtures to construct our Derivation structs to run the output path
calculations with.
However, as we now have a ATerm parser, we can load the ATerm
representation directly.
This also means we can test the output path calculation for non-UTF8
Derivations.
Change-Id: I0e53f41a23566b5ad5f0fed12724e02a10b02707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9731
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This provides a nom-based parser for Nix derivations in ATerm format,
which can be reached via `Derivation::from_aterm_bytes`.
Some of the lower-level ATerm primitives are moved into a (new) aterm
module, and some more higher-level ones that construct derivation-
specific types.
Also, move the escape_bytes function into there, this is a generic ATerm
thing.
Change-Id: I2b03b8a1461c7ea2fcb8640c2fc3d1fa3ea719fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9730
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We don't actually care if it's a BTreeMap of strings, bstrings or any of
that sort.
The only thing we want to be able to do is get a reference to the bytes
from the keys and values.
Change-Id: I21b85811a9ea47fe06afa3108836ef9295e5d89b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9737
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
We use test_resources and globbing for some of the test cases, so adding
additional files in there will also create new test cases, which we
don't always want.
Move it down one level to make some more space.
Change-Id: I619867dc80a4ced59d45096d0703678663b559cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9729
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Limit the amount of memory consumed on the stack for NixHash. Sha512
isn't used that often, so it's fine if we heap-allocate it.
Change-Id: I4a9eecd20c6184610124dc130c41bfa5d0dc04c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9726
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tonic support was added to tokio-listener upstream which removes the
need for use to have tonic compatibility wrapper types around it.
See: https://github.com/vi/tokio-listener/pull/2
Fixes b/311
Change-Id: I04a2dbb3bc3c8bfe9339583c0b46070c7ec97811
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9721
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is of known length.
Change-Id: Iba48ccc486d5bf9e38ec1a2da6e7b80997d2c6ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9723
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This updates the code to make use of the new Validate() function defined
on a Node.
Change-Id: I9b6ed694661f41e700f19cc78d53d2224b61852d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9718
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This uses the newly introduced StorePath message type to add a Deriver
field to the PathInfo message.
Support for validation is added to both the golang and rust
implementation. This includes extending unit tests.
Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This encodes a store path a bit more concise, which is used in the
Deriver field.
Change-Id: Ibfb54d3b206917e51970d1d5fe94fcedb901704b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9646
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
To make this easier, move it outside of TvixStoreFs, and accept the
DirectoryService as a function argument, so we don't need to worry about
the lifetime of self.
This also aligns with how we spawn async tasks inside the rest of
TvixStoreFs.
Change-Id: I3b95072209d32039f05aed122240f2d6db7ad172
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9713
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This moves the repetitive code to parse a URL and create a channel
connected to it into `tvix_castore::channel::from_url`.
Part of b/308
Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Move the channel creation depending on the string-based URL into its
own block.
Change-Id: I546b769acd2296b548eb966b62c495f910266df5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We had to have these all while the traits where sync, and there was a
lot of spawning and moving.
Most of this can now be removed in favor of some inline `.clone()`.
Change-Id: Id5466c32a403100bc3347866b3172e06a792e311
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Similar to cl/9715, this makes the validation checks more granular,
introducing a Validate on all *Node.
A check for symlink targets is added too.
Once merged, it can also be used from tvix/store/protos.
Change-Id: I0909a89fadcd74b74ef0c9a8a1f22658fccc83b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9716
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We can just check the digest length to be correct, and move the symlink
target checks to a single line.
Change-Id: I41d2e3a50e7990ef6c04f02acd754b1e17b43e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9717
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Implement `validate()` on `node::Node`, and call it from PathInfo's
validate() too. Node-related errors are moved to a ValidateNodeError
error type.
This additionally adds some more validations for symlink targets (they
must not be empty, and not contain null bytes).
Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
This teaches tonic to speak https, so `grpc+https` becomes a working
connection string.
Change-Id: I9c550ea665b829725b545a164cf6f78587bd5213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9714
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
Bump code.tvl.fyi/tvix/store/protos past cl/9649, where Validate()
already ensures the NarSha256 has the correct size.
Change-Id: I774668822f4d9dbd4dea47dde6e4745dc95e8e7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9665
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
As correctly mentioned in
https://cl.tvl.fyi/c/depot/+/9652/comment/03b9b96e_bbb337fd/,
we shouldn't be using these magic constants, but pull them from where
they're defined.
This already is a dependency of go-nix, and pkg/pathinfosvc/server.go,
so no changes in go.mod.
Change-Id: I0cc41ce040fcbddf4b6171417bc9b0de55af4991
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9653
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Explain a bit better what we do in the codeblock below,
and add a newline to make it clear we don't just document the first
branch.
Change-Id: Ifb142d9984e6d2cbca648525c10298f2fcbdfedf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9645
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
We have nixhash.FromHashTypeAndDigest now.
Also, run Validate() on the PathInfo received from the remote
PathInfoService.
Change-Id: I14db0d9356c539c084afc9dd712314b56da2587e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9652
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
… and nar size / sha256 digest.
Instead of producing sparse PathInfo messages when NARs are sent to
nar-bridge, the nar-bridge http server now keeps a lookup table
(narsha256) -> (rootNode, narSize)
This removes a whole bunch of noise, because we don't need to keep
sparse fields around.
A convenience function
`GenPathInfo(rootNode *castorev1pb.Node, narInfo *narinfo.NarInfo)` is
added, which is used to produce PathInfo messages, either when receiving
a NAR file over http and uploading it to a remote PathInfoService, or to
synthesize the PathInfoMessage to return to the client, if nar-bridge is
acting as a PathInfoService for a remove Nix HTTP Binary cache.
Change-Id: Ibba1ab6238a050816c4fab29cb21ae88877d8613
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9651
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Bumps the go module past cl/9604 and update the consumer side.
Change-Id: Id44245017f1dc2f8aac28051cdbb45b83bdc5be3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9650
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
Much friendlier than either Base64 or raw byte slices.
Change-Id: I9b4cdd57c83ddc76c0be8103da4320207657a72b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9622
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We use checked arithmetic for computing the total size, and verify
that size is in-bounds in Directory::validate.
If an out-of-bounds size makes it to the "unchecked" size method,
we either panic (in debug mode), or silently saturate to u32::MAX.
No new panic sites are added, since overflows in debug mode already
panic at the language level.
Change-Id: I95b8c066a42614fa447f08b4f8fe74e16fbe8bf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9616
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Behave like `cargo test` rather than `cargo test --release`.
Change-Id: Ie013d04ac68d7dec2a3b870fa9f0060a70a9635d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9621
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We don't need the full PathInfo message, only the root node.
Change-Id: I667045ed766875dfbf8ac126a50b02baa2df67a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9604
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
This removes the Export method in nar-bridge, and updates all users to
the version now in storev1pb.
It moves the roundtrip test to the importer crate, and some of the
utility functions into a separate util_test.go file.
Change-Id: I81d9e0b35dfd78ef1042bed307281eecd2aaa2a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9603
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
Export will traverse a given PathInfo structure, and write the contents
in NAR format to the passed Writer.
It uses directoryLookupFn and blobLookupFn to resolve references.
This is being moved over from nar-bridge. We need to keep the code there
around until we can bump go.mod to storev1 with this merged, but the
tests can already be moved entirely.
Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
We can use the helper to rename the node.
Change-Id: Id8defea7e5ebbd43d7b7a9b2992c62084e1828ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9601
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
This is preparation for adding an async port.
Change-Id: Id638ec1f6f46e2f3935448184eed51e2233263fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9618
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Consistent error messages, and slightly nicer code layout. We avoid
printing the input data, since we primarily want to point out the
specific violated invariant. In the one place where we do want to,
we use BStr's Debug implementation, since byte slices don't print
nicely.
Change-Id: I3a9a0c37be270ea5f16cf124922c254608fb849e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9617
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
Found by Clippy, which we should probably run in CI.
Change-Id: Id79c30b63f681021ab79358e02d29454d43c0aa6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9614
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This takes a castorev1pb.Node and updates its name, making sure to copy
it.
Change-Id: I57bf2b2b4a74496a1a697bcbce72bf67a12e69fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9594
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Convenience function, moves all code converting from a PathInfo struct
to to go-nix's NarInfo.
Change-Id: Idf0dcc38675674563f2dfd3286a4a55fa2a24a82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9593
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
We already have validation tests for Rust, let's add the missing ones
for golang too.
Change-Id: Iaf3a3e1ee72d5647da3f2aa977d6e0d0379b2ce5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9595
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This was unfortunately using ./castore/protos all the time, so not
testing castore at all.
Update src to make sure it does get built.
Change-Id: I563910c259ffca6aee7f964532bd39c4bcf7aed8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9596
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
The point of clearing and reusing the same Vec is to avoid transiently
allocating for every directory entry. This was lost in cl/8974 when we
switched from String to Vec<u8>.
Change-Id: I65647e5c4e54e88f1fe45e9a752cb5154d98fb33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9607
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Rather than using this loop, use exponential backoff while waiting for
the socket path to be created.
Change-Id: I3056b1525784cd712b1d81f84876c9ca0be10dc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9569
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Rather than using this loop, use exponential backoff while waiting for
the socket path to be created.
Change-Id: I18706a64ce06f8916a07892dfbcd409ac5b3bff1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9568
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We don't need to spawn two tokio runtimes anymore, and can do the URL
parsing at once, too.
Change-Id: I7885a894bb1250cd087d4e1893e3e73b631331da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9567
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We don't need to spawn two tokio runtimes anymore, and can do the URL
parsing at once, too.
Change-Id: I38ab96978cb7f8c31ded2726262e0b1366655094
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9566
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
tempdir pulls in remove_dir_all 0.5.3 with
https://rustsec.org/advisories/RUSTSEC-2023-0018.html, and we use
tempfile everywhere else too, so let's just migrate to that.
Change-Id: I735ade7b65e12fc26e3d43ca95fcfa07fcc64642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9565
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We already use store_path::build_nar_based_store_path as a helper
function down there.
Change-Id: I46842bb0a2f4cd5de7dcfc3c4b6aae399618667a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9571
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This bumps all these dependencies to their more recent versions.
We have to enable the `macros` tonic feature, and looks like we should
also enable `rt`, not only `rt-multi-thread`.
Change-Id: Icc3600848fca0bacce24f4889d088e75711594ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8984
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Similar to gen_directorysvc_grpc_client, introduce a
gen_blobsvc_grpc_client function that provides a gRPC client connected
to a blobservice.
The test is update to use that client to test against, rather than the
server trait, removing the last usage of tonic_mock, so it's removed
as well.
Fixes b/243.
Change-Id: If746e8600588da247eb53a63b70fe72f139e9e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9564
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
This probably was about passing around directory_putter at some point,
which we do, so whatever this meant, it's not actionable anymore.
Change-Id: I1b4e0cdd2119bf2b2a9cf06d186a3b476b0ff367
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9573
Reviewed-by: Linus Heckemann <git@sphalerite.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We can use DuplexStream to create to bidirectional pairs, which avoids
manually waiting for unix sockets to pop up and connect, and creating
temporary directoires to create the unix sockets in.
Turns out, we also don't actually need to spawn the server in a separate
runtime, it works just fine these days. This might be due to all the
sync barriers in between being gone.
Change-Id: I6b79823bc6209cbcb343b7a498c64a2ba6e0aee7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9562
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
For Unix and TCP streams, we can actually give more info than just ().
Change-Id: I80ca261b9e7a880389ec2029ea5893180314f0b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9561
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
The tests were using the gRPC Service/Server interface, not the gRPC
clients.
Use our new gen_directorysvc_grpc_client() function to spin up a server
in a separate thread, and test via that client, rather than *just* the
server implementation.
This is more correct, because we're now really exercising the gRPC
stack, and some of the stream implementations are also only implemented
on the client.
Change-Id: I99434a2399856a44c9e6988d0b720f7a429d0ad1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9560
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Move this code into a helper function, which we'll use in other places
in a bit.
Change-Id: Icae6f6dd2d4b2fa86fd2b836ddd7a4ca0e0354e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9559
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
So far, we provided a custom `default.nix` in the root of the tvix josh
workspace, which re-defined the shell attribute from `tvix/default.nix`.
Some of the recent fixes, e.g. the MacOS-specific additions to the list
of dependencies however didn't get ported over to this file, and in
general, it's quite annoying to have two different places for these
things.
Initially I explored the idea of moving this default.nix file to a
default-depot.nix file in the josh worktree only, and then "polyfill"
some of the dependencies, or set up readTree in the josh workspace too,
but it turned out to pull in too many dependencies to be worth
the effort (nix.sparseTree, tools.depotfmt, crate2nix overlay,
third_party.gitignoreSource).
I now took a different approach - moving the definition of the `shell`
attribute from `tvix/default.nix` to its own `shell.nix` file, which is
imported from `tvix/default.nix` in regular depot usecases.
Josh workspace consumers only see the `shell.nix`, which can be used
in a self-contained fashion, the other `default.nix` is gone entirely,
and we update the workspace file to also not show `tvix/default.nix` at
the root either, so running `nix-shell` and then `cargo build` should
still work.
Change-Id: I6cb54d45d150c597612530ba44bc578f9d7f9120
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9556
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
vhost-user-backend doesn't support macOS yet, so the virtiofs features
will not work on macOS. This removes it as a default feature which makes
`cargo build` work out of the box on macOS.
The `virtiofs` feature is enabled for Linux when building via Nix, but
if being built by cargo directly, the feature must be enabled via a
cargo flag.
Change-Id: I2aaca9582f8e3dbcf9ee5f1b9831d614909f3799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9555
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
This should make it quite quick to spot writing code breaking some of
the assumptions we have on PathInfo messages ourselves.
Change-Id: I480caaec41f8ea5246c3c3081460c7ad12e78569
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9554
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This saves us writing the name parsing code three times. We can also
delay parsing until we did other (cheaper) checks.
Change-Id: I1abe3f20dba4215b38839cf7466297e028d64656
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9548
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
We run narInfo.Check to ensure this parses to a StorePath, not
nixpath.Check.
Change-Id: Id91183128df74a60d98fa2a31174cd879194c34d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9550
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is very similar to the Rust counterpart.
Change-Id: I40d51aaac3fcf7f52e5896587e561bc2377f6269
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9549
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This check makes more sense there, and gives stronger semantics - Done()
only succeeds if the other side successfully received everything, *and*
came up with the same hashes as we did.
Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This adds an additional nar-bridge-pathinfo command.
It exposes a PathInfoService for a HTTP Binary Cache, ingesting data
into a BlobService/DirectoryService as it goes through the NAR file.
It does this whenever it receives a Get request for a specific output
path, and waits returning with the PathInfo response until it ingested
the data.
It does not do any sort of caching - this means it re-downloads NAR
files again whenever the PathInfo is requested again, so you most likely
do not want to use this currently.
It's one building component as soon as we have store composition (which
we currently don't, so don't use this).
It can be used as an alternative mechanism to ingest data (Blobs and
Directories) of a given store path from a binary cache into tvix-store.
```
❯ nix-build -A third_party.nixpkgs.hello
/nix/store/mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1
❯ nix hash to-sri --type sha1 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6
sha1-Rs/INeK+7IGbG/u7fHoVNm96Yqs=
❯ out=$(mg build //tvix/nar-bridge)
$out/bin/nar-bridge-pathinfo --log-level debug &
INFO[0000] Starting nar-bridge-pathinfosvc at [::]:8001
❯ mg run //tvix:store -- daemon &
[mg] building target //tvix:store
[mg] running target //tvix:store
2023-10-03T16:21:57.433739Z INFO tvix_store: tvix-store listening on [::]:8000
at src/bin/tvix-store.rs:229
❯ evans --host localhost --port 8001 -r repl
[…]
tvix.store.v1.PathInfoService@localhost:8001> call Get
✔ by_output_hash
by_output_hash (TYPE_BYTES) => Rs/INeK+7IGbG/u7fHoVNm96Yqs=
{
"narinfo": {
"narSha256": "sXrPtjqhSoc2u0YfM1HVZThknkSYuRuHdtKCB6wkDFo=",
"narSize": "226552",
"referenceNames": [
"aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8",
"mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1"
],
"signatures": [
{
"data": "7guDbfaF2Q29HY0c5axhtuacfxN6uxuEqeUfncDiSvMSAWvfHVMppB89ILqV8FE58pEQ04tSbMnRhR3FGPV0AA==",
"name": "cache.nixos.org-1"
}
]
},
"node": {
"directory": {
"digest": "xvo6BYbYaDw76IibLu5sr+VZoj9iM0ET2RUuYSYLwKE=",
"name": "bWRpN2x2cm4ybXg3cmZ6djNmZHEzdjV5dzhzd2lrczYtaGVsbG8tMi4xMi4x",
"size": 141
}
},
"references": [
"ptgFMIhdl2nJxMDdlDkITyXuBFc=",
"Rs/INeK+7IGbG/u7fHoVNm96Yqs="
]
}
```
Change-Id: I50167d0ac081c91adf5cf2733bbc4dc0993bd46e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9539
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Brian Olsen <me@griff.name>
This validates the size of reference digests in the PathInfo message,
as well as inside the narinfo submessage. If narinfo is set, they need
to parse to StorePath, and have the same digest there as in the PathInfo
message.
`proto::tests::pathinfo::validate_references` needed to be updated,
because we actually did not populate the proper references before.
Change-Id: I9545b2487aab9fe0d229c26aceba5ddc5e6daafd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9545
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
There's other digests in the PathInfo structure, that also might have
wrong digest lengths. Rename this to give some room for them, and update
the error message a bit as we go.
Change-Id: I06562664721156e658f2ed14ba1de907377d284b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9543
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Rename the nar-bridge CLI to nar-bridge-http, because it's the one
spinning up an http server.
Change-Id: I0fb75c50e4299272a128dd5ecaa4be8f06fa3dbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9538
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Use a genNarHandler() function accepting a boolean to construct the
HTTP handler.
Change-Id: I17c054826d91a9dbed8b1f53945a51f27fa60ace
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9537
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
This is only dealing with the HTTP interface.
Change-Id: I011b624fd9f11ea96231b92fea1166c118a219f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We can drop most of Hasher if we use a MultiWriter writing to the hash
function and a minimal CountingWriter.
This should make things a bit more understandable.
Change-Id: I37ee72d9a5c73f253aecc1ad761cb723389b89fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9529
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This aligns behaviour more with how it should be - it's the
responsibility of the callback functions to return digests of the things
they consume(d). It allows further cleaning up the hasher struct.
Change-Id: I9cbfc87e6abd4ff17fadf39eb6563ec3cb7fcc6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9528
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Make the import function usable on any reader.
Change-Id: I84d2004cb73cdd7a11fe8efb0f2efb6335d5e6b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9527
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This is only called once.
Change-Id: I342443b8d04050929733fc84d5f36cd64060afe3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9525
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Nix has historically rejected these. The current behaviour was
accidentally introduced in Nix 2.4, and is considered a bug.
Link: https://github.com/NixOS/nix/pull/9095
Change-Id: I38ffa911f0a413086479bd972de09671dbe85121
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9507
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
This partially fixes b/312 and gets FUSE to work again on MacOS.
It is mostly small type changes and an update to fuse-backend-rs because
upstream currently doesn't work with MacFuse. It also sets the default
FUSE thread count on MacOS to 1 because otherwise the mount command will
hang when shutting down as only one thread gets ENODEV and all the others
just keep blocking.
Change-Id: Ifb3c4268caf296c487049c1dc4618acb32497f44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9490
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
The unmount method in FuseDaemon calls join on a bunch of threads and that is
a blocking call but it is called from an async context in the tvix-store
binary.
This change wraps the call to unmount in a spawn_blocking.
Change-Id: If89183b4a3f890874e75f5faf90cd24cb18da1e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9489
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This used to be in tvix/test but was moved to tvix/boot, but not all the
command were updated so they no longer work as documented.
This updates the commands to use the new tvix/boot directory.
Change-Id: I8658d928a152140df182c159bf98ea9d9ecd1302
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9482
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This call was renamed to `Read` a while ago, but I forgot to update
the README.
Change-Id: Ibcb97fe2520a5e9bb8444ec1adade6890dda4a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9476
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This commit adds a test case for b/281.
Change-Id: I8dfbfc0ff636184d7882530d8aefb329a3af9e5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9288
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Curently both fuse and virtiofs features fail to build on Darwin.
Disable them for now until fixed.
Change-Id: Ic99fa5e3ebec8a7072ba884101a5c18652ed409c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9473
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
- rename the feature to `tonic-reflection` in both tvix-store and tvix-
castore.
- set it to disabled in tvix-castore by default
- enable it in tvix-store, and pull in tvix-castore with the feature
enabled.
Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
These enable fuse and virtiofs features, which fail to build on MacOS.
tvix-cli needs neither of these to be built, so we can disable them.
Fixes b/240 (tvix-cli build on Darwin)
Change-Id: I991d947f31d0185aedd1c8a341f714f4eedd03c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9474
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This will bring conditional features support.
Also invoke crate2nixgenerate with the --all-features argument, so all
dependencies, including the ones for optional features are included in
the Cargo.nix file.
Change-Id: I3bbcb200c9b481f660db89efba650ea4f7418a63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9470
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
It's in no release yet, but this is the current rev in master.
Change-Id: I9e4bc2fde26e85c4182bd66b3e566c2d957072ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9469
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This adds support for listening on UNIX domain sockets via the
tokio-listener crate. The crate will automatically determine whether to
start a TCP or UNIX domain socket server based on the listen address.
Unfortunately, it's not compatible with tonic right out of the box so I
added some wrapper types to implement the necessary traits to make
things work. We should investigate upstreaming a `tonic` option to the
tokio-listener crate which implements the relevant `tonic` traits.
Example:
```
$ tvix-store daemon -l /run/tvix-store.sock
INFO tvix_store: tvix-store listening on /run/tvix-store.sock
$ tvix-store mount -l /mnt/tvix --blob-service-addr grpc+unix:///run/tvix-store.sock --directory-service-addr grpc+unix:///run/tvix-store.sock --path-info-service-addr grpc+unix:///run/tvix-store.sock
$ ls /mnt/tvix
```
Change-Id: I91c4a4b0c5a177b3b90e6c01a4e5d263130e6bdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9429
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This is mostly boot tooling, the integration test is just one instance
making use of it.
Expose initrd, kernel and runVM as a separate target to CI, and move the
tests to a subdirectory.
Change-Id: I1d22cd68bf5af095bc11dd9d7117b62956c7f7f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9465
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We were previously not running as PID1, but were executing things via
uinitcmd (by u-root init).
This had some problems - a real system pid1 really wants to be pid1.
u-root init itself seems to be too complicated to configure as a
chainloader, so instead, we just provide init/pid1 ourselves and do the
little bit of mount setup / stage 1 that's necessary.
`tvix.exec` is renamed to `tvix.run`, to further distinguish it from the
`exec` bash function.
Our init will now also power off the machine if the shell, or the custom
tvix.run command finished. In the case of switch_root (`init=` being
passed), we terminate before, so it's up to the real system anyways.
Change-Id: If00c0c11ca9393968e76ae0d131594b5c135fa48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9454
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This commit makes catchable errors a variant of Value.
The main downside of this approach is that we lose the ability to
use Rust's `?` syntax for propagating catchable errors.
Change-Id: Ibe89438d8a70dcec29e016df692b5bf88a5cad13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9289
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
This commit creates a separate enum for "catchable" errors (the kind
that `builtins.tryEval` can detect).
Change-Id: Ie81d1112526d852255d9842f67045f88eab192af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9287
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
These existed with slightly similar descriptions in both places, and
castore and nar-bridge and castore were missing entirely.
Change-Id: I4794c18665e25ee1f812975b526ff27ce197d0af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9453
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
The README suggests putting state here for some examples, so let's make
sure it doesn't cause a dirty worktree.
Change-Id: I3a191f5b2681ef6937a03d3a9686ae0d7aebd76e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9450
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This adds a basic skeleton to test various tvix components in
combination.
As a first start, we test the virtiofs integration, by seeding the tvix-
store with some data, booting a VM, and listing the contents from there.
Change-Id: Ice6516d0eb508d19ea8fabefee860a0950046171
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9449
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Defaulting to where `tvix-store daemon` listens to is a sane choice.
Change-Id: Ic52fa856c5708e0e1a8d51618f8c9ad1894fd28f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9452
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
nar_bridge is an odd name for the binary.
Change-Id: I761ec5f986bde2f7e50e5a0c0b6182164a6cdc7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9451
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
This adds a virtiofs daemon implementation which hooks into the existing
tvix-store filesystem implementation that is used for FUSE.
This allows adding the filesystem to a microvm without having to set up
FUSE inside the guest.
Change-Id: If80c36c9657f2289853e8d9a364bf4f1f7b7559c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9344
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
As shown in the previous CLs, we can very well have store paths starting
with periods, but we can't have derivations with an empty name:
```
nix-build -E 'derivation { name = ""; builder = "/bin/sh"; system = "x86_64-linux"; }'
error: store path 'nr7i5pf18hw2zg487vkdyrbasdqylfcj-' has an empty name
```
I'm currently using ErrorKind::Abort here, because we don't have a
Derivation- related error in tvix-eval (and probably don't want to).
Change-Id: I0e9743cee98dbfa69e9caa2a58352176270f15bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9448
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Suggested in https://cl.tvl.fyi/c/depot/+/9108/5, but this disallows
adding a .gitignore file to the store.
Remove the check, and add a testcase.
Change-Id: Ieb78c417934756b2dbeb493040fe79726d1b9079
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9447
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Setting up the evaluator is a bit annoying currently, might get easier
with b/262, but it's better than no tests on that granularity at all.
Change-Id: Ie8c61466768f37f4efbc19ad497d37f87ddc2044
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9446
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Similar to cl/9350. Most of the times, this is still a regular string,
so let's print it like that if we can, and resort to base64 encoding if
we can't.
We were also wrongly always outputting the second character in the
string.
Change-Id: Id0e2a9d9f1ad3d2d7b554893ecd89a7e6383e9c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9445
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This couldn't be enabled before until go get was able to fetch the
modules again.
Change-Id: I82db7d22e497661ea2a0c8a45c0c865fcc0a1f18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9406
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We don't use this here, except in a test, where we don't really use
it either.
Change-Id: Ia6c45fccf663fe328942b1e1a2cd1c3a1b7f9ae5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9376
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.
Notable changes:
- `fixtures` and `utils` had to be moved out of the `test` cfg, so they
can be imported from tvix-store.
- Some ad-hoc fixtures in the test were moved to proper fixtures in the
same step.
- The protos are now created by a (more static) recipe in the protos/
directory.
The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)
Bug: https://b.tvl.fyi/issues/301
Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
With the move of this code out into castore it has become apparent this
is a general descent inside the castore. Concerns like making sure the
whole Directory closure has been fetched/is fetched initially is nothing
this layer needs to worry about. We can handle this during substitution
of a new PathInfo, once there's store composition.
Closes b/270.
Change-Id: I661ed08e54bc81478e032cfb9abeb23e5b337fbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9373
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This helps spotting unused crate dependencies.
It's preferred over cargo-udeps for the reasons described in
https://blog.benj.me/2022/04/27/cargo-machete/
Change-Id: Ie801004485858741f2fa5ae6f33bd0ddfc292cd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9374
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This is not a simple get and put returning digest.
You currently can get by output hash, and a put gives you back the
(possibly modified) PathInfo message.
Change-Id: I43174d87a764417b8620c4d5bdac058cc25b0373
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9371
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
There's some more left, but they've been renamed/refactored out of
sight.
Change-Id: I41579dedc74342b4c5f8cb39d2995b5b0c90b0f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9372
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
We've decided to asyncify all of the services to reduce some of the
pains going back and for between sync<->async. The end goal will be for
all the tvix-store internals to be async and then expose a sync
interface for things like tvix eval io.
Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In prepration for adding virtiofs support, I thought it would make sense
to split out the filesystem implementation from FUSE itself.
The `fs` module holds the tvix-store filesystem implemetation and the
`fuse` module holds the code to spawn a FUSE daemon backed by multiple
threads.
Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
This switches the FUSE implementation from fuser to fuse-backend-rs.
fuse-backend-rs is designed to work with both FUSE and virtiofs.
Virtiofs support will make it possible to plug the tvix-store into a
microvm and have `/nix/store` access without having to setup FUSE inside
the guest.
Additionally fuse-backend-rs has nice support for running multiple FUSE
threads and has some async support.
The goal of this commit is to mechanically switch over to
fuse-backend-rs with minimal changes. I did have to add some locks here
and there because fuse-backend-rs uses `&self` on all methods whereas
fuser uses `&mut self`. `&self` is required for concurrent access to the
FUSE server, so this makes sense.
We can consider switching to concurrent maps and use some other
techniques to reduce lock contention and critical section size.
Issue: https://b.tvl.fyi/issues/305
Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
More aligned with how it's called in other places
Change-Id: I759ac7ca3b5b69c1101d2d51a569d76c183a6330
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9362
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Create a pipe, pass the read end, and have a goroutine write to the
write end.
Change-Id: I301c273355705e60113b018e7e84b76972200e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9361
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Only keep the `serve` subcommand, and make it appear at the root.
Introduce a --log-level argument, and be a bit less noisy in normal
operation.
Change-Id: I86b8abde1869a5c0c947508bcc29f845222aac09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9360
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
`nix copy` checks if NARs and NARInfo files are present, before
uploading them. That's not an error, but normal behaviour, so no need to
log with level info for these cases.
We only want to log if the error is not a 404, and log with Warn level.
Change-Id: I762de3b862d070a0f18bc62e324e94ca5c7c3693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9359
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Let's make sure we don't end up blocking a client too much when
inserting very small blobs.
Change-Id: I640dda92efae538c70d32a40e6e85a23e9749e20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9358
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Instead of creating one big BlobChunk containing all data, and creating
way too large proto messages, chunk blobs up to a reasonable (1MiB)
chunk size, and send them to the server like that.
Change-Id: Ia45a53956a6d7c0599cc59ac516ba37e9fb1b30e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9357
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Include the changes from cl/9351
Change-Id: Ie60c9dddcafaeee190439fa19fa7704917600fdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9363
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This wasn't removed yet, and no code is using/populating it so far.
It's confusing, let's update it to the state of things now, and re-
introduce it once we get there.
Change-Id: I68f5ba17a8eee604d8ccd82749da7c8be094cb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9351
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We were blindly returning 0o444 for all regular files, but regular files
with executable bit need to be 0o555.
This wasn't spotted because stat'ing executable files was not part of
the test suite, it's now added.
Change-Id: I04c69784053e7e43d838c01bb288f2df48f40b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9345
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We previously kept the trait of a BlobService sync.
This however had some annoying consequences:
- It became more and more complicated to track when we're in a context
with an async runtime in the context or not, producing bugs like
https://b.tvl.fyi/issues/304
- The sync trait shielded away async clients from async worloads,
requiring manual block_on code inside the gRPC client code, and
spawn_blocking calls in consumers of the trait, even if they were
async (like the gRPC server)
- We had to write our own custom glue code (SyncReadIntoAsyncRead)
to convert a sync io::Read into a tokio::io::AsyncRead, which already
existed in tokio internally, but upstream ia hesitant to expose.
This now makes the BlobService trait async (via the async_trait macro,
like we already do in various gRPC parts), and replaces the sync readers
and writers with their async counterparts.
Tests interacting with a BlobService now need to have an async runtime
available, the easiest way for this is to mark the test functions
with the tokio::test macro, allowing us to directly .await in the test
function.
In places where we don't have an async runtime available from context
(like tvix-cli), we can pass one down explicitly.
Now that we don't provide a sync interface anymore, the (sync) FUSE
library now holds a pointer to a tokio runtime handle, and needs to at
least have 2 threads available when talking to a blob service (which is
why some of the tests now use the multi_thread flavor).
The FUSE tests got a bit more verbose, as we couldn't use the
setup_and_mount function accepting a callback anymore. We can hopefully
move some of the test fixture setup to rstest in the future to make this
less repetitive.
Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This change got lost in the rebases in cl/9348. There's unnecessary
`break`/`continues` that can be replaced by moving the conditional into
the for loop condition.
Change-Id: I559e21087630b05e483f768ab59f8067961a2eae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9352
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Previously, nar-bridge, had a couple of bugs with tracking the current
directory when traversing a NAR file.
The included test case looks like:
```
/ (dir)
/test (dir)
/test/tested (file)
/tested (file)
```
Previously, we would do a string prefix match between the current node
and the top of the directory stack to determine if the node is in the
directory. In this case `/test` is a substring of `/tested`; however,
`/tested` is not in the `/test` directory. The fix is to append a `/` to
the directory name when doing the prefix match, so `/test/` is not a
prefix of `/tested`.
Additionally, when popping the stack, we need to continuously pop the
stack until the new node is in the directory at the top of the stack
(stopping before we pop the root directory)
Example:
```
/ (dir)
/a (dir)
/a/b (dir)
/a/b/c (file)
/z (file)
```
Previously, `z` would end up in directory `/a` because we only the pop
the stack once.
The included test case requires both of these issues to be fixed for it
to pass, so I think it is sufficient.
Change-Id: I22f601babf04d39d85535ba7ad585d3970757211
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9348
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
In some cases, Nix is not able to stream the NAR file fast enough. Bump the
timeouts for now. We might want to get a better understanding in what's
happening here long-term, and/or make the timeouts configurable.
Change-Id: Ieaa9c8f04bc73c6ce0679a058d07eaf87126634e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9340
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This provides a Nix HTTP Binary Cache interface in front of a tvix-store
that's reachable via gRPC.
TODOs:
- remove import command, move serve up to toplevel. We have nix-copy-
closure and tvix-store commands.
- loop into CI. We should be able to fetch the protos as a third-party
dependency.
- Check if we can test nar-bridge slightly easier in an integration
test.
- Ensure we support connecting to unix sockets and grpc+http at least,
using the same syntax as tvix-store.
- Don't buffer the entire blob when rendering NAR
Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Co-Authored-By: Vo Minh Thu <noteed@gmail.com>
Change-Id: I6064474e49dfe78cea67676957462d9f28658d4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9339
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This shuts off the following warning:
```
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
```
Stuff still seems to work with it, so no need to stick with "1".
See https://doc.rust-lang.org/edition-guide/rust-2021/default-cargo-resolver.html
for more details.
Change-Id: I6056d95cd5cb793f37ef843ed43009a27ad36367
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9342
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
WrongSorting(Vec<u8>) actually encodes the name of a node, so if we can,
we want this to display it as a string (and fall back to the base64
encoding if we can't).
Before:
> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: [108, 111, 99, 97, 108, 101] is not sorted
After:
> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: locale is not sorted
Change-Id: I68420c53a89cb1aa96e4bdce414366cebcb7915f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9350
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
