After finishing the ingestion, the directory putter was not being
closed. This caused a race where the root directory node was accessed
before the directory node had been flushed to the server.
This patch makes it so we close the putter before returning the root
node which should ensure that the root node exists on the directory
service server before the `ingest_path` function returns.
Fixes b/326
Change-Id: Id16cf46bc48962121dde76d3c9c23a845d87d0f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9761
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
That's needed to build nar-bridge, as well as things in tvix/*store/
protos.
Change-Id: If5f71b6c528a27689faa3c5ab3489cc528f9e6e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9759
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We need to invoke protoc at build time (only).
Change-Id: Ib4c101d2ccdbdbb078725fc98c30d6c00086667d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9754
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
On Darwin, some crates producing binaries need to be able to link
against security.
Change-Id: I5bdd69247c12729b9efd5c4f18527d361ef99e87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9758
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Also, make some of them buildInputs, not nativeBuildInputs.
We likely need to link against libiconv, so it shouldn't be a native
build input.
Change-Id: I757bc7f5d8c864ecaa3db26988d00b496c085c8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9757
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
It looks like we need Security Framework in the shell to be able to
imperatively run `cargo build`.
Change-Id: Ia5df9052bafca3610d4ed235b180b812d33d0d15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9756
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
The previous CLs did already absorb all the logic into the common tests,
no need to write this here again.
Change-Id: I7ba84ba86d5445ed247e5d11d5e59b7fa815670e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9732
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Due to the lack of a ATerm parser, we were previously loading the JSON
fixtures to construct our Derivation structs to run the output path
calculations with.
However, as we now have a ATerm parser, we can load the ATerm
representation directly.
This also means we can test the output path calculation for non-UTF8
Derivations.
Change-Id: I0e53f41a23566b5ad5f0fed12724e02a10b02707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9731
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This provides a nom-based parser for Nix derivations in ATerm format,
which can be reached via `Derivation::from_aterm_bytes`.
Some of the lower-level ATerm primitives are moved into a (new) aterm
module, and some more higher-level ones that construct derivation-
specific types.
Also, move the escape_bytes function into there, this is a generic ATerm
thing.
Change-Id: I2b03b8a1461c7ea2fcb8640c2fc3d1fa3ea719fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9730
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Gerrit instructions now recommend fetching the commit-msg hook via HTTP
in all three cases, we should do the same.
Change-Id: I12d5784ccacd3ea68fb81fc15508b209a22cd2c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9745
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Bump gerrit plugins dep hashes, and for code-owners rebase it against
master.
Change-Id: If7da0ca391b4a5c0102560ca8d52b6f5a2dfd223
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9734
Autosubmit: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
We can cross-reference all of these to schema.org, it should work for
most of the fields.
Change-Id: I38d8dbc7e964764886ddd156c4148bcf3ee376f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9744
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Ideally we can figure out how to search for single songs by grepping
through musicbrainz. For this we kinda need the jsonld results, so
this is a first step which visualizes the structure and makes it
easy-ish to lazily traverse it.
Change-Id: Ieca21674dee8e8c2dacbab4f2f15ccbe067da647
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9743
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
We want to use this quite generic parser type for other things as
well.
Change-Id: I890b43c58e479bdf2d179a724280ef1d8748fafa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9742
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
We don't actually care if it's a BTreeMap of strings, bstrings or any of
that sort.
The only thing we want to be able to do is get a reference to the bytes
from the keys and values.
Change-Id: I21b85811a9ea47fe06afa3108836ef9295e5d89b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9737
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
We use test_resources and globbing for some of the test cases, so adding
additional files in there will also create new test cases, which we
don't always want.
Move it down one level to make some more space.
Change-Id: I619867dc80a4ced59d45096d0703678663b559cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9729
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Limit the amount of memory consumed on the stack for NixHash. Sha512
isn't used that often, so it's fine if we heap-allocate it.
Change-Id: I4a9eecd20c6184610124dc130c41bfa5d0dc04c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9726
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tonic support was added to tokio-listener upstream which removes the
need for use to have tonic compatibility wrapper types around it.
See: https://github.com/vi/tokio-listener/pull/2
Fixes b/311
Change-Id: I04a2dbb3bc3c8bfe9339583c0b46070c7ec97811
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9721
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is of known length.
Change-Id: Iba48ccc486d5bf9e38ec1a2da6e7b80997d2c6ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9723
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This updates the code to make use of the new Validate() function defined
on a Node.
Change-Id: I9b6ed694661f41e700f19cc78d53d2224b61852d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9718
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This uses the newly introduced StorePath message type to add a Deriver
field to the PathInfo message.
Support for validation is added to both the golang and rust
implementation. This includes extending unit tests.
Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This encodes a store path a bit more concise, which is used in the
Deriver field.
Change-Id: Ibfb54d3b206917e51970d1d5fe94fcedb901704b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9646
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
To make this easier, move it outside of TvixStoreFs, and accept the
DirectoryService as a function argument, so we don't need to worry about
the lifetime of self.
This also aligns with how we spawn async tasks inside the rest of
TvixStoreFs.
Change-Id: I3b95072209d32039f05aed122240f2d6db7ad172
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9713
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This moves the repetitive code to parse a URL and create a channel
connected to it into `tvix_castore::channel::from_url`.
Part of b/308
Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Move the channel creation depending on the string-based URL into its
own block.
Change-Id: I546b769acd2296b548eb966b62c495f910266df5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We had to have these all while the traits where sync, and there was a
lot of spawning and moving.
Most of this can now be removed in favor of some inline `.clone()`.
Change-Id: Id5466c32a403100bc3347866b3172e06a792e311
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Similar to cl/9715, this makes the validation checks more granular,
introducing a Validate on all *Node.
A check for symlink targets is added too.
Once merged, it can also be used from tvix/store/protos.
Change-Id: I0909a89fadcd74b74ef0c9a8a1f22658fccc83b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9716
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We can just check the digest length to be correct, and move the symlink
target checks to a single line.
Change-Id: I41d2e3a50e7990ef6c04f02acd754b1e17b43e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9717
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Implement `validate()` on `node::Node`, and call it from PathInfo's
validate() too. Node-related errors are moved to a ValidateNodeError
error type.
This additionally adds some more validations for symlink targets (they
must not be empty, and not contain null bytes).
Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
