This provides a DirectoryService implementation which uses
redb (https://github.com/cberner/redb) as the database. It provides both
in-memory and persistent on-filesystem implementations.
Change-Id: Id8f7c812e2cf401cccd1c382b19907b17a6887bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12038
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
The "path" key in the arguments to builtins.path supports any
path-coercible type (a string, a path...). Coerce it to a path in the
argument rather than just requiring it already be one and throwing an
error if it's not.
This is... annoying to test, since it requires a file with known
contents that's available in the build sandbox. But it works! Trust me!
Fixes: b/412
Change-Id: I3c8e339bf344a208d5ed5990193942651f318745
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12053
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
This change exposes the already existing wrapper for treefmt/depotfmt
that supports running it inside a sandbox. We now reuse it inside
//tvix/crate2nix-check, where we previously duplicated the code. The
check is now stricter and will also fail on changes, so I had to set the
rust edition in the treefmt config.
Change-Id: I000e52421258979c038ba6b1f1ff2db14e391b0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12068
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
The environment variables caused both nar-bridge and tvix-store daemon
to try to connect to the same store, which fails due to locking issues.
Pass the config to `tvix-store daemon` directly. Also, add the
`--otlp=false` to the instructions to remove spam, most users don't have
a OTLP collector running.
Explain when to use gRPC in `tvix-store virtiofs` and when not.
Also, point out to be able to boot a NixOS closure, it needs to be
copied into the store first.
Change-Id: If4eda07bba28ad0bbe70e468cb727441a21b0588
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12067
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
It was previously pointing to a sled implementation which no longer
exists. It was also using nar-bridge-go instead of the new nar-bridge
(rust).
Closes: https://b.tvl.fyi/issues/413
Change-Id: Id0df61d4728198c3ae95a8e27ba7303434892966
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12063
Reviewed-by: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
This is nice to test too - it's similar to hello, but runs for a lot
longer (like 7.5 seconds on my laptop) which means we get even better
stats for stuff.
Change-Id: I7935818f10a6d846d446e685b9263a72d7e2aabd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12061
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
We already do this for redb and for sled in SledDirectoryService.
Change-Id: I34c7178257a6a04e9f12ed4037a4ef585d7b0d54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12060
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Add more logging and remove context from errors because that's already
provided by the logs (Errors also need to be refactored anyway, there's
also confusion about StorageError vs InvalidRequest, there's no
consistency)
Change-Id: Ia43c0d237d9075152490c635b05fb3fb343abcc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12058
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The decode function didn't check that the input had a valid length and
so would panic when given input with invalid length.
Change-Id: Ie27d006b8fe20f005b4a47a1763821a61e9a95c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12051
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: Brian Olsen <me@griff.name>
Previously, OpConstant would display some detail about its
ConstantIdx: whether it's a thunk or closure, and what its address
is. This has been expanded to also show when the ConstantIdx is a
blueprint, along with the blueprint's address, and to the other
opcodes that use a ConstantIdx.
Currently, it seems like blueprint addresses don't correspond to the
address of the thunk listed in the bytecode output, but it's still
useful to see that the constant being grabbed is a blueprint, and
maybe this pointer can be made to make more sense in the future.
Change-Id: Ia212b0d52b004c87051542c093274e7106ee08e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12044
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: chickadee <matthewktromp@gmail.com>
path_exists was returning an error when certain common IO errors were
encountered. e.g. in the path "/dev/null/.", path_exists would throw
an error because the underlying call to Path::try_exists threw an
error because null isn't a directory. But if null isn't a directory,
then the path is invalid, so this should really be returning
false. That's what nix's behavior is and that's what makes sense.
The trait function isn't being changed because some other
implementers (e.g. tvix_store_io) have actual errors they can throw.
Fixes: b/411
Change-Id: I9e810e7a198bffe61365697c6d3d7e71f264c20b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12042
Tested-by: BuildkiteCI
Autosubmit: chickadee <matthewktromp@gmail.com>
Reviewed-by: aspen <root@gws.fyi>
This is the `{fixed,fixed:r,text}:{sha*,md5}` prefix used in various
string representations.
Factor that code out, and use it in the two places it can be used in.
Change-Id: Ic9555fa9e1884198d435e55c7f630b8d3ba2a032
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12041
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
When retrieving a closure with get_recursive, the following could happen in the GRPC client:
- The first reference to the deduplicated directory is added to expected_directory_digests
- The deduplicated directory is obtained removed from expected_directory_digests
- The second reference to the deduplicated directory is added to expected_directory_digests
- The deduplicated directory has already been sent, but is still in the
expected_directory_digests. It looks to the GRPC client like the
closure is incomplete and the stream ended prematurely.
Change-Id: Ic62bca12e7f8fb85af5fa4dacd199f0f3b8eea8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12033
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Use the ChunkedReader in CombinedBlobService instead which also supports seeking.
Change-Id: I681331a80763172c27e55362b7044fe81aaa323b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12031
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Ensure nar-bridge is healthy before connecting to it, don't just check
for the unix socket to be present.
We don't have a proper /health endpoint yet, but nix-cache-info works
fine for now.
Change-Id: I22df2c3b7bffcf52dbd3d00f3ba5382dc06ab03d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12030
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Ensure the service is healthy before connecting to it, don't just check
for the unix socket to be present.
Change-Id: If6501828677c247910d91f35b860960802084691
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12029
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This provides a PathInfoService implementation using redb
(https://github.com/cberner/redb) as the underlying storage engine.
Both an in-memory variant, as well as a filesystem one is provided,
similar how it's done with the sled implementation.
Supersedes: https://cl.tvl.fyi/c/depot/+/11692
Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc
Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Parsing of the narinfo file sets the compression field to None instead
of Some("none"). The mapping selecting the decompression reader expected
the former in //tvix/store/src/pathinfoservice/nix_http.rs.
Change-Id: I254a825b88a4016aab087446bdc0c7b6286de40c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12007
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a generic `SigningKey` struct that can be used to sign
NARInfos with signers.
It also includes tooling to parse keypairs from bytes generated by Nix,
returning a specialized ed25519_dalek variant.
Change-Id: Ic9780c370939af54e7177c93cde3321adf189fc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12014
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Align these with the way it's called in the ed25519 crates.
Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This documents some thoughts and goals of the Tvix Build protocol, and
how it is possible to express Nix builds with it.
Additionally, it explains a proposed design for reference scanning.
Change-Id: I4b1f3feb2278e3c7ce06de831eb8eb1715cba1c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12012
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
We still have the unique store name to identify which instantiation caused the error. For recursion errors, the full chain is still retained inside the CompositionError.
Change-Id: Iaddcece445a5df331e578d7c69d710db3d5f8dcd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12002
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Having `tvix-store daemon` and `nar-bridge` listen on the same port by
default is silly.
Use the same port that nar-bridge-go was using previously.
Change-Id: Id374a12e52ba9dac4318e43d0ca0853866eadabc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11998
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
It introduces a new accept_trace function for axum0.7 which can be used
to accept a header trace from a received request. This function can be
used for tonic 0.12 once that version is released, and the specific
`accept_trace` function within `tvix_tracing::propagate::tonic` can then
be removed.
This also integrates http propagation into the nar_bridge crate.
Change-Id: I46dcc797d494bb3977c2633753e7060d88d29129
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11925
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
This switches the boot tests from the golang implementation to the rust
one.
Change-Id: Ide2a47aebe40c172077147c05bb6dacd74ba6b1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11997
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
This got broken while moving things around. We need to parse the
b64-decoded bytes.
Since we're now validating the root node, we also need to rename the
root node to get past the node name validation.
There probably should be some tests for this.
Co-Authored-By: sinavir@sinavir.fr
Change-Id: I8f24a4a0ac107b1ea5b94c0e0ed872a34eb7b587
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11996
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
In case two crate versions are present, the key in the Cargo.nix file
includes the version number too.
To be able to set a specific hash for "tracing-opentelemetry 0.25.0" for
example, this needs to account for keys with the version included.
Access `crateName`, `version` and `src.outputHash` individually.
Change-Id: Ib9800691a445ac403ff646cb32e85a7a4cbef9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11994
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
