This will allow feature-flagging some of the backends.
Change-Id: Ie92914c3e2ad870eee87e73b3b5abe605fb56fe7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11202
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This will allow feature-flagging some of the backends.
Change-Id: Iddbdb89d3cf9c966a2c25b06b03e6917b284cae5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11201
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This will allow feature-flagging some of the backends.
Change-Id: Idffbf8b3fd154f5a3d938225c3871feffea8ff8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11200
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Setting the wire module as public to re-use it from tvix-daemon.
Change-Id: I570cffc480c7b784d813663f77572bbe9d4e8259
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11197
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The accompanying test case shows that we need to bubble up the catchable
error from the equality check if one is created.
Change-Id: Ic9929a57aa7653c8aa5a72d1711cf3264798c731
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11159
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
It is pretty pointless to force the function argument if we are going to
use a suspended call later since forcing the function may fail in ways
that are not covered by Catchables (non-recoverable errors, infinite
recursions). From this, it kind of seems as if using #[catch] is never
correct and should be replaced by #[lazy]. Also we should probably try
to come up with more test cases for stuff where laziness gets us out of
the jam as an equivalent to the catchable tests for nonrecoverable
errors.
Fixes b/386.
Change-Id: Ia926df4ac1b440ec430403ab7b40924a0c97221b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11153
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
This will give us the sha256: prefix, and hashes we're more used to, in
that context.
Change-Id: I72e42fe685e365ba9baa7cd81001387d239fa7c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11194
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Ensure consistent formatting of blake3 digests.
It looks like we don't actually construct these error types anywhere, so
no code needs to be refactored to it currently, but still good to be
consistent.
Change-Id: I49dc8a7f3cb4245ac06b9a6a44b72060434a3d32
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11193
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
We don't need to use BASE64 here on our own, B3Digest has a Display
impl.
This will also make sure the `b3:` digest is present in field values.
Change-Id: I0ce6ee0f7e7e99fb9b16872953a1b742e99be291
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11192
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This can be used in the future to boot proper NixOS VMs, which want a
init= in their cmdline.
Change-Id: Iad4a25d5081f3a6af1c7f62f15853f3afaae4a0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11190
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We don't have a OTLP collector inside the Nix build, this removes some
noise from the log about not being able to reach it.
Change-Id: I058839c8f214821b536306c8c30e1a8e7192153b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11189
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This updates the parameters mkBootTest can be called with. It now
accepts a `path`, and then either importPathName needs to be set, or
isClosure needs to be set to true.
The former activates the existing functionality, tvix-store import is
used to import contents as a NAR-addressed store path.
The latter uploads the path as a closure (so including its references,
and keeping the store paths intact) to tvix-store.
We use nar-bridge, and the HTTP interface it provides to do this.
As `nix copy` can't be used inside a Nix build, we use
`pkgs.mkBinaryCache` to come up with the .narinfo and .nar files that
would be in a binary cache, and then use a bit of GNU Parallel and bash
to upload store paths ourselves.
Change-Id: Icfa5c0af0c22ab5418686947aa2c060f5987b873
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11188
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
We currently use a pretty big kernel to boot the tvix VMs, with a lot
of drivers.
It can probably be slimmed down significantly, at least on the hardware
driver and filesystem front.
Change-Id: I4c7f73fb2dafbf2bcdec8057d2b14a9b0e9b3275
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11187
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Fork off a long-living tvix-store daemon in the background, and have
both the import, as well as the virtiofs invocation connect to that one,
rather than use the underlying storage directly.
This also opens up the potential to have other things seed the store
(like nar-bridge).
Change-Id: Ie13aedaf7ab31e5ebe78a7be42f144af3920fa9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11186
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Have derive_{blob,chunk}_path emit trace-level events for both the
values they're called with, as well as the return value.
With RUST_LOG in place, it doesn't get lost in other unrelated noise.
Change-Id: Id2451e3657324eff482841eb26a22d19e22bde30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11136
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This allows selectively increasing the log level for only parts of the
stack.
For example, the following RUST_LOG env var enables "tracing" level
logging for `tvix_store` and `tvix_castore`, while keeping it at "info"
for the rest of the stack:
export RUST_LOG='info,tvix_store=trace,tvix_castore=trace'
It only affects logs, not traces (if enabled).
Change-Id: Ib936bd132a405f216e75c843db83fbd71d20a18a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11182
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Whenever this encounters an open_read(), it'll first check for more
granular chunking. If there's more granular chunking data available, a
ChunkedReader is constructed (which supports seeking backwards).
This currently is still a bit stupid, and doesn't compose, as
`ChunkedReader` uses `self` as the `BlobService` to ask for the
individual chunks.
In store composition future, we might want to compose this differently,
essentially constructing `ChunkedReader` with another `BlobService`
representing the entire hierarchy, so there's a chance to locally cache
things, and do less requests.
Change-Id: I22e0df4d6245f666d083b4f0b7114d3ac41d1dce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11185
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This simply checks for the address to contain slashes, and if so, opens
a unix socket, rather than a tcp one. We'll use this in //tvix/boot
tests to simplify waiting for nar-bridge to be up.
Change-Id: I7184f548d57142b1c5f698a1f0c30343489373a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11184
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Use the same format as Display, b3: followed by the base64
representation. This makes the debug implementation of everything
containing a b3 digest much nicer to read.
Change-Id: I3ca3154d0b6fb07781c8f9c83ece3ff1a6957902
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11181
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This bumps tonic and surrounding crates to 0.11.x.
We added support for tonic 0.11.x into tokio-listener
(https://github.com/vi/tokio-listener/pull/4), so that's bumped as well.
Change-Id: Icfade5894403228299836fefb21b2f9ae59dbebb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11156
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
`build.rs` emits rerun-if-changed statements for all proto files, as
well as all include paths we pass it.
Unfortunately, due to protobufs include path rules, we need to specify
the path to the depot root itself as an include path, at least when
building impurely with `cargo`. This causes cargo to essentially always
rebuild, as it also puts its own temporary files in there.
Unfortunately, tonic-build does not chase down to individual .proto
files that are included.
Disable emitting these `rerun-if-changed` statements for now.
This could cause cargo to not rebuild protos every time, causing stale
data until the next local `cargo clean`, but considering the protos
change not that frequently, and it'll immediately surface if trying to
build via Nix (either locally or in CI), it's a good-enough compromise.
Change-Id: Ifd279a2216222ef3fc0e70c5a2fe6f87997f562e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11157
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
When we added the Nix language test suite in cl/6126, we excluded the
whole tvix tests folder from Nix formatting. This is unintentional, as
we probably want *our* tests to be formatted correctly.
Change-Id: I2b66d79e30fae17e75d5a1f8c44e279886091c5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11154
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This introduces a version reading sized byte packets. Both read_bytes,
accepting a range of allowed sizes, as well as read_bytes_unchecked,
which doesn't care, are added, including tests.
Co-Authored-By: picnoir <picnoir@alternativebit.fr>
Change-Id: I9fc1c61eb561105e649eecca832af28badfdaaa8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11150
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
These are not streams, but AsyncRead and AsyncWrite.
Change-Id: I7d988fa0490800b72862f4f0fcac3dceac70ec26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11149
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
This brings some initial Nix wire format parsing code, used in the nix
daemon protocol, remote store/builder protocol, as well as the NAR
format itself (note we already have more specialized code for the last
one).
Thanks to embr, this code already exists, in
https://codeberg.org/gorgon/gorgon/src/branch/main/nix-daemon/src/wire.rs,
and we can vendor it into here, as EUPL is compatible with GPL (in that
direction).
The code uses the tokio::io Reader and Writer traits, not the ones from
the `futures` crate, as they provide some more convenient `read_u64_le`
functions.
More application-specific parsing code, as well as code to read strings,
or bytes are left out for now, as we want to be be more restrictive
w.r.t allowed max sizes, and need to parse bytes, not strings.
The code slightly diverges, as we have clippy looped into CI.
`Ok(…?)` can be turned into just the inner expression, and
some .and_then can be expressed in a simpler fashion.
Change-Id: Ie3adcb485e9d66786673b1962a08d4e5df3781d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11148
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
derivation_or_fod_hash constructs ATerm bytes and feeds them to sha256.
input_derivations being slightly modified is an implementation detail,
so move the BTreeMap construction inline, and have aterm_bytes in a
let binding (and feed it to the hash function directly while
constructing it).
This makes it a bit more understandable what's going on.
Change-Id: I2f5cfbd1c964fd39ac731ca39e76cfc168f4c7d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11147
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
This is only used inside this function, in 2 of the match cases.
Change-Id: Ib361f5ee0e3b203802f7d05b9a7f332d14bbcf80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11143
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Tested-by: BuildkiteCI
The outer fingerprint used for store path calculation is always a sha256
digest. This includes both input and output-addressed store paths.
We used a NixHash here, which can also represent other hash types, and
that had a bunch of annoyances:
- Whenever we had the bytes, we had to wrap them in a NixHash::Sha256().
- Things like AtermWriteable had to be implemented on NixHash,
even though we then had an assertion it was only called in the
NixHash::Sha256 case.
Change-Id: Ic895503d9b071800d2e52ae057666f44bd0ab9d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11142
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
This makes BLOB_SERVICE_ADDR configurable, and creates a flavor setting
it to objectstore+file://$PWD/blobs.
Change-Id: I68c21367f83f68b4dee701c5678f438c7d8fbe43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11137
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
The object_store crate supports a ton of different stores, with different schemes.
For now, use a objectstore+ scheme prefix to enable these.
Change-Id: I946f76e32a0fb0867ef59060217894cda5b959b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11080
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
This uses the `object_store` crate to expose a tvix-castore BlobService
backed by object storage.
It's using FastCDC to chunk blobs into smaller chunks when writing to
it.
These are exposed at the .chunks() method.
Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
This was only possible by disabling without the otlp feature flag so
far.
Introduce the same --otlp=false mechanism that nar-bridge also supports
to be able to turn it off at runtime.
Change-Id: Ib22a364c35056ca9d8e327c0e2a79970a4cf4b2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11135
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Add a new AsyncRead wrapper, DecompressedReader, that wraps an
underlying AsyncRead, but sniffs the magic bytes at the start of the
stream to determine which compression format is being used out of the
three that are supported by builtins.fetchTarball, and switches to the
correct decompression algorithm adapter dynamically.
This will be used in the implementation of builtins.fetchTarball
Change-Id: I892a4683d5c93e67d4c173f3d21199bdc6605922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11019
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Implement the fetchurl builtin, and lay the groundwork for implementing
the fetchTarball builtin (which works very similarly, and is implemented
using almost the same code in C++ nix).
An overview of how this works:
1. First, we check if the store path that *would* result from the
download already exists in the store - if it does, we just return
that
2. If we need to download the URL, TvixStoreIO has an `http_client:
reqwest::Client` field now which we use to make the request
3. As we're downloading the blob, we hash the data incrementally into a
SHA256 hasher
4. We compare the hash against the expected hash (if any) and bail out
if it doesn't match
5. Finally, we put the blob in the store and return the store path
Since the logic is very similar, this commit also implements a *chunk*
of `fetchTarball` (though the actual implementation will likely include
a refactor to some of the code reuse here).
The main thing that's missing here is caching of downloaded blobs when
fetchurl is called without a hash - I've opened b/381 to track the TODO
there.
Adding the `SSL_CERT_FILE` here is necessary to teach reqwest how to
load it during tests - see 1c16dee20 (feat(tvix/store): use reqwests'
rustls-native-roots feature, 2024-03-03) for more info.
Change-Id: I83c4abbc7c0c3bfe92461917e23d6d3430fbf137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
`#tvl` is the general TVL community channel, `#tvix-dev` is dedicated to
Tvix development discussion.
Change-Id: If899d9ae70dffb4acf6a41ded54f80a1f0551c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11110
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
request only contains the outer metadata wrapping, and that's not too
interesting:
> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }
Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.
Also, ensure we handle all error cases properly, and log them. We
don't use `err` from instrument, as that'd also log an error on
`Status::not_found`.
Change-Id: Id1b983cb8b059c148c8a376f8802a1d28c59ba97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11103
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This only contains the outer metadata wrapping, and that's not too interesting:
> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }
Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.
Clean up the error logging a bit.
Change-Id: Ife1090ed411766a61e1fa60fd4c9570f38de1e98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11102
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This only contains the outer metadata wrapping, and that's not too interesting:
> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }
Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.
Log errors in some places where we didn't so far.
Change-Id: Ia68d6c526987d3716be62a0809195401cf28512b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11101
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Otherwise this causes conflicts when adding a more recent zstd version
to another crate.
Change-Id: I02d1b1a37f18711c0969cb04e761e967fc4655dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11100
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
For everything using reqwest here during test cases, we also need to
set SSL_CERT_FILE.
Change-Id: If8aeda65f3d75cb9ac5c9bc64e37a0cb7dffc17c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11092
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
implement From<&nix_compat::nixhash::CAHash> for nar_info::ca::Hash
individually, and make
From<&nix_compat::nixhash::CAHash> for nar_info::Ca a small wrapper that
uses it, as well as the .hash().digest_as_bytes() for the digest.
Change-Id: I7e9b6edd1e3f149eb270faf2928cd846d74e77ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11071
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
If there's an unexpected test failure, print it out, rather than just
saying something is false even though it should be true.
Use .expect() for this, which displays the error if it failed.
We can't use expect_err(), as our stores are not display'able, so use an
assertion with a message there.
Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
This was true while we didn't do string contexts, but drvPath should not
differ anymore.
Change-Id: I58e6ed439519bd4d68259d1577946f3c65110dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11074
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
