It introduces a new accept_trace function for axum0.7 which can be used
to accept a header trace from a received request. This function can be
used for tonic 0.12 once that version is released, and the specific
`accept_trace` function within `tvix_tracing::propagate::tonic` can then
be removed.
This also integrates http propagation into the nar_bridge crate.
Change-Id: I46dcc797d494bb3977c2633753e7060d88d29129
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11925
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
This switches the boot tests from the golang implementation to the rust
one.
Change-Id: Ide2a47aebe40c172077147c05bb6dacd74ba6b1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11997
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
This got broken while moving things around. We need to parse the
b64-decoded bytes.
Since we're now validating the root node, we also need to rename the
root node to get past the node name validation.
There probably should be some tests for this.
Co-Authored-By: sinavir@sinavir.fr
Change-Id: I8f24a4a0ac107b1ea5b94c0e0ed872a34eb7b587
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11996
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
In case two crate versions are present, the key in the Cargo.nix file
includes the version number too.
To be able to set a specific hash for "tracing-opentelemetry 0.25.0" for
example, this needs to account for keys with the version included.
Access `crateName`, `version` and `src.outputHash` individually.
Change-Id: Ib9800691a445ac403ff646cb32e85a7a4cbef9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11994
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
This adds support to upload NARInfo files. We lookup the root node from
the LRU cache, rename it appropriately and then put it into the
PathInfoService.
Change-Id: I5479032b51cd855363bc016dee63cf84b3304a36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11988
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
We currently send 405, and that confuses `nix copy`.
Send a 404 for now, and add a futurework, as we can actually at least do
something more meaningful in case we still have that nar hash in our
LRU, which would avoid some unnecessary uploads in some cases.
Change-Id: If625e9bd0fd6506cb73b88962d889aa08315fcea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11987
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
This ingests NAR files into the {Blob,Directory}Service, which are
already part of the AppState.
As we then need to correlate the root node to the uploaded PathInfo, we
need to keep a (short-lived) lookup table from NARHash to root node
around. We insert it into a `LruCache` after the NAR is uploaded, and
use `peek()` to do the lookup, which doesn't update the LRU list.
Change-Id: I48a4c6246bacf76559c5a4ccad2a0bc25c1b7900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11986
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
This adds an implementation of nar-bridge in Rust.
Currently, only the GET parts are implemented.
Contrary to the Go variant, this doesn't try to keep a mapping from nar
hashes to root node in memory, it simply encodes the root node itself
(stripped by its basename) into the URL.
This pulls in a more recent version of axum than what we use in
tonic, causing two versions of http and hyper, however dealing with
`Body::from_stream` in axum 0.6 is much more annoying, and
https://github.com/hyperium/tonic/pull/1740 suggests this will be fixed
soon.
Change-Id: Ia4c2dbda7cd3fdbe47a75f3e33544d19eac6e44e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11898
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This allows avoiding a `.node.unwrap()`` after validation.
Change-Id: Ieef1ffebab16cdca94c979ca6831a7ab4f6007da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11989
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Also reorder to how it's ordered in the .proto file.
Change-Id: I87c422feac4d12f6a84f99aa889f9c524a9878b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11985
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Uses `Set` to mark that the given list is actually a set.
The format on stream is no different from `List` but is nice to
know.
Best effort document string formats.
Documents default values for fields not used in old protocol
versions.
Document handshake, AddMultipleToStore format, Export format and
Import paths format.
Change-Id: I6cbe4c7de3770cc5caafc26229f99a07353ee3e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11984
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Brian Olsen <me@griff.name>
This will be necessary for the PathInfoService composition, as some
PathInfoService implementations require a BlobService & DirectoryService
to ingest into.
Using the Extend trait for creating compositions allows extending the same
composition with configs of various types e.g. BlobStore, DirectoryStore
Generics are moved from the Composition struct to the functions.The storage of
the InstantiatonStates uses the TypeId in the key and a Box<dyn Any> in the
value, which is downcasted to InstantiatonState<T>.
Change-Id: I2af11f26c535029adfb1c62905e0e7c4aaed7b51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11980
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
We were wrongly using `buf.initialized()` which contains more than the
filled portion to compute the number of bytes read during a poll call.
This made us go into the trailer reading state too early and finally
failing due to invalid trailer data.
Fixes b/405.
Co-authored-by: Florian Klink <flokli@flokli.de>
Change-Id: I66ba6e2116389e6b97305f85c4e0922195000e45
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11978
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
this is useful when oneshot-instantiating a store from a single config
Change-Id: I08538fdee1d0bb26b3ae2da7d3b2339b2e93bc0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11975
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
The Cargo.lock also got changed because of crate2nix using --freezed.
Change-Id: I79b062eddb13f59c4656a62735df1e527fc91a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11970
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The Cargo.lock also got changed because of crate2nix using --freezed.
Change-Id: I48cabe514e0cf1dd1f00870bcb7958a2a49663d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11969
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
The Cargo.lock also got changed because of crate2nix using --freezed.
Change-Id: I90245bad27f6fee01455557bbccc865301a4f1a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11968
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The Cargo.lock also got changed because of crate2nix using --freezed.
Change-Id: Id5225e73ec5894bc324f363660472a7efdd803cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11967
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Previously we had to make a mutable Config instance and set bytes and
other values in it because they were not exposed to the builder pattern
(https://github.com/hyperium/tonic/issues/908) but now they are, so we
just set them through the builder.
Change-Id: I8904c6b93f09173b56586024b1ced59d622bce66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11966
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
reqwest wants to be able to read a file of trust roots when constructed,
but as it doesn't actually do any HTTPS connections inside the nix
build, an empty list of trust roots is totally sufficient.
Thankfully /dev/null provides such a file.
Change-Id: I9bd1619b2c9f8ff2a6640d2ac410d4de5b20c2ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Now that we can bind (potentially lazy, potentially lambda-containing)
values in the REPL and then reference them in subsequent evaluations,
it's important that the values to which we construct shared references
are shared across those subsequent evaluations - otherwise, we get
panics due to unknown source map locations, or dropped weak references
to globals.
This change assigns both the globals and the source map as fields on the
Repl after the first evaluation, and then passes those in (to the
EvaluationBuilder) on subsequent evaluations.
On the EvaluationBuilder side, there's some panicking introduced - this
is intentional, as my intent is for the builder to be configured
statically enough that panicking is the best way to report errors
here (it's always a bug to misconfigure an Evaluation, and we'd never
want to handle it dynamically).
Change-Id: I37225697235c22b683ca48a17d30fa8fedd12d1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11960
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Juggle around the internals of the tvix-cli crate so that we expose the
Repl as a public type with a `send` method, that sends a string to the
repl and *captures all output* so that it can be subsequently asserted
on in tests. Then, demonstrate that this works with a single (for now)
REPL test using expect-test to assert on the output of a single command
sent to the REPL.
As the REPL gets more complicated, this will allow us to make tests that
cover that complex behavior.
Change-Id: I88175bd72d8760c79faade95ebb1d956f08a7b83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11958
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Construct the Rc<GlobalsMap> for the evaluation as part of
EvaluiationBuilder::build, rather than deferring it until we actually
compile. This changes nothing functionally, but gets us one step closer
to sharing this globals map across evaluations.
Change-Id: Id92e9fb88d974d763056d4f15ce61962ab776e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11957
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Make constructing of a new Evaluation use the builder pattern rather
than setting public mutable fields. This is currently a pure
refactor (no functionality has changed) but has a few advantages:
- We've encapsulated the internals of the fields in Evaluation, meaning
we can change them without too much breakage of clients
- We have type safety that prevents us from ever changing the fields of
an Evaluation after it's built (which matters more in a world where we
reuse Evaluations).
More importantly, this paves the road for doing different things with
the construction of an Evaluation - notably, sharing certain things like
the GlobalsMap across subsequent evaluations in eg the REPL.
Fixes: b/262
Change-Id: I4a27116faac14cdd144fc7c992d14ae095a1aca4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11956
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
This is done now, as of fc6359463 (feat(tvix/repl): Allow binding
variables at the top-level, 2024-07-04)
Change-Id: I2126cbadeb66625cda93a7cd7ddb31409ec60fd9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11955
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
With the recent changes to crate2nix and buildRustCrate in nixpkgs it is
now possible to build tvixbolt via crate2nix like we do for other tvix
crates. We can reuse a lot of the customizations done in //tvix in
tvixbolt to avoid repeating ourselves.
A script for serving tvixbolt locally for testing purposes is also
available now through the .serve attribute of tvixbolt.
This change supersedes https://cl.tvl.fyi/c/depot/+/11821.
Change-Id: I4864df8b75aec73cf5fee2428924ed4cfbb32902
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11952
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
This function can be reused across the rest of the repo to make use of
our overriden defaultCrateOverrides with support for tvix crates.
Change-Id: I8c554dece052bd9dd32acac13dab8114933272a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11951
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Previously we would unconditionally add Cargo.toml to the fileset. We
mostly use buildRustCrate in tvix so it does not make sense to add it by
default, instead I made it so you enable cargoSupport if you want
Cargo.{toml,lock} to be added to the fileset.
Change-Id: I5a6016534fc5599e85ab581fe3d9b81e7a24f940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11950
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Allow binding variables at the REPL's toplevel in the same way the Nix
REPL does, using the syntax <ident> = <expr>. This fully, strictly
evaluates the value and sets it in the repl's "env", which gets passed
in at the toplevel when evaluating expressions.
The laziness behavior differs from Nix's, but I think this is good:
❯ nix repl
Welcome to Nix version 2.3.18. Type :? for help.
nix-repl> x = builtins.trace "x" 1
nix-repl> x
trace: x
1
nix-repl> x
1
vs tvix:
tvix-repl> x = builtins.trace "x" 1
trace: "x" :: string
tvix-repl> x
=> 1 :: int
tvix-repl> x
=> 1 :: int
Bug: https://b.tvl.fyi/issues/371
Change-Id: Ieb2d626b7195fa87be638c9a4dae2eee45eb9ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11954
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Allow passing in a top-level env, a map from name to value, to
evaluation. The intent is to support bound identifiers in the REPL just
like upstream nix does.
Getting this working involves mucking around a bit with internals - most
notably, locals now only optionally have a Span (since locals don't have
an easy span we can use) - and getting that working requires propagating
some minor hacks to places where we currently *need* a span (and which
would require too much changing now to make spans optional; my guess is
that that would essentially end up making spans optional throughout the
codebase).
Also, some extra care has to be taken to close out the scope in the case
that we do pass in an env, to avoid breaking our assumptions about the
size of the stack when we return from the toplevel
Change-Id: Ie475b2d3dfc72ccbf298d2a3ea28c63ac877d653
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11953
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
crate2nix can now build WASM, and cl/11859 showcases this now.
Even if it's not in the same cargo workspace, we should still migrate
tvixbolt to a crate2nix build.
Remove the part about Build/Store frontends, it was more of an
explanation why we want WASM builds to be nice, it can be tracked in
separate TODOs once more concrete.
Change-Id: If4f5e0994b55520ba70cabefb4fcef9dc17bc394
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11945
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
This was made unnecessary in c92d06271 (feat(tvix/eval): drop
LightSpan::Delayed, 2023-12-08) because it didn't improve benchmarks as
much as expected and has been vestigial since; this continues the
cleanup by just removing it altogether
Change-Id: I21ec7ae9b52a5cccd2092696a5a87f658194d672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11949
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Includes hacky, but workable solution for the pronunciation ‘issue’
Karl comes from: http://ipa-reader.xyz/?text=tvɪks&voice=Karl
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Iad7788ec7295902fd2159766a664016c7b1e2ae9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11908
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We were wrongly comparing the raw row_key with the store path
digest, but we hexlower-encode the digest in the row key (via
derive_pathinfo_key).
Update the logic to fix that.
Change-Id: I8916d8de9fb8b25a6986d4158faa91ec97c57347
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11926
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Introduces a helper function within tvix-tracing that returns a reqwest
tracing middleware that will ingest the traceparent if otlp is enabled.
It is feature flagged in tvix-tracing so not every consumer of that
library automatically has reqwest in its dependencies.
Tested using netcat to verify that the `traceparent` header is there if
otlp is enabled and missing if otlp feature is disabled.
Change-Id: I5abccae777b725f5ff7382e3686165383c477a39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
See https://github.com/NixOS/nixpkgs/pull/323753 for details.
Changes:
* git: temporarily comment out dottime patch (it doesn't apply, but it's not critical)
* third-party/cgit: use an older git version where dottime patch still applies
* 3p/crate2nix: remove crate2nix patches included in latest release
* tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs)
* tvix: regenerate Cargo.nix
* tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function
* tvix/nix-compat: remove redundant trait bounds
* tvix/glue: use clone_into() to set drv.{builder,system}
* tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327
* toold/depotfmt: expose treefmt config as passthru
* tools/crate2nix: undo some more hacks in the crate2nix-check drv
Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c
Co-Authored-By: Florian Klink <flokli@flokli.de>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is allows adding a cache in front of tvix-store daemon, and is less
code duplication than cl/11902, means we can probably land that until we
have proper store composition config.
It can be used to provide a tvix-store daemon interface for a Nix HTTP
Binary cache, saving all calculated PathInfo to another PathInfoService
after ingestion.
Change-Id: If141d718c2635f66aa90d46a80fd79c86c07d9ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11903
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This allows binding on unix sockets, as well as systemd socket
activation.
Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Using std::io::{Stdout,StdErr} directly will clobber the output by an
active progress bar. To resolve this issue the exposed writers should be
prefered over `println!` and `eprintln!`.
Change-Id: Ic79465cd4e8b9dad5a138f6b08c5f0de9dcf54a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11860
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The aarch64-linux version of this is too big for cloud-hypervisor to
accept.
The correct place for this is in
`${kernel}/${stdenv.hostPlatform.linux-kernel.target}`, the other output
is only useful for debugging purposes.
This gets cloud-hypervisor to start booting up the kernel. It still
doesn't boot successfully, but it's getting further.
Part of b/407.
Change-Id: I3fbe52597fb1a6f5728a7f375dd35b50e00b3b1a
Co-Authored-By: Alyssa Ross <hi@alyssa.is>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11905
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Autosubmit: flokli <flokli@flokli.de>
This uses the `preStart` mechanism to boot up a minio instance, which
we point blob/directoryservice to.
Change-Id: I9e67b19006cb08ce82fe63c885cb0178e97389e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11896
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We previously called ObjectStoreBlobService::parse_url, which passes an
empty list of options when constructing the ObjectStore.
This is most likely not what we want. The more reasonable thing to do is
pass along the query string (pairs) as options to
`object_store::parse_url_opts`, and remove them from the plain URL we
pass to object_store itself.
Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Get rid of the `let grpc_client` and `let resp` in some cases.
Change-Id: Idc1c0f566a3b1b48da62e6f1977b07620656b16c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11884
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This uses the `preStart` mechanism to boot up a bigtable emulator, which
we point directory/pathinfoservice to.
Change-Id: Icb383403cb2d1f7e19c71b6298e82b391f9c8d22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11895
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This allows running some code before tvix-daemon is started up, which is
helpful to spin up some mock services.
Change-Id: Ie7b6eaaf76c40def493f50879ee5255fd1ebd2fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11893
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
This is still useful to test the ingestion part of nar-bridge.
Reintroduce the logic removed in cl/11398, but put it behind a
`useNarBridge` parameter, defaulting to false.
Add an additional target to tvix.boot.tests,
`closure-nixos-nar-bridge`, which does the same as `closure-nixos`,
except seeding the contents through nar-bridge.
Change-Id: Iee87d5d9c7b7d02df11ee47fbc3f1fbe951935e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11891
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
These fields don't need to be set if isClosure is false.
Change-Id: Id29ed034e0757c61cef09885962dfc5507ca106c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11890
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Otherwise these are not accessible after parsing if parsed from another
crate.
Change-Id: Iba7b442722a2eb1d37c2c496a45aa559aa49c1f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11889
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This allows parsing files like `/etc/nix/nix.conf` into the `NixConfig`
struct.
Change-Id: I90b25f43c429dd56127500ff5068e83852adee13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11888
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
We want to use this in another struct that we want to compare and clone.
Change-Id: I5c11d589a4922d85e190e1a9199468411016ff1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11887
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
A lot of this content needs to be duplicated from the monorepo’s root
directory. However, some of the content was not relevant or could have
been more condensed in descriptions or split up for consumption that
also matched the styles of some other library documentation I had been
looking at for examples. I wanted to add the new access to Admonish
callouts but also aimed at the readablitiy in one’s text editor for the
user that needs to `cat` the docs.
In the future, a “Introduction” page would be nice to add to “Welcome”,
but I don’t think that is as clear.
Change-Id: Ibfc169c5fc3501d7eeb9a76423c6f64cd68897aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11869
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
these just indicate the start of the prompt by convention & if the user
is root or not with `#` & `$` respectively
Change-Id: Id65f5f879b067ee96715b3e7dc63f68c7ad791b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11868
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
NixString::iter_plain() didn't make much sense, especially without a docstring.
Rename it to iter_ctx_plain(), and copy the docstring from NixContext.
Do the same for the two other context element types too.
Change-Id: I1bbfcb967d8d9b14487d069bfe3a1f762253ef4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11882
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
We currently only had a dummy implementation that didn't actually
persist the files in the store(s).
Copy the contents to the BlobService, and do the output path calculation
as part of the upload.
Use the plain context elements to construct the references.
Change-Id: Ibdaf7a645ddc31e847faa4b87a79f2f95116a7ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11879
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
See discussion in cl/11869. Let's use `#` consistently, we also use `##`
for deeper nested subheadings.
Change-Id: Ie25ebda708639fb617d456c275ae5a264fc4ce85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11881
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: toastal <toastal@posteo.net>
Add some eval-fail-fetcht-*.nix test cases.
Change-Id: Ic02beabf120d1de8df6f9c098d5a074127069f81
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11880
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
In case the otlp feature is not enabled, these generate warnings during
`cargo check`.
Fix by moving some imports into their functions, or using the
fully-qualified name (and one #[allow(unused_mut)])
Change-Id: I5afd89dcd4c772b6002cebdd5d0469932eacfdac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11873
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
There's other places where unexpected arguments can be provided, like
in builtins.
Make space for that error type.
Change-Id: Ic831497a3a1dd36a3a184bedadcf1374bf0ae6db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11876
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
The only two consumers (fetchurl, fetchtarball) of these do try to parse
it as URL, so do it in the helper.
Update url_basename to take a &url::URL, not a &str.
Also update the test to use rstest for the fixtures to reduce some
boilerplate there.
Change-Id: I1f85fe2803060dc4423e673cb7b9f9bf799d09b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11875
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We use a bit less cloning that way.
Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
If builtins.path is passed a regular file, no filtering is applied.
We use the just-introduced file_type function in the EvalIO trait for
that.
This means, we don't need to pass through filtered_ingest, and can
assemble the FileNode directly in that specific match case.
This also means, we can explicitly calculate the sha256 flat digest,
and avoid having to pipe through the file contents again (via
blob_to_sha256_hash) to construct the sha256 digest.
Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This allows peeking of the type at a given path.
It's necessary, as an open() might not fail until you try to read()
from it, and generally, stat'ing can be faster in some cases.
Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
The builtins `placeholder` and `filterSource` are now implemented in
tvix-glue and so we no longer need their placeholder "implementations" in
tvix-eval.
Change-Id: Ibf161ccf1ad40103e9fbb688e9f6be58e7772af1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11867
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
OTLP shutdown behaviour was fixed in cl/11803, and tracy landed in
cl/10952 and cl/11815.
Change-Id: I310dfcc41ba62ce5b006e75a35feb839327b0e7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11866
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
We don't need to require these things for these impl blocks yet.
Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.
The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.
This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.
Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
By default tokio::spawn does not instrument the spawned task with the
current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do
this manually for all tokio::spawn functions in functions that are
instrumented.
Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
Disable the progressbar on default and provide a interface for
optionally enabling the progressbar.
Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
We were currently always using SimpleRenderer, which would mean the
client would download every blob locally to calculate the checksum,
which of course is very slow.
Detect this special case and create a second instance (and client) for
now.
Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
This patch has already been applied on upstream but there hasn't been an
official release yet so we patch it ourselves.
Upstreamed patch: 0209f258cd
Follow-up of: https://cl.tvl.fyi/c/depot/+/11856
Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Since https://github.com/NixOS/nixpkgs/pull/220212 libiconv is now
automatically added for Darwin.
Change-Id: I2aaa266bcbc4d5a8256131aed36b4c2d040a8758
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11856
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Since crate2nix v0.11 the new preferred way of importing Cargo.nix is
with callPackage instead of import.
Change-Id: Ifcc5c21e6c101cb9d0b3f1e8025b5ddb8cf99f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11855
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.
Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.
Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.
Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
This is currently still taking a noticeable amount of time, so make sure
we show it is happening.
Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.
So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.
Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.
Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.
Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.
Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Use the new helper introduced in CL 11708 instead of rolling our own.
Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
ClosureValidator was previously only suitable for a very narrow use case:
Validating incoming uploads, which are in leaves-to-root order.
This is because the ordering validation was hard-wired into the add()
function.
This
- Re-name ClosureValidator to DirectoryGraph, which is more suitable
since it actually stores the Directory structs and is drained in the end.
- Move the ordering-related logic to a separate OrderValidator, which
can be used independently.
- re-write DirectoryGraph to be a general purpose
validator which can accept the input in both orders
and can be drained in both orders as well.
This means the DirectoryGraph and OrderValidator can now serve
multiple new purposes:
- Validating the incoming closure on the client while downloading.
- Validating the incoming closure downloaded in a caching layer from the
`far` cache, and re-order it for insertion into the `near` cache.
Change-Id: I2b4b226348416912d7a31935bec050e53d911b70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
If compiled with this features, this emits packets compatible with the
[Tracy](https://github.com/wolfpld/tracy) format.
Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We can now cross-link between pages, and it actually works.
Change-Id: Id8bd85111672c687118db9adae7f5066f22441f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11817
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
This has been implemented in cl/11786, cl/11790 and cl/10975.
This includes support for the `executable` and `unpack` variants of it.
Change-Id: I5bcad5b8e86b9fd341219a59ba86660b692e66e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11818
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This introduces another feature flag, "tracy" to the `tvix-tracing` crate.
If enabled (not enabled by default), it'll add an additional layer
emitting packets in a format that https://github.com/wolfpld/tracy can
display.
I had to be a bit tricky with the combinatorial complexity when adding
this, but the resulting code still seems manageable.
Change-Id: Ica824496728fa276ceae3f7a9754be0166e6558f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10952
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
Having all this in the main control flow makes it a bit hard to read.
Moving it into a helper function makes it a bit cleaner.
Change-Id: Ibdb739dbd1e013b4f8c4aaf9b036a6bd556a1871
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11814
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
The problem is that opentelemetry_otlp tonic batch exporter tries to
exports if either the `scheduled_delay` or if the
`max_export_batch_size` is reached. Per default the
`max_export_batch_size` is set to 512 spans, which means that we try to
export these spans once that counter is reached. Each export will then
try to connect to the exporter (if that not already happening) and will
result in a `tcp connect error`.
Increasing the max_export_batch_size to 4096 will then ensure that the
export only happens if the `scheduled_delay` is met after the 10
seconds.
`max_queue_size` is also increased, because `max_export_batch_size`
should not be greater than `max_queue_size`, so similar to the default
config its set to `max_export_batch_size * 4`.
This will reduce the amount of tries to otlp if the collector is not
available and otlp enabled.
Change-Id: Ic3430006e8a104fa3b34d274678cae55b3620ce9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11791
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
Currently we apply the EnvFilter only to the stderr output writer.
This didn't affect any other layer, like the otlp layer, causing spans
from `h2`, `tokio_util` or other third party crate dependencies to be
always sent out via OTLP.
This changes that behaviour, applying EnvFilter to all exports, leading
to a lot less spans being exported.
Change-Id: I9f3a7233e9d0aeaa81fe08914579f0b3c80d134e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11813
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
Provide a new interface for forcing a flush of otlp traces and use this
interface to shutdown otlp prior to exiting tvix-store, either if the
tool was stopped with a SIGTERM or ended regularly.
This also fixes an issue where traces were not even exported if for
example we just imported 10 paths and never even emitted more than 256
traces. The implementation uses a mpsc channel so a flush can be done
without having to wait for it to complete. If you want to wait for a
flush to complete you can provide a oneshot channel which will receive a
message once flushing is complete.
Because of a otlp bug `force_flush` as well as
`shutdown_tracer_provider` need to be executed using `spawn_blocking`
otherwise the function will deadlock. See
https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335
Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
This provides a plugin for callouts.
It needs to have additional CSS file added to `book.toml`,
which can be (re-)generated by `mdbook-admonish install`.
See https://github.com/tommilligan/mdbook-admonish/issues/171#issuecomment-2009166079
for more context.
Use it by adding a warning one to the architecture document.
Change-Id: I75c9a33d00acb603c6da10d3f9ce3485731c1672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11805
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
These also need to be present in the input nodes of the BuildRequest.
Change-Id: Ie9b957805e42f766002581adc6182a6543c5333b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11802
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
… and add an introductionary page at the root.
Change-Id: I89ce69a4221c332b7c0d99ac26b4e26bfd248341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11806
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
We need some shared queue, preventing the same fetches/builds from
getting triggered multiple times unnecessarily.
Change-Id: I7c4a3c66db558f5cccd66865b170242b758e3e02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11800
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This allows explicitly opting in to get DEBUG-level log lines, by
setting RUST_LOG.
It currently also causes traces to be emitted in all cases, so we might
do some runtime filtering there too, as discussed in cl/11791.
Change-Id: I2865bb06a62465836d63196422f5f734f7165386
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11801
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
This gives some better feedback something is going on, gives a (named)
root for all the progress children we're drawing, and also counts the
time we are in eval.
Change-Id: Ibe81dcebf0a2b59bb0680da62e206eb5270f9d3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11798
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
We immediately reported "Building", even though then populated necessary
inputs, which looked a bit odd. Make it clear we're still waiting, and
update the spinner message once we have all inputs we were waiting for.
In the future, we might want to have separate spans for this, so the
timer gets reset, but that's something for later.
Change-Id: Ic22c9a906d0e7e7179c5ee328162401261efc224
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11799
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This should also report progress on fetches which we couldn't delay
until actually having to IO into them, like `builtins.fetchurl` calls
without a upfront-provided hash.
While at it, upgrade the progress spinners to progress bars, which
increment if we know the size of the fetch.
Change-Id: Ic3f332286d8bc2177f3d994ba25b165728d4b702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11797
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
In `store_path_to_node`, in case we need to build or fetch something,
render a progress bar, using the spinner for now.
We can upgrade this to a progress *bar* later.
Change-Id: I4a7cf5ef8f639076f176af9b39d276be3f37c8ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11793
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
nixpkgs calls <nix/fetchurl.nix> during nixpkgs bootstrap.
This produces a fake derivation with system = builtin
and builder = builtin:fetchurl, and needs to download files from the
internet.
At the end of the Derivation construction, if we have such a derivation,
also synthesize a `Fetch` struct, which we add to the known fetch paths.
This will then cause these fetches to be picked up like all other
fetches in TvixStoreIO.
Change-Id: I72cbca4f85da106b25eda97693a6a6e59911cd57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10975
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
The magic <nix/fetchurl.nix> derivation can cause two other types of
fetch to happen, one that unpacks NAR files, and another one that puts a
file as an executable at the store path root.
This adds the necessary enum type and path calculation logic for it to
the fetcher code.
It also adds code to do the actual NAR fetching. The executable case is
still stubbed out.
Change-Id: I79103fd58c7e22ad7fde34efa5e2d89cad7d5a0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11790
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Move the part asking a fetch for its store_path() to the place where
this function is defined, and add some more test cases.
Change-Id: I96f326d0d56aa5835f23274b8cd1b1afe3724153
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11789
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This allows giving more self-speaking names, as well as documenting each
field individually.
Change-Id: Ide164d684b7f819aac279cc8e657c02fc24d093f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11786
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
These are all derived from the FOD representation of the ATerm
serialization, so no new code necessary to calculate these hashes
correctly.
Change-Id: I8c5dacd2a8de3d5042c9dd56910511c19e1b9b31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11788
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This wraps ingest_nar, but also keeps track of the number of bytes read,
and calculates the sha256 digest of it.
Make use of it in the NixHTTPPathInfoService, where this code is coming
from.
Change-Id: I9c54e93d3ec8ed9ede87aed43e04d114fb06897b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This prevents the following statements from looping endlessly:
```
builtins.split "(.*)" ""
builtins.split "([abc]*)" "abc"
builtins.split "(.*)" "abc"
builtins.split ".*" ""
```
Cover these (and some more examples) in the test suite.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Ibd339f971e0f4e3e5c229816e2be5a8e3836fec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11743
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Introduce a new common crate that contains tracing boilerplate which then
can be used in the cli, tvix-store and tvix-build crates.
It has otlp as an optional feature, which is currently only used by
tvix-store.
Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
I tried moving web/tvixbolt to tvix/tvixbolt, and adding it to the cargo
workspace.
I then made `crates =` in `default.nix` a function accepting `pkgs`
(so we can pass in another nixpkgs for some invocations), and then
constructed another nixpkgs and crates instance like this:
```
pkgs-wasm = (import pkgs.path {
localSystem = localSystem;
crossSystem = {
system = localSystem;
rustc.config = "wasm32-unknown-unknown";
};
}
crates-wasm = (crates pkgs-wasm);
tvixbolt-test = crates-wasm.workspaceMembers.tvixbolt.build;
```
… leading to the architecture build failures described in the TODO.
Change-Id: I32112d75f8c098d9810ca52b2d07cd76fae8d8d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11777
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Fix some typos found while reading various documents, mostly those
relating to the castore.
Here is a summary of the edits.
- fix broken link between documents in the store and castore directories
- clarify expression in castore's data model document that indicates
that the *name* of each child node of a directory must be unique
across all three lists of children
- add missing closing parenthesis in castore's data model document
- replace "how" with "what" in the phrase "unclear how a ... would even
look like" in castore's why-not-git-trees document
- remove unnecessary articles in castore's blobstore chunking document
- add missing "y" to "optionall" in eval's compilation of bindings
document
Change-Id: I1997ea91bb4e9c40abcd81e0cde9405968580ba6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11763
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
* agenix has not been updated (https://github.com/ryantm/agenix/pull/241).
* wasm-bindgen bumped to 0.2.92 in Rust WASM projects
* 3p/lisp: port lispPackages from stable channel
The Lisp package set we are using (`pkgs.lispPackages`) is the "old
old" package set, whereas we were supposed to have been using
`pkgs.lispPackages_new` (which is the "old new" package set).
Either way we missed that train, and now there's a "new new" package
set, but with a twist: Lisp packages in nixpkgs are now tied to
their compilers, so the most generic way to access them seems to be
from `pkgs.sbclPackages`.
Switching to the packages from the "new new" package set doesn't
work: Lots of stuff stops building if we just switch the sources
over, and not everything is trivially fixable.
For now we stay on the lispPackages from the stable channel. We need
to look into the migration later.
Or rewrite panettone.
* tvix: update generated protobuf files
* 3p/nixpkgs: pick trunk from stable channel; newer versions try to
read files and do network I/O during build, but don't print enough
details in error messages to figure out why.
* 3p/overlays: remove tdlib override (nixpkgs is currently new enough)
* 3p/overlays: override telega.el sources while updates are lagging in
nixpkgs
* users/flokli/ipu6-softisp: update firmware paths, which NixOS now
stores zstd-compressed.
Change-Id: I5a7a6c8b5d0688461bca92b9e6d654356d3a1cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11711
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
We don't use this in tvix-glue currently. A previous version of cl/11747
did, and I forgot to remove it afterwards.
Change-Id: I6191ffc60450f3e79f22a3aca55246e3956ee9cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11756
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
This adds the tracing-indicatif crate, and configures it as a layer in
our tracing_subscriber pipeline to emit progress for every span that's
configured so.
It also moves from using std::io::stderr to write logs to using their
writer, to avoid clobbering output.
Progress bar styles are defined in a lazy_static, moving this into a
general tracing is left for later.
This adds some usage of this to the `imports` and `copy` commands.
The output can still be improved a bit - we should probably split each
task up into a smaller (instrumented) helper functions, so we can create
a progress bar for each task.
Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
The derivation name for the check will now be calculated from the hash
of all Cargo related files (including in subdirs), this makes it less
likely for the drv to be cached and for CI to miss an outdated
Cargo.lock.
Change-Id: I900e9355be3f8a9d6f01162e8ef0da4d8901af30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11753
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Context: https://github.com/la10736/rstest/issues/256
Cargo will now cause a rebuild whenever a new test case file is added.
Previsouly running `cargo test` after adding a new test case resulted in
that case not being picked up and ignored.
Change-Id: Ibfc420b5bfe3f3ee41d3ebd3fb9d248819fa6ed9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
This one's relatively simple - we just check if the store path exists,
and if it does we make a new contextful string containing the store path
as its only context element.
Automatic testing seems tricky for this (I think?) so I tested it
manually:
tvix-repl> builtins.storePath /nix/store/yn46i4xx5alh7gs6fpkxk430i34rp2q9-hello-2.12.1
=> "/nix/store/yn46i4xx5alh7gs6fpkxk430i34rp2q9-hello-2.12.1" :: string
Change-Id: I8a0d9726e4102ab872c53c2419679c2c855a5a18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11696
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Write down some of the thoughts after brainstorming with yuka.
Change-Id: I01c94474dc643b8c4993db80e50d3ec65f5c17f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11749
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Closes: https://b.tvl.fyi/issues/401
With this change all crate features (and their combinations) will be built and
tested in CI.
From now on, when adding/removing a Cargo feature for a crate,
you will want to add it to the features power set that gets tested in CI.
For each crate there's a default.nix with a `mkFeaturePowerset` invocation,
modify the list to include/remove the feature.
Note that you don't want to add "collection" features,
such as `fs` for tvix-[ca]store or `default`.
Change-Id: I966dde1413d057770787da3296cce9c1924570e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11717
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Makes the following build:
`cargo test --no-default-features --features async`
`cargo test --no-default-features --features wire`
Change-Id: I47ba0c944f08895f67ed3b861706ef2e4ba384b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11739
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Add a command, :p, to evaluate an expression and recursively print the
result, as if `--strict` had been passed on the command line.
Demonstration of this working:
❯ cargo r --bin tvix
Finished dev [unoptimized + debuginfo] target(s) in 0.27s
Running `target/debug/tvix`
tvix-repl> { x = (x: x) 1; }
=> { x = <CODE>; } :: set
tvix-repl> :p { x = (x: x) 1; }
=> { x = 1; } :: set
Change-Id: I1a81d7481160c30d2a4483c6308e25fa45f2dfdf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11738
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Add a command to display help for the REPL, which can be either :? or
:h.
Change-Id: Ifdfd8c31130ca5afcde05a4c4276b768eb54c06f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11737
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Prepare for introducing additional REPL commands by splitting out
the *parsing* of the repl commands, which returns a new ReplCommand
type, from the actual *handling* of the commands.
Change-Id: If81a53c1e2d90204d26ce3bb2ea9eebf7bb3fd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11734
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In preparation for adding some new functionality to the tvix REPL, break
it out into its own module.
Change-Id: I4fb78320e92562e3474a3724536cb22c1d893e57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11733
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Transparently support multiline input in the Tvix REPL, by handling the
UnexpectedEOF error returned by the parser and using it to progressively
build up an input expr over multiple iterations of the REPL's outer
loop.
This works quite nicely:
❯ cargo r --bin tvix
Compiling tvix-cli v0.1.0 (/home/aspen/code/depot/tvix/cli)
Finished dev [unoptimized + debuginfo] target(s) in 1.72s
Running `target/debug/tvix`
tvix-repl> { foo
> =
> 1;
> }
=> { foo = 1; } :: set
<press up arrow>
tvix-repl> { foo
=
1;
}
=> { foo = 1; } :: set
Change-Id: Ib0ed4766b13e8231d696cdc27281ac158e20a777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11732
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This fixes the following command:
'cargo test --no-default-features --features nix_tests'
Change-Id: I9883c39e1e428c72a0e7e0b75a73c8ed734abd3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11740
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
`cargo test --no-default-features` fails, if we don't conditionalize
this on the `arbitrary` feature too.
Change-Id: I81a277810119fed0cfc37c942c422f731aa14b2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11726
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
`Evaluation::new_impure()` would require the test to be impure, but
there's nothing in this test specifically requiring us to make use of
impure features.
Change-Id: Idb24981195d1a94f51053ae04403eb5f0e27f3d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11725
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This uses StdIO, which is only available when the `impure` feature is
enabled.
Change-Id: I039b1f45f6619dd099fa943e58322ff521482dfa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Gate this behind the nix_tests feature flag (which wasn't applied
consistently).
Also, at least right now all of these use StdIO internally, either by
creating it on their own, or through the `eval_test` helper.
Once we have some proper classification system for tests we can probably
split this up further, but for now only run them if the impure feature
is enabled.
Change-Id: I3236cf09b3391585df99073367c4e4832c5e7898
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11723
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
These VM requests are only emitted by code gated behind the impure feature.
To prevent warning from popping up when building without default
features, allow these to be unused if `impure` is not enabled.
Change-Id: Id871a5215e9a0f09aa78edecdd111369ee7ffe34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11722
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
These use StdIO, which is only available if the impure feature is
enabled.
Change-Id: I18d8e191a7eba6ba5bd59f43631973eaa796c7bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11721
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
These are only needed when building with the impure feature enabled.
This removes some warnings when building with --no-default-features.
Change-Id: I3139d9133d4846aeb1b1b5f3830c0d078d047292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11720
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
If building tvix-build with the tonic-reflection feature, it needs
to import `tvix_castore::proto::FILE_DESCRIPTOR_SET`, which is only
available if tvix-castore is built with the `tonic-reflection` feature.
Change-Id: I355b4c5b4c1333d5cc56335de47ad5d2f1db6337
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11716
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
These tests only interact with the FUSE layer, and
import super::fuse to do its work.
However, this only works if the `fuse` feature is enabled, which we
don't do if we enable the `virtiofs` feature only, causing the tests
to fail:
```
❯ cargo test --no-default-features --features virtiofs
Compiling tvix-castore v0.1.0 (/home/flokli/dev/nixos/code.tvl.fyi-submit2/tvix/castore)
error[E0432]: unresolved import `super::fuse`
--> castore/src/fs/tests.rs:14:13
|
14 | use super::{fuse::FuseDaemon, TvixStoreFs};
| ^^^^ could not find `fuse` in `super`
```
We move src/fs/tests.rs to src/fs/fuse/tests.rs
(and src/fs/fuse.rs to src/fs/fuse/mod.rs) to better structure this,
which will automatically cause both tests and code to only be built if
we have the `fuse` feature enabled.
Change-Id: I8fbbad3e4457e326bdfd171aa5c43d25d3187b5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Previously changing any file (including default.nix or README.md) would
cause Nix to compute a different derivation path, resulting in needing
to (very often redundantly) rebuild the crate/package. With this change
you can expect less rebuilds.
Crate2nix currently does its own insufficient src filtering and it does
not expose an API to override the filtering, so instead I created a
function that we can use to override the src to have stricter filtering.
I implemented the filtering for all of the workspace members, if you
want to modify any of them, please read the lib.fileset docs
https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset
Change-Id: I7ab5a0064a76938d14f7f65801be9f3a5c3bad04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11714
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
cl/11712 simultaneously introduced this check and made it unnecessary,
since NixString::context should never return `Some` for empty contexts
now.
Change-Id: I41a655ff33910e8326cbb7d7526eb91bd19e9585
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11713
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Both `Some(NixContext::new())` and `None` represent empty contexts,
but the former trips up `NixString::has_context`, and seems likely
to trip up other things.
We could hide the difference in the accessors, but we don't really
*want* the distinction to exist, since heap-allocating a null value
is pretty much always a mistake.
Change-Id: Ie84d26fb0d4b59e68354891ba13bde3bae40ab6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11712
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In places where we want to extend context with that from another
NixString, use take_context() to split it off, then call .extend(),
making use of IntoIterator to avoid a bunch of clones.
Change-Id: I2460141a3ed776c64c36132b2203b6a1d710b922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11705
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
XmlEmitter gains a NixContext field, and `write_typed_value` extends it
with all context elements present in the passed value.
Once all serialization is done, a into_context() function returns the
collected context, so we can construct a NixString with context.
Tests for this live in tvix-glue, as we use builtins.derivation, which
is not present in the tvix-eval crate.
Fixes b/398.
Change-Id: I85feaaa17b753885f8a017a54e419ec4e602af21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11704
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
This is a slightly less annoying version of `join`, which does not
consume self. It's more consistent with HashSet::extend().
Change-Id: Ifd0872da36fe8e7b2aa6948674cb8e4023abe9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11703
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
In order to be compatible with the nix XML generator, it’s easier to
generate the XML directly, instead of going through a library which we
have to bend to do what we need.
Removes dependency on `xml-rs`, which came with a full XML parser that
we didn’t use. Only takes a tiny bit of code for the XML escaping,
somewhat simplified.
I add a little escaping value, to make sure we have the same behaviour
as nix proper.
Interestingly enough, we never need to escape XML attribute names,
because the `builtins.toXML` format encodes user-defined values as
attribute keys only. So we only escape attribute values.
Fixes: https://b.tvl.fyi/issues/399
Change-Id: If4d407d324864b3bb9aa3160e2ec6889f7727127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11697
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Currently all blobs are uploaded serially when ingesting NARs. If a NAR
contains many, small blobs, ingestion may become slow if there is a lot
of round-trip latency to the blob service.
This makes the NAR ingester use the ConcurrentBlobUploader which allows
for buffering small blobs in memory so they can be uploaded concurrently
to the blob service without blocking further deserialization.
Change-Id: I093a73770232df12d9a11e5d901b99c08505c3cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11694
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The archive ingester has a mechanism for concurrently uploading small
blobs to the blob service in order to hide round trip latency with the
blob service when ingesting many small blobs.
Other ingestion sources like NARs also need a similar mechanism, this
extracts the concurrent blob uploading mechanism into its own struct to
make it more reusable.
Change-Id: I05020419ff4b9ad5829fbfb5cd08d36db983b8c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11693
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Since cl/…, a PathInfoService doesn't need to implement `calculate_nar`
anymore, so most of them don't actually have a handle to a
{Blob,Directory}Service anymore.
This means, we can simplify the construction of them for test cases
a lot.
Change-Id: I100e9e1c9b00a049b4d6136c57aad4cdb04461c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11691
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Have this return the same values as builtins.unsafeGetAttrsPos, rather
than returning a CatchableErrorKind, which crashes the CLI if it bubbles
up.
The environment we're in doesn't allow emitting a warning, as we don't
have `co` in scope, but that's probably OK as a stopgap solution.
Alternative to cl/11665.
Change-Id: I5b2c2530842547c93b6533ed9601ee9b2923b1bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11685
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds rough notes documenting the history of the Nix
daemon protocol, how logging works as well as begins
documenting inputs and outputs for all operations.
Change-Id: Id24a9a658c3e4e7c350ca1e4622f63ed96ccef5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11594
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Using remove_node messed up the extraction of nodes from the graph. Use
into_nodes_edges() instead, to remove the nodes without cloning.
Change-Id: Id76c7935d082d6f26192cc3cd490483594f1d1e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11684
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Drop the (unused) feature flags here, and get rid of some crate
dependencies.
Change-Id: Id64852b498725467d56abb12b548301cfba6a760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11678
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This has already been dropped for tvix-store, drop it here as well.
Change-Id: Ib3aa37bbe9fd4c498b5ba1213f7d922d0c64ffc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11677
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This gives multi-listener support, as well as listening on named socket
FDs.
Change-Id: I92b441f854e2faf80074463d0ca6bdc23cbd890a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11663
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
This aligns the tonic version we use directly in Tvix to the one pulled
in by the OTLP stack.
Change-Id: I658528c8dabb7cd6948f1207ddcdef1984e82037
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11666
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Make some space for the rust implementation.
Change-Id: I924dc1657be10abe5a11951c3b9de50bae06db19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11662
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
This allows querying two PathInfoService implementations sequentially,
and inserts into the "near" one if it's not there yet.
There is no negative cache, and put / listing is not implemented (for
now).
Change-Id: I24c3d0e0c3c2f0524a6cc7b2f3cbc33eb20cf92b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11636
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We can just use the `BoxStream` directly, or a `once` with the single
`Directory`.
In the recursive case, we also did not properly close the channel after
the first error.
Change-Id: Ifad56d307fc7861107b6d3cffd28d35631d526e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11635
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Necessary to directly use this in the GRPC DirectoryService wrapper
directly.
Change-Id: Ic6a0038a40dc30071d145af5035345fcd93288ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11634
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Use try_stream! rather than stream!, and a bit more map_err and ok_err
to make things a bit more concise. Once we have proper error types here,
and impl Froms, a lot of the error mapping would disappear entirely.
Change-Id: I5240a6b0ff7818b94c151322774242b2c142e33b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11633
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Replace the loop manually driving the iterator with a for … in, and some
of the match with ok_or_else.
Change-Id: I6d7b3ef1bf1c7aa128bd6adef09390b54f79479e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
… in Cargo.toml.
This gets an imperative `cargo clippy` run to pick up that config,
so `-A clippy::blocks_in_conditions` doesn't need to be explicitly
specified anymore.
Change-Id: I32b6cc50c77c22cba0d816d0db508c2f94b2c383
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11659
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Add instrumentation to the get() and put() implementations of all
PathInfoService.
Use the nixbase32 representation of the digest, not the base64 one.
Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We don't produce these erorrs anymore, no need to provide a conversion
to it.
Change-Id: I37933e436ad15c5d90b3ac270c4ef5742980513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11614
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
We don't want to block here, and this also means there's no poisoning to
deal with.
Change-Id: Ic375571970c48beace0005ae2c012135086a4d67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This one doesn't require us to deal with poisoning, is upgradeable and
the right thing to use when locking access to data, not IO resources.
Change-Id: I78634953a73404500d28f51f1d93a87e215c8149
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
