Commit graph

2799 commits

Author SHA1 Message Date
edef
f3c80c10b9 feat(tvix/tools/narinfo2parquet): preserve the deriver
Change-Id: Idbada585b87eef81cad5d40cb7592a7890704695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12037
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-27 21:31:18 +00:00
Matthew Tromp
0b987a509f fix(tvix/eval): don't bubble up io errors from path_exists
path_exists was returning an error when certain common IO errors were
encountered. e.g. in the path "/dev/null/.", path_exists would throw
an error because the underlying call to Path::try_exists threw an
error because null isn't a directory. But if null isn't a directory,
then the path is invalid, so this should really be returning
false. That's what nix's behavior is and that's what makes sense.

The trait function isn't being changed because some other
implementers (e.g. tvix_store_io) have actual errors they can throw.

Fixes: b/411
Change-Id: I9e810e7a198bffe61365697c6d3d7e71f264c20b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12042
Tested-by: BuildkiteCI
Autosubmit: chickadee <matthewktromp@gmail.com>
Reviewed-by: aspen <root@gws.fyi>
2024-07-27 21:29:46 +00:00
Aspen Smith
4769d047b2 fix(tvix/cli): Make :q actually quit
whoops

Change-Id: I5f24163c276992c7858b038ae2bb636da75f3f91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12043
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2024-07-27 21:19:43 +00:00
Florian Klink
c8f92c6e9f refactor(tvix/nix-compat): introduce CaHash::algo_str
This is the `{fixed,fixed:r,text}:{sha*,md5}` prefix used in various
string representations.

Factor that code out, and use it in the two places it can be used in.

Change-Id: Ic9555fa9e1884198d435e55c7f630b8d3ba2a032
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12041
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-27 10:53:56 +00:00
Yureka
5114504b67 test(tvix/castore/dirsvc): check for a pitfall with deduplicated dirs
Change-Id: I3cff2c2e8b2c8a2fd8d8074647d0d99a1db8e693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12034
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-25 09:52:29 +00:00
Yureka
6a988a1598 fix(tvix/castore/GRPCDirectorySvc): fix a bug in the get_recursive fn
When retrieving a closure with get_recursive, the following could happen in the GRPC client:
- The first reference to the deduplicated directory is added to expected_directory_digests
- The deduplicated directory is obtained removed from expected_directory_digests
- The second reference to the deduplicated directory is added to expected_directory_digests
- The deduplicated directory has already been sent, but is still in the
  expected_directory_digests. It looks to the GRPC client like the
  closure is incomplete and the stream ended prematurely.

Change-Id: Ic62bca12e7f8fb85af5fa4dacd199f0f3b8eea8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12033
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-25 09:52:29 +00:00
Yureka
e97202e54a refactor(directoryservice): get rid of NaiveSeeker helper
Use the ChunkedReader in CombinedBlobService instead which also supports seeking.

Change-Id: I681331a80763172c27e55362b7044fe81aaa323b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12031
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-23 20:53:08 +00:00
Florian Klink
9385b3bcfb fix(tvix/boot/tests): check nar-bridge being fully up
Ensure nar-bridge is healthy before connecting to it, don't just check
for the unix socket to be present.

We don't have a proper /health endpoint yet, but nix-cache-info works
fine for now.

Change-Id: I22df2c3b7bffcf52dbd3d00f3ba5382dc06ab03d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12030
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-23 14:47:03 +00:00
Florian Klink
636647ce03 fix(tvix/boot/tests): use grpc health check
Ensure the service is healthy before connecting to it, don't just check
for the unix socket to be present.

Change-Id: If6501828677c247910d91f35b860960802084691
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12029
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-07-23 14:47:03 +00:00
Yureka
94a0e21e68 fix(tvix/directoryservice): clarify get_recursive not found
Change-Id: I47f21b4b7db9304eadb04094d41bf04d443fcc3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12025
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-23 14:04:23 +00:00
Florian Klink
b9aa6456e2 docs(tvix/docs): add TODO for protobuf
Change-Id: Iacf4ef517885bdd6d6366446124863d9a138b12b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12024
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-07-23 13:24:42 +00:00
Ilan Joselevich
9f10a71ec5 feat(tvix/store): add redb PathInfoService
This provides a PathInfoService implementation using redb
(https://github.com/cberner/redb) as the underlying storage engine.

Both an in-memory variant, as well as a filesystem one is provided,
similar how it's done with the sled implementation.

Supersedes: https://cl.tvl.fyi/c/depot/+/11692
Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc
Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-22 18:49:04 +00:00
Yureka
dbe698042d test(tvix/store): add xp-store-composition to feature powerset
Change-Id: Ibbd9a7585ec2f70c8f0f4d25ad858aa5ebc031dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12023
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 16:30:08 +00:00
Yureka
2547ff2cf6 feat(tvix/store): add grpc healthcheck service to daemon
Change-Id: Ib95fc9352a45d54f9a16c8841c7e8f7cbeeaee8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12019
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 15:57:30 +00:00
Yureka
39c8b6dece fix(store): add toml dependency for feature xp-store-composition
Fixes 'use of undeclared crate or module ' with --features xp-store-composition

Change-Id: I44dce86e656094d180b91a00f385f685d21f3fbd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12021
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 15:56:59 +00:00
Yureka
14a4b4cbc3 feat(tvix): add experimental-store-composition option
Change-Id: I61661fbb0e77ce3c00c2a467dfabdf3fc77d8575
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12011
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 13:36:08 +00:00
Yureka
67335c41b7 refactor(tvix): move service addrs into shared clap struct
Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 13:36:08 +00:00
Yureka
6774d9c59c chore(tvix/tracing): switch tracing-opentelemetry from git to release
Change-Id: Ib830c8b642496a6cdf3d4b9093f0343f5fb80622
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12018
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 13:36:08 +00:00
Yureka
8342f70c60 feat(tvix/composition): improve error message for unknown tag
Change-Id: I61a31488de17725ae4311f5f5bf8e02edb752cf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12017
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-22 13:36:08 +00:00
sinavir
5d82e1e62f fix(tvix/store): Fix narinfo compression selection
Parsing of the narinfo file sets the compression field to None instead
of Some("none"). The mapping selecting the decompression reader expected
the former in //tvix/store/src/pathinfoservice/nix_http.rs.

Change-Id: I254a825b88a4016aab087446bdc0c7b6286de40c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12007
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 22:11:20 +00:00
Florian Klink
a3194e9280 feat(tvix/nix-compat): add SigningKey, NARInfo::add_signature
This adds a generic `SigningKey` struct that can be used to sign
NARInfos with signers.

It also includes tooling to parse keypairs from bytes generated by Nix,
returning a specialized ed25519_dalek variant.

Change-Id: Ic9780c370939af54e7177c93cde3321adf189fc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12014
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 21:41:54 +00:00
Florian Klink
05b4e805ee refactor(tvix/nix-compat): rename PubKey to VerifyingKey
Align these with the way it's called in the ed25519 crates.

Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-07-21 21:41:54 +00:00
Florian Klink
62184ee35a docs(tvix): document the builder API
This documents some thoughts and goals of the Tvix Build protocol, and
how it is possible to express Nix builds with it.

Additionally, it explains a proposed design for reference scanning.

Change-Id: I4b1f3feb2278e3c7ce06de831eb8eb1715cba1c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12012
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-21 21:35:22 +00:00
Yureka
e408783bac test(tvix/composition): add recursion and concurrent tests
Change-Id: Ic10773b08d940b45c3067bd514c3080cadac9606
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12003
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 21:18:17 +00:00
Yureka
eea9f3d9ee docs(tvix/composition): add comment about stack
Change-Id: I3888b5034c53728e2f9cfe24213f8854eb38bfe1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12005
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 18:37:50 +00:00
Yureka
76394daca3 feat(tvix/composition): downcast boxed error if possible
We still have the unique store name to identify which instantiation caused the error. For recursion errors, the full chain is still retained inside the CompositionError.

Change-Id: Iaddcece445a5df331e578d7c69d710db3d5f8dcd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12002
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 17:03:27 +00:00
Yureka
a805a80b48 fix(tvix/composition): include typeid in recursion check
Change-Id: Icc279d41a4980d4b57acbb4243bbd509039b753f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12000
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 13:20:00 +00:00
Florian Klink
ba38883a4f fix(tvix/nar-bridge): remove leftover debug statement
Change-Id: Ie6fb822e1a4b76f2f6aef15bbe846e0a89c644fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11999
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-07-21 11:26:13 +00:00
Florian Klink
98f59cb828 feat(tvix/nar-bridge): change default port
Having `tvix-store daemon` and `nar-bridge` listen on the same port by
default is silly.
Use the same port that nar-bridge-go was using previously.

Change-Id: Id374a12e52ba9dac4318e43d0ca0853866eadabc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11998
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-21 11:26:13 +00:00
Yureka
ca8e2b9fbf chore(tvix): upgrade to tonic 0.12 / hyper 1.0
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 11:01:38 +00:00
Simon Hauser
1515a970be feat(tvix/tracing): http propagation for axum
It introduces a new accept_trace function for axum0.7 which can be used
to accept a header trace from a received request. This function can be
used for tonic 0.12 once that version is released, and the specific
`accept_trace` function within `tvix_tracing::propagate::tonic` can then
be removed.

This also integrates http propagation into the nar_bridge crate.

Change-Id: I46dcc797d494bb3977c2633753e7060d88d29129
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11925
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 05:45:19 +00:00
Florian Klink
fdc0cf0c94 refactor(tvix/boot/tests): use nar-bridge-rs
This switches the boot tests from the golang implementation to the rust
one.

Change-Id: Ide2a47aebe40c172077147c05bb6dacd74ba6b1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11997
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-21 00:06:23 +00:00
Florian Klink
d136f2c881 fix(tvix/nar-bridge): fix root node decoding and validation
This got broken while moving things around. We need to parse the
b64-decoded bytes.

Since we're now validating the root node, we also need to rename the
root node to get past the node name validation.

There probably should be some tests for this.

Co-Authored-By: sinavir@sinavir.fr
Change-Id: I8f24a4a0ac107b1ea5b94c0e0ed872a34eb7b587
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11996
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-21 00:06:23 +00:00
Florian Klink
fca8462530 fix(tvix): fix outputHashes calculation for crates with versions
In case two crate versions are present, the key in the Cargo.nix file
includes the version number too.

To be able to set a specific hash for "tracing-opentelemetry 0.25.0" for
example, this needs to account for keys with the version included.

Access `crateName`, `version` and `src.outputHash` individually.

Change-Id: Ib9800691a445ac403ff646cb32e85a7a4cbef9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11994
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-07-20 22:17:15 +00:00
Yureka
8b77c7fcd7 refactor(tvix/store): use composition in tvix_store crate
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-20 19:37:27 +00:00
Florian Klink
6180a7cecf feat(tvix/nar-bridge): implement PUT $outhash.narinfo
This adds support to upload NARInfo files. We lookup the root node from
the LRU cache, rename it appropriately and then put it into the
PathInfoService.

Change-Id: I5479032b51cd855363bc016dee63cf84b3304a36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11988
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:38:58 +00:00
Florian Klink
5bd48de418 feat(tvix/nar-bridge): add 404 handler for GET/HEAD /nar/…
We currently send 405, and that confuses `nix copy`.
Send a 404 for now, and add a futurework, as we can actually at least do
something more meaningful in case we still have that nar hash in our
LRU, which would avoid some unnecessary uploads in some cases.

Change-Id: If625e9bd0fd6506cb73b88962d889aa08315fcea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11987
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:38:58 +00:00
Florian Klink
5d906054da feat(tvix/nar-bridge): support uploading NAR files
This ingests NAR files into the {Blob,Directory}Service, which are
already part of the AppState.

As we then need to correlate the root node to the uploaded PathInfo, we
need to keep a (short-lived) lookup table from NARHash to root node
around. We insert it into a `LruCache` after the NAR is uploaded, and
use `peek()` to do the lookup, which doesn't update the LRU list.

Change-Id: I48a4c6246bacf76559c5a4ccad2a0bc25c1b7900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11986
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:23:18 +00:00
Florian Klink
861cc1f341 feat(tvix/nar-bridge): init
This adds an implementation of nar-bridge in Rust.
Currently, only the GET parts are implemented.

Contrary to the Go variant, this doesn't try to keep a mapping from nar
hashes to root node in memory, it simply encodes the root node itself
(stripped by its basename) into the URL.

This pulls in a more recent version of axum than what we use in
tonic, causing two versions of http and hyper, however dealing with
`Body::from_stream` in axum 0.6 is much more annoying, and
https://github.com/hyperium/tonic/pull/1740 suggests this will be fixed
soon.

Change-Id: Ia4c2dbda7cd3fdbe47a75f3e33544d19eac6e44e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11898
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-20 16:53:23 +00:00
Florian Klink
0244ae6eaf feat(tvix/castore/protos): return root node after validation
This allows avoiding a `.node.unwrap()`` after validation.

Change-Id: Ieef1ffebab16cdca94c979ca6831a7ab4f6007da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11989
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-20 16:35:07 +00:00
Florian Klink
c64c028366 fix(tvix/store): add missing nar_info::ca::Hash::TextSha256
Also reorder to how it's ordered in the .proto file.

Change-Id: I87c422feac4d12f6a84f99aa889f9c524a9878b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11985
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 16:35:07 +00:00
Brian Olsen
18da8a205e docs: More documentation of nix daemon protocol
Uses `Set` to mark that the given list is actually a set.
The format on stream is no different from `List` but is nice to
know.

Best effort document string formats.

Documents default values for fields not used in old protocol
versions.

Document handshake, AddMultipleToStore format, Export format and
Import paths format.

Change-Id: I6cbe4c7de3770cc5caafc26229f99a07353ee3e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11984
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Brian Olsen <me@griff.name>
2024-07-20 15:33:04 +00:00
Yureka
4801d6bf85 feat(tvix): different service types in one composition
This will be necessary for the PathInfoService composition, as some
PathInfoService implementations require a BlobService & DirectoryService
to ingest into.

Using the Extend trait for creating compositions allows extending the same
composition with configs of various types e.g. BlobStore, DirectoryStore
Generics are moved from the Composition struct to the functions.The storage of
the InstantiatonStates uses the TypeId in the key and a Box<dyn Any> in the
value, which is downcasted to InstantiatonState<T>.

Change-Id: I2af11f26c535029adfb1c62905e0e7c4aaed7b51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11980
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
2024-07-19 15:13:37 +00:00
Yureka
7ccdf6dad5 test(tvix/castore/ObjectStoreBlobService): test correct chunking
Change-Id: I4a8f5cda2705560a6d52a2eb2be0dde35b784c8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11982
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-19 12:57:41 +00:00
Yureka
3c160731ed fix(tvix/castore/blob/object_store): empty chunk list for single chunk
Change-Id: Ie3609ac6184ecead11c4e03073433c26ce129848
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11981
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-19 10:55:47 +00:00
Ryan Lahfa
f0b0a6572f fix(tvix/nix-compat): use buf.filled() to track read bytes
We were wrongly using `buf.initialized()` which contains more than the
filled portion to compute the number of bytes read during a poll call.

This made us go into the trailer reading state too early and finally
failing due to invalid trailer data.

Fixes b/405.

Co-authored-by: Florian Klink <flokli@flokli.de>
Change-Id: I66ba6e2116389e6b97305f85c4e0922195000e45
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11978
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-07-18 22:13:44 +00:00
Yureka
168e4fda59 refactor(tvix): use composition & registry for from_addr
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
2024-07-18 19:19:26 +00:00
Yureka
79317be214 feat(tvix/castore): add sled to composition registry
Change-Id: I03fa8dfabcee14c5f657380f86bb1a7aa00e08ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11977
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-18 19:19:10 +00:00
Yureka
b8415a94d9 feat(tvix/composition): allow creating a blank CompositionContext
this is useful when oneshot-instantiating a store from a single config

Change-Id: I08538fdee1d0bb26b3ae2da7d3b2339b2e93bc0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11975
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-18 18:47:42 +00:00
Yureka
1a6b6e3ef3 feat(tvix/castore): add composition module
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748
Signed-off-by: Yureka <tvl@yuka.dev>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-18 10:42:11 +00:00
Ilan Joselevich
64fd1d3e56 chore(tvix/tools/weave): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: I79b062eddb13f59c4656a62735df1e527fc91a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11970
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 16:47:56 +00:00
Ilan Joselevich
52b8cf8d1f chore(tvix/tools/turbofetch): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: I48cabe514e0cf1dd1f00870bcb7958a2a49663d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11969
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-09 16:47:56 +00:00
Ilan Joselevich
716f5624fe chore(tvix/tools/crunch-v2): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: I90245bad27f6fee01455557bbccc865301a4f1a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11968
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 16:47:56 +00:00
Ilan Joselevich
f52b142035 chore(tvix/tools/narinfo2parquet): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: Id5225e73ec5894bc324f363660472a7efdd803cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11967
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-09 16:47:56 +00:00
Ilan Joselevich
4f7b19701c fix(tvix/{store,castore,build}): Compile tonic w/o config in build.rs
Previously we had to make a mutable Config instance and set bytes and
other values in it because they were not exposed to the builder pattern
(https://github.com/hyperium/tonic/issues/908) but now they are, so we
just set them through the builder.

Change-Id: I8904c6b93f09173b56586024b1ced59d622bce66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11966
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 13:24:22 +00:00
Florian Klink
d17c3d96b6 refactor(tvix): point SSL_CERT_FILE to /dev/null
reqwest wants to be able to read a file of trust roots when constructed,
but as it doesn't actually do any HTTPS connections inside the nix
build, an empty list of trust roots is totally sufficient.

Thankfully /dev/null provides such a file.

Change-Id: I9bd1619b2c9f8ff2a6640d2ac410d4de5b20c2ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-07-07 15:11:29 +00:00
Aspen Smith
89d204d295 feat(tvix/eval): Bump imbl to 3.0.0
Change-Id: I88053fa2df35dc486c51d015f4b2156541c07af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11962
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 15:05:27 +00:00
Aspen Smith
8821746d6c fix(tvix/repl): Share globals and sourcemap across evaluations
Now that we can bind (potentially lazy, potentially lambda-containing)
values in the REPL and then reference them in subsequent evaluations,
it's important that the values to which we construct shared references
are shared across those subsequent evaluations - otherwise, we get
panics due to unknown source map locations, or dropped weak references
to globals.

This change assigns both the globals and the source map as fields on the
Repl after the first evaluation, and then passes those in (to the
EvaluationBuilder) on subsequent evaluations.

On the EvaluationBuilder side, there's some panicking introduced - this
is intentional, as my intent is for the builder to be configured
statically enough that panicking is the best way to report errors
here (it's always a bug to misconfigure an Evaluation, and we'd never
want to handle it dynamically).

Change-Id: I37225697235c22b683ca48a17d30fa8fedd12d1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11960
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-07-07 15:04:26 +00:00
Aspen Smith
01765c3717 test(tvix/cli): Add some additional REPL tests
A couple of already-passing tests covering REPL behavior

Change-Id: Ie21f4abf68ab12827fd15128a8ef810cd8592d07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11959
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-07-07 14:19:17 +00:00
Aspen Smith
0ad986169d test(tvix/cli): Make the REPL testable
Juggle around the internals of the tvix-cli crate so that we expose the
Repl as a public type with a `send` method, that sends a string to the
repl and *captures all output* so that it can be subsequently asserted
on in tests. Then, demonstrate that this works with a single (for now)
REPL test using expect-test to assert on the output of a single command
sent to the REPL.

As the REPL gets more complicated, this will allow us to make tests that
cover that complex behavior.

Change-Id: I88175bd72d8760c79faade95ebb1d956f08a7b83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11958
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 14:19:17 +00:00
Aspen Smith
3a79f93795 refactor(tvix/eval): Construct globals in EvaluationBuilder::build
Construct the Rc<GlobalsMap> for the evaluation as part of
EvaluiationBuilder::build, rather than deferring it until we actually
compile. This changes nothing functionally, but gets us one step closer
to sharing this globals map across evaluations.

Change-Id: Id92e9fb88d974d763056d4f15ce61962ab776e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11957
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-06 15:24:50 +00:00
Aspen Smith
dfe137786c refactor(tvix/eval): Builderize Evaluation
Make constructing of a new Evaluation use the builder pattern rather
than setting public mutable fields. This is currently a pure
refactor (no functionality has changed) but has a few advantages:

- We've encapsulated the internals of the fields in Evaluation, meaning
  we can change them without too much breakage of clients
- We have type safety that prevents us from ever changing the fields of
  an Evaluation after it's built (which matters more in a world where we
  reuse Evaluations).

More importantly, this paves the road for doing different things with
the construction of an Evaluation - notably, sharing certain things like
the GlobalsMap across subsequent evaluations in eg the REPL.

Fixes: b/262
Change-Id: I4a27116faac14cdd144fc7c992d14ae095a1aca4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11956
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-06 15:03:46 +00:00
Aspen Smith
d5964c1d54 chore(tvix/docs): Cross off a TODO
This is done now, as of fc6359463 (feat(tvix/repl): Allow binding
variables at the top-level, 2024-07-04)

Change-Id: I2126cbadeb66625cda93a7cd7ddb31409ec60fd9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11955
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-07-06 03:06:51 +00:00
Ilan Joselevich
7ca32d9f0b refactor(web/tvixbolt): buildRustPackage -> crate2nix
With the recent changes to crate2nix and buildRustCrate in nixpkgs it is
now possible to build tvixbolt via crate2nix like we do for other tvix
crates. We can reuse a lot of the customizations done in //tvix in
tvixbolt to avoid repeating ourselves.

A script for serving tvixbolt locally for testing purposes is also
available now through the .serve attribute of tvixbolt.

This change supersedes https://cl.tvl.fyi/c/depot/+/11821.

Change-Id: I4864df8b75aec73cf5fee2428924ed4cfbb32902
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11952
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 20:19:39 +00:00
Ilan Joselevich
6a7069904e feat(tvix/utils): Add defaultCrateOverridesForPkgs function
This function can be reused across the rest of the repo to make use of
our overriden defaultCrateOverrides with support for tvix crates.

Change-Id: I8c554dece052bd9dd32acac13dab8114933272a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11951
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 20:19:14 +00:00
Ilan Joselevich
63654fbeb1 fix(tvix/utils): Optionally add support for Cargo in filterRustCrateSrc
Previously we would unconditionally add Cargo.toml to the fileset. We
mostly use buildRustCrate in tvix so it does not make sense to add it by
default, instead I made it so you enable cargoSupport if you want
Cargo.{toml,lock} to be added to the fileset.

Change-Id: I5a6016534fc5599e85ab581fe3d9b81e7a24f940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11950
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-05 20:07:09 +00:00
Aspen Smith
fc63594631 feat(tvix/repl): Allow binding variables at the top-level
Allow binding variables at the REPL's toplevel in the same way the Nix
REPL does, using the syntax <ident> = <expr>. This fully, strictly
evaluates the value and sets it in the repl's "env", which gets passed
in at the toplevel when evaluating expressions.

The laziness behavior differs from Nix's, but I think this is good:

    ❯ nix repl
    Welcome to Nix version 2.3.18. Type :? for help.

    nix-repl> x = builtins.trace "x" 1

    nix-repl> x
    trace: x
    1

    nix-repl> x
    1

vs tvix:

    tvix-repl> x = builtins.trace "x" 1
    trace: "x" :: string
    tvix-repl> x
    => 1 :: int
    tvix-repl> x
    => 1 :: int

Bug: https://b.tvl.fyi/issues/371
Change-Id: Ieb2d626b7195fa87be638c9a4dae2eee45eb9ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11954
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-07-05 16:40:35 +00:00
Aspen Smith
ac3d717944 feat(tvix/eval): Allow passing in an env to evaluation
Allow passing in a top-level env, a map from name to value, to
evaluation. The intent is to support bound identifiers in the REPL just
like upstream nix does.

Getting this working involves mucking around a bit with internals - most
notably, locals now only optionally have a Span (since locals don't have
an easy span we can use) - and getting that working requires propagating
some minor hacks to places where we currently *need* a span (and which
would require too much changing now to make spans optional; my guess is
that that would essentially end up making spans optional throughout the
codebase).

Also, some extra care has to be taken to close out the scope in the case
that we do pass in an env, to avoid breaking our assumptions about the
size of the stack when we return from the toplevel

Change-Id: Ie475b2d3dfc72ccbf298d2a3ea28c63ac877d653
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11953
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 16:39:34 +00:00
Florian Klink
af933c177a docs(tvix): update WASM status
crate2nix can now build WASM, and cl/11859 showcases this now.

Even if it's not in the same cargo workspace, we should still migrate
tvixbolt to a crate2nix build.

Remove the part about Build/Store frontends, it was more of an
explanation why we want WASM builds to be nice, it can be tracked in
separate TODOs once more concrete.

Change-Id: If4f5e0994b55520ba70cabefb4fcef9dc17bc394
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11945
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
2024-07-05 16:26:14 +00:00
Aspen Smith
6037888e18 refactor(tvix/eval): Drop LightSpan entirely
This was made unnecessary in c92d06271 (feat(tvix/eval): drop
LightSpan::Delayed, 2023-12-08) because it didn't improve benchmarks as
much as expected and has been vestigial since; this continues the
cleanup by just removing it altogether

Change-Id: I21ec7ae9b52a5cccd2092696a5a87f658194d672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11949
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 03:53:44 +00:00
toastal
8e31088a01 docs(tvix): introduction
Includes hacky, but workable solution for the pronunciation ‘issue’

Karl comes from: http://ipa-reader.xyz/?text=tvɪks&voice=Karl

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Iad7788ec7295902fd2159766a664016c7b1e2ae9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11908
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-04 14:18:09 +00:00
Florian Klink
9ed967791c test(tvix/store): test listing endpoint too
Change-Id: Ia4035aca43cf9d3f7de982dd154715120ba25496
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11927
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-03 12:35:07 +00:00
Florian Klink
eb9a74d966 fix(tvix/store/pathinfo/bigtable): fix listing endpoint
We were wrongly comparing the raw row_key with the store path
digest, but we hexlower-encode the digest in the row key (via
derive_pathinfo_key).

Update the logic to fix that.

Change-Id: I8916d8de9fb8b25a6986d4158faa91ec97c57347
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11926
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-03 12:35:07 +00:00
Simon Hauser
618aacaa61 feat(tvix/tracing): http trace propagation
Introduces a helper function within tvix-tracing that returns a reqwest
tracing middleware that will ingest the traceparent if otlp is enabled.

It is feature flagged in tvix-tracing so not every consumer of that
library automatically has reqwest in its dependencies.

Tested using netcat to verify that the `traceparent` header is there if
otlp is enabled and missing if otlp feature is disabled.

Change-Id: I5abccae777b725f5ff7382e3686165383c477a39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-02 13:43:09 +00:00
Vincent Ambo
4b2f3c5454 chore(3p/sources): bump to OpenSSH vulnerability hotfix
See https://github.com/NixOS/nixpkgs/pull/323753 for details.

Changes:

* git: temporarily comment out dottime patch (it doesn't apply, but it's not critical)
* third-party/cgit: use an older git version where dottime patch still applies
* 3p/crate2nix: remove crate2nix patches included in latest release
* tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs)
* tvix: regenerate Cargo.nix
* tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function
* tvix/nix-compat: remove redundant trait bounds
* tvix/glue: use clone_into() to set drv.{builder,system}
* tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327
* toold/depotfmt: expose treefmt config as passthru
* tools/crate2nix: undo some more hacks in the crate2nix-check drv

Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c
Co-Authored-By: Florian Klink <flokli@flokli.de>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 17:42:30 +00:00
Florian Klink
17bdf9a574 feat(tvix/store): add --remote-path-info-service-addr option
This is allows adding a cache in front of tvix-store daemon, and is less
code duplication than cl/11902, means we can probably land that until we
have proper store composition config.

It can be used to provide a tvix-store daemon interface for a Nix HTTP
Binary cache, saving all calculated PathInfo to another PathInfoService
after ingestion.

Change-Id: If141d718c2635f66aa90d46a80fd79c86c07d9ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11903
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 17:14:19 +00:00
Florian Klink
830fdda8d4 feat(tvix/store): use tokio-listener for tvix-store daemon command
This allows binding on unix sockets, as well as systemd socket
activation.

Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 15:53:53 +00:00
Florian Klink
f6c759de58 chore(tvix): bump bigtable_rs to latest release
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a
while, no need to use our own checkout here.

Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-07-01 15:53:53 +00:00
Simon Hauser
6a9a4d56a4 feat(tvix/tracing): expose stdout_writer and stderr_writer
Using std::io::{Stdout,StdErr} directly will clobber the output by an
active progress bar. To resolve this issue the exposed writers should be
prefered over `println!` and `eprintln!`.

Change-Id: Ic79465cd4e8b9dad5a138f6b08c5f0de9dcf54a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11860
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 13:55:02 +00:00
Florian Klink
87f38cad61 fix(tvix/boot): use shrunk kernel image
The aarch64-linux version of this is too big for cloud-hypervisor to
accept.

The correct place for this is in
`${kernel}/${stdenv.hostPlatform.linux-kernel.target}`, the other output
is only useful for debugging purposes.

This gets cloud-hypervisor to start booting up the kernel. It still
doesn't boot successfully, but it's getting further.

Part of b/407.

Change-Id: I3fbe52597fb1a6f5728a7f375dd35b50e00b3b1a
Co-Authored-By: Alyssa Ross <hi@alyssa.is>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11905
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Autosubmit: flokli <flokli@flokli.de>
2024-07-01 07:24:52 +00:00
Florian Klink
afbe995aa9 chore(tvix): bump to data-encoding 2.6.0
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-01 06:57:01 +00:00
Florian Klink
ffdc96ca59 feat(tvix/boot/tests): add closure-nixos-s3
This uses the `preStart` mechanism to boot up a minio instance, which
we point blob/directoryservice to.

Change-Id: I9e67b19006cb08ce82fe63c885cb0178e97389e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11896
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 19:23:57 +00:00
Florian Klink
bba64d8762 fix(tvix/castore/object_store): make query pairs object_store opts
We previously called ObjectStoreBlobService::parse_url, which passes an
empty list of options when constructing the ObjectStore.

This is most likely not what we want. The more reasonable thing to do is
pass along the query string (pairs) as options to
`object_store::parse_url_opts`, and remove them from the plain URL we
pass to object_store itself.

Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-29 19:23:57 +00:00
Florian Klink
d0ef6a50df refactor(tvix/castore/blobsvc): inline things a bit
Get rid of the `let grpc_client` and `let resp` in some cases.

Change-Id: Idc1c0f566a3b1b48da62e6f1977b07620656b16c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11884
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 17:51:33 +00:00
Florian Klink
10c7701ac1 docs(tvix/castore): drop wrong comment
This doesnt panic anymore.

Change-Id: I1987f8b0640b1e5fe33abcbf46164449c766fd0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11883
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-06-29 17:51:33 +00:00
Florian Klink
86ab052fe0 feat(tvix/boot/tests): closure-nixos-bigtable
This uses the `preStart` mechanism to boot up a bigtable emulator, which
we point directory/pathinfoservice to.

Change-Id: Icb383403cb2d1f7e19c71b6298e82b391f9c8d22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11895
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-29 16:56:08 +00:00
Florian Klink
6cf8003fa1 fix(tvix/boot/tests): escape addrs
Otherwise some URL parameters (containing &) might break the shell.

We cannot use `$PWD` anymore, but that's statically known anyways.

Change-Id: I739249111461ecdb33d42a48a4352f7612fce90f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11894
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-29 16:56:08 +00:00
Florian Klink
50ed7f27d0 feat(tvix/boot/tests): add preStart parameter
This allows running some code before tvix-daemon is started up, which is
helpful to spin up some mock services.

Change-Id: Ie7b6eaaf76c40def493f50879ee5255fd1ebd2fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11893
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-06-29 16:56:08 +00:00
Florian Klink
0fd4a1a655 fix(tvix/boot/tests): set SSL_CERT_FILE
This is necessary for http clients to successfully construct.

Change-Id: If0d968e964f5c58d9956b92ee0ab3d97c39391d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11892
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-29 16:56:08 +00:00
Florian Klink
7294a2f9aa feat(tvix/boot): introduce useNarBridge parameter
This is still useful to test the ingestion part of nar-bridge.

Reintroduce the logic removed in cl/11398, but put it behind a
`useNarBridge` parameter, defaulting to false.

Add an additional target to tvix.boot.tests,
`closure-nixos-nar-bridge`, which does the same as `closure-nixos`,
except seeding the contents through nar-bridge.

Change-Id: Iee87d5d9c7b7d02df11ee47fbc3f1fbe951935e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11891
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-06-29 16:38:03 +00:00
Florian Klink
4ded170e97 refactor(tvix/boot/tests): only use exportReferencesGraph when needed
These fields don't need to be set if isClosure is false.

Change-Id: Id29ed034e0757c61cef09885962dfc5507ca106c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11890
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 16:38:03 +00:00
Florian Klink
de8340335e feat(nix-compat/nixcpp/conf): make fields pub
Otherwise these are not accessible after parsing if parsed from another
crate.

Change-Id: Iba7b442722a2eb1d37c2c496a45aa559aa49c1f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11889
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-28 14:04:14 +00:00
Florian Klink
03c9917749 feat(nix-compat/nixcpp): init nix.conf config parsing
This allows parsing files like `/etc/nix/nix.conf` into the `NixConfig`
struct.

Change-Id: I90b25f43c429dd56127500ff5068e83852adee13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11888
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-28 12:23:03 +00:00
Florian Klink
327d115f3f feat(nix-compat/narinfo/pubkey): derive Clone, [Partial]Eq
We want to use this in another struct that we want to compare and clone.

Change-Id: I5c11d589a4922d85e190e1a9199468411016ff1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11887
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-06-28 12:23:03 +00:00
toastal
ade0781416 docs(tvix): create Welcome + Contributing sections
A lot of this content needs to be duplicated from the monorepo’s root
directory. However, some of the content was not relevant or could have
been more condensed in descriptions or split up for consumption that
also matched the styles of some other library documentation I had been
looking at for examples. I wanted to add the new access to Admonish
callouts but also aimed at the readablitiy in one’s text editor for the
user that needs to `cat` the docs.

In the future, a “Introduction” page would be nice to add to “Welcome”,
but I don’t think that is as clear.

Change-Id: Ibfc169c5fc3501d7eeb9a76423c6f64cd68897aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11869
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-27 07:16:23 +00:00
toastal
fbce56b849 docs: prompts in shell sessions shouldn’t be selectable
these just indicate the start of the prompt by convention & if the user
is root or not with `#` & `$` respectively

Change-Id: Id65f5f879b067ee96715b3e7dc63f68c7ad791b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11868
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-27 07:16:23 +00:00
Florian Klink
cd48566173 fix(tvix/glue): reject unknown attrset args for fetch builtins
This now uses UnexpectedArgumentBuiltin in case builtins.fetchurl or
builtins.fetchTarball are called with the wrong arguments:

```
note: while evaluating this Nix code
 --> [code]:1:1
  |
1 | builtins.readDir (builtins.fetchTarball { url = "https://git"; hash = "sha1-NKNeU1csW5YJ4lCeWH3Z/apppNU=";})
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E040]: Unexpected agrument `"hash"` passed to builtin
 --> [code]:1:19
  |
1 | builtins.readDir (builtins.fetchTarball { url = "https://git"; hash = "sha1-NKNeU1csW5YJ4lCeWH3Z/apppNU=";})
  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ while calling this builtin

```

Change-Id: I51124255a46b78d3cf4dc89a1eca9e68750858d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11878
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-26 16:06:52 +00:00
Florian Klink
aa7d125c12 refactor(tvix/eval): prefix ctx iterators in NixString
NixString::iter_plain() didn't make much sense, especially without a docstring.

Rename it to iter_ctx_plain(), and copy the docstring from NixContext.

Do the same for the two other context element types too.

Change-Id: I1bbfcb967d8d9b14487d069bfe3a1f762253ef4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11882
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-26 14:25:20 +00:00
Florian Klink
b757897e97 feat(tvix/glue): implement builtins.toFile
We currently only had a dummy implementation that didn't actually
persist the files in the store(s).

Copy the contents to the BlobService, and do the output path calculation
as part of the upload.
Use the plain context elements to construct the references.

Change-Id: Ibdaf7a645ddc31e847faa4b87a79f2f95116a7ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11879
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-26 14:25:20 +00:00
Florian Klink
888028b674 docs(tvix): use # for toplevel headings
See discussion in cl/11869. Let's use `#` consistently, we also use `##`
for deeper nested subheadings.

Change-Id: Ie25ebda708639fb617d456c275ae5a264fc4ce85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11881
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: toastal <toastal@posteo.net>
2024-06-26 14:22:19 +00:00
Florian Klink
50150fa3c8 feat(tvix/glue): add eval-fail-* support to glue tests
Add some eval-fail-fetcht-*.nix test cases.

Change-Id: Ic02beabf120d1de8df6f9c098d5a074127069f81
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11880
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 11:17:30 +00:00
Florian Klink
89361b2a7f fix(tvix/tracing): make cargo check and clippy happy
In case the otlp feature is not enabled, these generate warnings during
`cargo check`.
Fix by moving some imports into their functions, or using the
fully-qualified name (and one #[allow(unused_mut)])

Change-Id: I5afd89dcd4c772b6002cebdd5d0469932eacfdac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11873
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 06:27:15 +00:00
Florian Klink
a0993e7304 feat(tvix/eval): add ErrorKind::UnexpectedArgumentBuiltin
Change-Id: Ieb091b32aad566719fbe8604c4a589f5ccaaf6b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11877
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
970e2a045b refactor(tvix/eval): rename UnexpectedArgument{,Formals}
There's other places where unexpected arguments can be provided, like
in builtins.

Make space for that error type.

Change-Id: Ic831497a3a1dd36a3a184bedadcf1374bf0ae6db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11876
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-26 04:51:31 +00:00
Florian Klink
ea6f511241 refactor(tvix/glue): return a parsed Url in NixFetchArgs
The only two consumers (fetchurl, fetchtarball) of these do try to parse
it as URL, so do it in the helper.

Update url_basename to take a &url::URL, not a &str.

Also update the test to use rstest for the fixtures to reduce some
boilerplate there.

Change-Id: I1f85fe2803060dc4423e673cb7b9f9bf799d09b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11875
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
540e566900 refactor(tvix/glue): take &CAHash, not CAHash
We use a bit less cloning that way.

Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-26 04:51:31 +00:00
Florian Klink
78eb22c54d feat(tvix/glue): handle regular file at builtins.path import
If builtins.path is passed a regular file, no filtering is applied.
We use the just-introduced file_type function in the EvalIO trait for
that.

This means, we don't need to pass through filtered_ingest, and can
assemble the FileNode directly in that specific match case.

This also means, we can explicitly calculate the sha256 flat digest,
and avoid having to pipe through the file contents again (via
blob_to_sha256_hash) to construct the sha256 digest.

Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
080654aaf9 feat(tvix/eval): add file_type method
This allows peeking of the type at a given path.

It's necessary, as an open() might not fail until you try to read()
from it, and generally, stat'ing can be faster in some cases.

Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Ilan Joselevich
04f04cfc27 refactor(tvix/eval): drop placeholder impls of builtins in glue
The builtins `placeholder` and `filterSource` are now implemented in
tvix-glue and so we no longer need their placeholder "implementations" in
tvix-eval.

Change-Id: Ibf161ccf1ad40103e9fbb688e9f6be58e7772af1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11867
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-22 21:50:30 +00:00
toastal
bf541acf29 docs: use admonish for callouts
Change-Id: Icec7321a49c40e7f7bcb1e4b307f13b8db4eac25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11862
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-21 08:08:17 +00:00
Florian Klink
8dde022d31 docs(tvix): drop done TODOs in the O11Y section
OTLP shutdown behaviour was fixed in cl/11803, and tracy landed in
cl/10952 and cl/11815.

Change-Id: I310dfcc41ba62ce5b006e75a35feb839327b0e7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11866
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Florian Klink
7f625e2fc1 refactor(tvix/*store): remove some trait bounds
We don't need to require these things for these impl blocks yet.

Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Simon Hauser
639a00e2ab feat(tvix/tracing): gRPC trace context propagation
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.

The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.

This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.

Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-20 19:21:01 +00:00
Simon Hauser
2b20d8d82d feat(tvix/castore): instrument tokio task with current span
By default tokio::spawn does not instrument the spawned task with the
current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do
this manually for all tokio::spawn functions in functions that are
instrumented.

Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 15:48:14 +00:00
Simon Hauser
bd8d74a3ee feat(tvix/tracing): optional progressbar
Disable the progressbar on default and provide a interface for
optionally enabling the progressbar.

Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 10:28:54 +00:00
Florian Klink
1446e3be99 docs(tvix/docs): document nar renderer optimization potential
Change-Id: I747ab5c8ba3424828b1c88e0353170ad0efab3a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11849
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 10:28:33 +00:00
Florian Klink
8137077a74 feat(tvix/store/utils): detect gRPC NAR calculation service
We were currently always using SimpleRenderer, which would mean the
client would download every blob locally to calculate the checksum,
which of course is very slow.

Detect this special case and create a second instance (and client) for
now.

Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 10:28:26 +00:00
Ilan Joselevich
9fd601e53d chore(tvix): patch crate2nix to drop darwin dontStrip
This patch has already been applied on upstream but there hasn't been an
official release yet so we patch it ourselves.

Upstreamed patch: 0209f258cd

Follow-up of: https://cl.tvl.fyi/c/depot/+/11856

Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-18 14:13:11 +00:00
Ilan Joselevich
92c5433304 chore(tvix): move filterRustCrateSrc to utils.nix
Change-Id: Ib6bae1ea0457d3309a6ec1e08b9e4b320523c161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11857
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 15:26:19 +00:00
Ilan Joselevich
ac51803ad8 chore(tvix): Remove no longer needed libiconv addition for darwin
Since https://github.com/NixOS/nixpkgs/pull/220212 libiconv is now
automatically added for Darwin.

Change-Id: I2aaa266bcbc4d5a8256131aed36b4c2d040a8758
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11856
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-17 15:08:10 +00:00
Ilan Joselevich
210b4277b6 chore(tvix): Use callPackage instead of import on Cargo.nix
Since crate2nix v0.11 the new preferred way of importing Cargo.nix is
with callPackage instead of import.

Change-Id: Ifcc5c21e6c101cb9d0b3f1e8025b5ddb8cf99f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11855
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:46 +00:00
Ilan Joselevich
fbe29615df chore(tvix): Remove no longer needed rlib stripping fix for darwin
This issue was fixed in https://github.com/NixOS/nixpkgs/pull/255900

Change-Id: I4f569099e62c4284bdea18a512387735f139e702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:04 +00:00
Florian Klink
28b692fd50 feat(tvix/tvix-store): improve progress bars
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.

Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.

Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.

Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 12:57:34 +00:00
Florian Klink
cfab953094 feat(tvix/store): display progress for NAR calculation
This is currently still taking a noticeable amount of time, so make sure
we show it is happening.

Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 09:47:54 +00:00
Florian Klink
71a29ceff4 docs(tvix/store/import): update import_path_as_nar_ca docstr
Change-Id: I697172220efe73b2a215185027df09f815d32627
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11846
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 08:49:52 +00:00
Florian Klink
4be250109c refactor(tvix/store): drop FUSE multithread workaround for Darwin
The underlying issue in macFUSE has been fixed in
https://github.com/osxfuse/osxfuse/issues/974.

Bump our `macfuse` in nixpkgs to a version containing the fix.

This can be removed while our nixpkgs pin is bumped past
https://github.com/NixOS/nixpkgs/pull/320197.

Change-Id: Ia0e644fb13198e45018b0a218647ef211acf4df1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11824
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-16 12:02:11 +00:00
Florian Klink
7e42b4f314 fix(tvix/store/bin): fix shutdown behaviour for FUSE
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.

So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.

Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.

Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.

Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.

Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-06-16 12:02:11 +00:00
Yureka
452299dcd2 feat(castore/directory): add cache combinator
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 08:42:53 +00:00
Yureka
30e72d2d52 refactor(castore/directory/objectstore): use new order validation logic
Use the new helper introduced in CL 11708 instead of rolling our own.

Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 08:40:20 +00:00
Yureka
daada1b2fa refactor(castore/directory): separate order logic from ClosureValidator
ClosureValidator was previously only suitable for a very narrow use case:
Validating incoming uploads, which are in leaves-to-root order.
This is because the ordering validation was hard-wired into the add()
function.
This
- Re-name ClosureValidator to DirectoryGraph, which is more suitable
  since it actually stores the Directory structs and is drained in the end.
- Move the ordering-related logic to a separate OrderValidator, which
  can be used independently.
- re-write DirectoryGraph to be a general purpose
  validator which can accept the input in both orders
  and can be drained in both orders as well.

This means the DirectoryGraph and OrderValidator can now serve
multiple new purposes:
- Validating the incoming closure on the client while downloading.
- Validating the incoming closure downloaded in a caching layer from the
  `far` cache, and re-order it for insertion into the `near` cache.

Change-Id: I2b4b226348416912d7a31935bec050e53d911b70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 05:01:07 +00:00
Florian Klink
01aadace80 feat(tvix/cli,store): add tracy feature
If compiled with this features, this emits packets compatible with the
[Tracy](https://github.com/wolfpld/tracy) format.

Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 05:24:37 +00:00
Ilan Joselevich
cd0b92419a docs(tvix/TODO): dibs on crate2nix WASM
Change-Id: Idde9aaa31ecaa9570559ffe3271bc23b7a71133c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11820
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 04:02:33 +00:00
Florian Klink
86928b5e7f docs(tvix/TODO): add builtins.path roundtrip for flat
Change-Id: I8291d2fdfa2ea31ec24c82dcfb447fa9eaa3a961
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11819
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-15 04:02:33 +00:00
Florian Klink
6e321c615c docs(tvix/*store): add some internal links
We can now cross-link between pages, and it actually works.

Change-Id: Id8bd85111672c687118db9adae7f5066f22441f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11817
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 03:56:11 +00:00
Florian Klink
46585bee15 docs(tvix/TODO): drop builtins:fetchurl item
This has been implemented in cl/11786, cl/11790 and cl/10975.

This includes support for the `executable` and `unpack` variants of it.

Change-Id: I5bcad5b8e86b9fd341219a59ba86660b692e66e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11818
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 01:01:04 +00:00
Florian Klink
6b6a34065e feat(tvix/tracing): add tracing-tracy support
This introduces another feature flag, "tracy" to the `tvix-tracing` crate.

If enabled (not enabled by default), it'll add an additional layer
emitting packets in a format that https://github.com/wolfpld/tracy can
display.

I had to be a bit tricky with the combinatorial complexity when adding
this, but the resulting code still seems manageable.

Change-Id: Ica824496728fa276ceae3f7a9754be0166e6558f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10952
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-14 19:33:44 +00:00
Florian Klink
d25f047b9d refactor(tvix/tracing): move otlp setup into helper function
Having all this in the main control flow makes it a bit hard to read.
Moving it into a helper function makes it a bit cleaner.

Change-Id: Ibdb739dbd1e013b4f8c4aaf9b036a6bd556a1871
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11814
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-14 14:52:19 +00:00
Simon Hauser
118e69d81d fix(tvix/tracing): reduce the error logs of otlp if collector is offline
The problem is that opentelemetry_otlp tonic batch exporter tries to
exports if either the `scheduled_delay` or if the
`max_export_batch_size` is reached. Per default the
`max_export_batch_size` is set to 512 spans, which means that we try to
export these spans once that counter is reached. Each export will then
try to connect to the exporter (if that not already happening) and will
result in a `tcp connect error`.
Increasing the max_export_batch_size to 4096 will then ensure that the
export only happens if the `scheduled_delay` is met after the 10
seconds.
`max_queue_size` is also increased, because `max_export_batch_size`
should not be greater than `max_queue_size`, so similar to the default
config its set to `max_export_batch_size * 4`.
This will reduce the amount of tries to otlp if the collector is not
available and otlp enabled.

Change-Id: Ic3430006e8a104fa3b34d274678cae55b3620ce9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11791
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 10:20:31 +00:00
Simon Hauser
a857a2b978 feat(tvix/tracing): apply EnvFilter to all layers
Currently we apply the EnvFilter only to the stderr output writer.
This didn't affect any other layer, like the otlp layer, causing spans
from `h2`, `tokio_util` or other third party crate dependencies to be
always sent out via OTLP.

This changes that behaviour, applying EnvFilter to all exports, leading
to a lot less spans being exported.

Change-Id: I9f3a7233e9d0aeaa81fe08914579f0b3c80d134e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11813
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 10:18:59 +00:00
Simon Hauser
fa7ed39bf4 feat(tvix/tracing): correctly close otlp on exit
Provide a new interface for forcing a flush of otlp traces and use this
interface to shutdown otlp prior to exiting tvix-store, either if the
tool was stopped with a SIGTERM or ended regularly.
This also fixes an issue where traces were not even exported if for
example we just imported 10 paths and never even emitted more than 256
traces. The implementation uses a mpsc channel so a flush can be done
without having to wait for it to complete. If you want to wait for a
flush to complete you can provide a oneshot channel which will receive a
message once flushing is complete.

Because of a otlp bug `force_flush` as well as
`shutdown_tracer_provider` need to be executed using `spawn_blocking`
otherwise the function will deadlock. See
https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335

Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 09:34:51 +00:00
Florian Klink
5077ca70de chore(tvix/eval): move eval docs to tvix/docs
Change-Id: I75b33c43456389de6e521b4f0ad46d68bc9e98f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11809
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-06-14 08:00:34 +00:00
Florian Klink
6947dc4349 chore(tvix/docs): move [ca]store docs to tvix/docs
Change-Id: Idd78ffae34b6ea7b93d13de73b98c61a348869fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11808
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2024-06-14 08:00:34 +00:00
Florian Klink
adc7353bd1 feat(tvix/docs): add d2 support
This provides a nicer syntax to draw some diagrams.

Change-Id: I2ab580e61399a45f01ed422ee29382e860753ffa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11807
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-06-14 07:04:23 +00:00
Florian Klink
4df3284dd0 feat(tvix/docs): add mdbook-admonish
This provides a plugin for callouts.

It needs to have additional CSS file added to `book.toml`,
which can be (re-)generated by `mdbook-admonish install`.

See https://github.com/tommilligan/mdbook-admonish/issues/171#issuecomment-2009166079
for more context.

Use it by adding a warning one to the architecture document.

Change-Id: I75c9a33d00acb603c6da10d3f9ce3485731c1672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11805
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
2024-06-14 07:04:23 +00:00
Florian Klink
9d44da01ee fix(tvix-glue/fetchers/decompression): propagate error with ?
Change-Id: I5e9d4f067f09307cb6f990ed6eca48345e2fb342
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11810
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-13 21:07:40 +00:00
Florian Klink
b992ca49a6 fix(tvix/glue/tvix_store_io): also populate input sources
These also need to be present in the input nodes of the BuildRequest.

Change-Id: Ie9b957805e42f766002581adc6182a6543c5333b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11802
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 19:59:28 +00:00
Florian Klink
c30fded880 chore(tvix/docs): move daemon protocol to its own chapter
… and add an introductionary page at the root.

Change-Id: I89ce69a4221c332b7c0d99ac26b4e26bfd248341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11806
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-13 19:23:20 +00:00
Florian Klink
d947f61d36 fix(tvix/glue/tvix_store_io): disable concurrent fetches for now
We need some shared queue, preventing the same fetches/builds from
getting triggered multiple times unnecessarily.

Change-Id: I7c4a3c66db558f5cccd66865b170242b758e3e02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11800
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-13 16:19:18 +00:00
Florian Klink
79bfa931ed feat(tvix/tracing): set release_max_level_debug for tracing
This allows explicitly opting in to get DEBUG-level log lines, by
setting RUST_LOG.

It currently also causes traces to be emitted in all cases, so we might
do some runtime filtering there too, as discussed in cl/11791.

Change-Id: I2865bb06a62465836d63196422f5f734f7165386
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11801
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 16:18:47 +00:00
Florian Klink
29eda2d5b2 feat(tvix/cli): add toplevel progress span
This gives some better feedback something is going on, gives a (named)
root for all the progress children we're drawing, and also counts the
time we are in eval.

Change-Id: Ibe81dcebf0a2b59bb0680da62e206eb5270f9d3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11798
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-06-13 16:18:15 +00:00