Commit graph

261 commits

Author SHA1 Message Date
Ilan Joselevich
9803712ab8 fix(tvix): Follow-up fixing no-default-features and comments
In https://cl.tvl.fyi/c/depot/+/12389 I accidentally deleted some
comments. Also fixed some useless no-default-features with futures crate, making it match the previous feature combination.

Change-Id: I72bb2cfb88719ff2f8812d90193de2bd49149cce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12395
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-30 13:55:11 +00:00
Ilan Joselevich
5a97888d8b chore(tvix): Migrate members to inherit deps from workspace
From now on we will add the dependencies and their version in the root
Cargo.toml and in order to enable the dependency for a workspace member
we set `workspace = true` in the member's Cargo.toml.

Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-29 14:37:22 +00:00
Ilan Joselevich
2945a359b4 chore(tvix): Bump versions of all compatible deps via cargo upgrade
This updates all the dependencies and their "minimum" versions in
Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s
`cargo upgrade` command that will eventually be merged into `cargo
update`.

Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-29 11:26:15 +00:00
Florian Klink
02ee441626 chore(tvix/[ca]store): bump bigtable_rs
This bumps bigtable_rs to
https://github.com/liufuyang/bigtable_rs/pull/86, allowing us to drop
our second set of prost/tonic/http/axum crates.

Change-Id: I70f9150289c3e8611ebe8a7d99490e3dfd085a6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12384
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-08-28 20:15:55 +00:00
Florian Klink
b6dd9d2d5b chore(tvix/[ca]store): bump hyper-util to 0.1.7
Change-Id: Ia5eaf7f4614701ad4dd06114476a73e1a041d830
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12383
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-28 20:15:55 +00:00
Florian Klink
b2b38f23d0 chore(tvix): bump tonic[-build] from 0.12.1 to 0.12.2
Change-Id: If7d1ae4491d3c70f323cf872d1a70afe6238f78a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12382
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-28 19:01:04 +00:00
Brian Olsen
a5fcfd80a1 fix(tvix/nix-compat-derive): Get rid of external feature flag
The external feature flag was there because I couldn't find a way to
refer to crate and nix-compat with the same name so that the generated
code could be the same.

In essence `use nix_compat::nix_daemon:🇩🇪:NixDeserialize` is an error
when used inside nix_compat crate.

So my best fix was the external feature flag until I found the solution
used here which also removes the flag completely.

Change-Id: Ia3e89c6c350c3fb22ca87f974a39c21542aae152
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12376
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
2024-08-28 14:43:00 +00:00
Florian Klink
2fa5e71d5e fix(tvix/store): restore v1alpha reflection endpoint
tonic-reflection 0.12.x moved from the v1alpha to v1 of the reflection
protocol.

However, most clients, like Postman, Kreya and evans don't support that
one yet.

Bump tonic-reflection to 0.12.2, which re-introduces v1alpha support
alongside the v1 version of it, registering both services.

This fixes the example documented in tvix/store/README.md, it was
previously failing as evans couldn't find the v1alpha reflection
service.

See https://github.com/hyperium/tonic/pull/1888 for details.

Change-Id: I55438877317f82dc39face13afeb9594cda07a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12353
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-27 10:34:43 +00:00
Brian Olsen
ced05a2bb6 feat(tvix/nix-compat-derive): Add deriver for NixDeserialize
This adds a nix-compat-derive derive crate that implements a deriver
for NixDeserialize implementations. This is to reduce the amount of
code needed to implement deserialization for all the types used by
the Nix daemon protocol.

Change-Id: I484724b550e8a1d5e9adad9555d9dc1374ae95c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12022
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-25 15:05:50 +00:00
Brian Olsen
9af6920478 feat(nix-compat): Add NixDeserialize and NixRead traits
Add a trait for deserializing a type from a daemon worker connection.
This adds the NixDeserialize trait which is kind of like the serde
Deserialize trait in that individual types are meant to implement it
and it can potentially be derived in the future.

The NixDeserialize trait takes something that implements NixRead as
input so that you can among other things mock the reader.

Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-25 15:05:50 +00:00
Florian Klink
e03ea11bad feat(nix-compat/nix_http): init parse_nar[info]_str
This moves the URL component parsing code we had in nar-bridge to
nix-compat.

We change the function signature to return an Option, not a
Result<_, StatusCode>.

This allows returning more appropriate error codes, as we can
ok_or(…) at the callsite, which we now do: on an upload to an
invalid path, we now return "unauthorized", while on a GET/HEAD, we
return "not found".

This also adds support to parse compression suffixes. While not
supported in nar-bridge, other users of nix-compat might very well want
to parse these paths.

Also fix the error message when parsing NAR urls, it mentioned 32, not
52, which is a copypasta error from the narinfo URL parsing code.

Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-21 09:40:32 +00:00
Florian Klink
a259613c76 feat(nix-compat/narinfo/signature): generalize name field
Requiring `name` to be a `&str` means it'll get annoying to pass around
`Signature`, but being able to pass them around in an owned fashion is
kinda a requirement for a stronger typed `PathInfo` struct, where we
want to have full ownership.

Rework the `Signature` struct to become generic over the type of the
`name` field. This means, it becomes possible to have owned versions
of it.

We don't want to impose `String` or `SmolStr` for example, but want to
leave it up to the nix-compat user to decide.

Provide a type alias for the existing `&str` variant (`SignatureRef`),
and use it where we previously used the non-generic `Signature` one.

Add some tests to ensure it's possible to *use* `Signature` with both
`String` and `SmolStr` (but only pull in `smol_str` as dev dependency
for the tests).

Also, add some more docstrings, these were a bit sparse.

Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 19:46:34 +00:00
Vincent Ambo
abff828ccc refactor(tvix/eval): remove use of imbl::OrdMap
Removes imbl::OrdMap in favour of an Rc over the standard library's BTreeMap,
which allows us to drop the imbl dependency completely.

In my local tests this is actually slightly faster for `hello` and `firefox`.

Change-Id: Ic9597ead4e98bf9530f290c6a94a3c5c3efd0acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12201
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-08-19 11:02:50 +00:00
Vincent Ambo
d6c57eb957 refactor(tvix/eval): ensure VM operations fit in a single byte
This replaces the OpCode enum with a new Op enum which is guaranteed to fit in a
single byte. Instead of carrying enum variants with data, every variant that has
runtime data encodes it into the `Vec<u8>` that a `Chunk` now carries.

This has several advantages:

* Less stack space is required at runtime, and fewer allocations are required
  while compiling.
* The OpCode doesn't need to carry "weird" special-cased data variants anymore.
* It is faster (albeit, not by much). On my laptop, results consistently look
  approximately like this:

  Benchmark 1: ./before -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings
  Time (mean ± σ):      8.224 s ±  0.272 s    [User: 7.149 s, System: 0.688 s]
  Range (min … max):    7.759 s …  8.583 s    10 runs

  Benchmark 2: ./after -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings
  Time (mean ± σ):      8.000 s ±  0.198 s    [User: 7.036 s, System: 0.633 s]
  Range (min … max):    7.718 s …  8.334 s    10 runs

  See notes below for why the performance impact might be less than expected.
* It is faster while at the same time dropping some optimisations we previously
  performed.

This has several disadvantages:

* The code is closer to how one would write it in C or Go.
* Bit shifting!
* There is (for now) slightly more code than before.

On performance I have the following thoughts at the moment:

In order to prepare for adding GC, there's a couple of places in Tvix where I'd
like to fence off certain kinds of complexity (such as mutating bytecode, which,
for various reaons, also has to be part of data that is subject to GC). With
this change, we can drop optimisations like retroactively modifying existing
bytecode and *still* achieve better performance than before.

I believe that this is currently worth it to pave the way for changes that are
more significant for performance.

In general this also opens other avenues of optimisation: For example, we can
profile which argument sizes actually exist and remove the copy overhead of
varint decoding (which does show up in profiles) by using more adequately sized
types for, e.g., constant indices.

Known regressions:

* Op::Constant is no longer printing its values in disassembly (this can be
  fixed, I just didn't get around to it, will do separately).

Change-Id: Id9b3a4254623a45de03069dbdb70b8349e976743
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12191
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 11:02:50 +00:00
Brian Olsen
3cd57ce4e3 feat(tvix/glue): Add refscanner pattern and AsyncRead
This splits the existing ReferenceScanner into a ReferenceScanner and
ReferencePattern as well as adds an AsyncRead implementation that can
do a scan while you read from it.

The reason to split the scanner in two is that generating the pattern
is expensive and when ingesting build results with multiple outputs you
want to do several independant scans that look for the same pattern.

The reader is for scanning files without having to load the entire file
into memory.

Change-Id: I993f5a32308c12d9035840f8e04fe82e8dc1d962
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12052
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-16 16:26:18 +00:00
Florian Klink
d3c32f214c chore(tvix): bump object_store
Fix for RUSTSEC-2024-0358.

Change-Id: Id9e8c6ae67845d95a52c884035fcea02d2d71505
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12214
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-16 06:52:15 +00:00
Florian Klink
5d3ae11aa5 chore(tvix/nix-compat): bump curve25519-dalek
Fix for RUSTSEC-2024-0344.

Change-Id: I9e9cac5f0dac9e5b2e5a934c99762d982136c0e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12210
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-16 06:45:37 +00:00
Aspen Smith
d378111d77 feat(tvix/eval): Store hash in key of interner
Rather than storing the leaked allocation for the string as the key in
the interner, store the hash (using NoHashHashBuilder). I thought this
would improve performance, but it doesn't:

hello outpath           time:   [736.85 ms 748.42 ms 760.42 ms]
                        change: [-2.0754% +0.4798% +2.7096%] (p = 0.72 > 0.05)
                        No change in performance detected.

but it at least doesn't *hurt* performance, and it *does* avoid an
`unsafe`, so it's probably net good.

Change-Id: Ie413955bdb6f04b1f468f511e5ebce56e329fa37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12049
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-08-10 15:28:30 +00:00
Ilan Joselevich
9c4b57ac63 fix(third_party/overlays): Patch crate2nix to use mkDerivation for tests
The problem with using runCommand and recreating the src directory with
lndir is that it changes the file types of individual files, they will
now be a symlink instead of a regular file. If you have a crate that tests
that a file is of regular type then it will fail inside the crate2nix
derivation.

Also regenerate Cargo.nix for //tvix as it will be needed in the next
commit.

Change-Id: I9275602cc17a428f9fdf0e55daf12cd673bbc030
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12131
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-09 14:35:57 +00:00
Ilan Joselevich
f648f17ec3 feat(tvix): Jemalloc -> MiMalloc
Use the faster and newer MiMalloc memory allocator for all endpoints in
the workspace.

Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-08-08 08:06:23 +00:00
Aspen Smith
b8f92a6d53 feat(tvix/eval): Forbid Hash{Map,Set}, use Fx instead
Per https://nnethercote.github.io/perf-book/hashing.html, we have
basically no reason to use the default hasher over a faster,
non-DoS-resistant hasher. This gives a nice perf boost basically for
free:

hello outpath           time:   [704.76 ms 714.91 ms 725.63 ms]
                        change: [-7.2391% -6.1018% -4.9189%] (p = 0.00 < 0.05)
                        Performance has improved.

Change-Id: If5587f444ed3af69f8af4eead6af3ea303b4ae68
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12046
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: aspen <root@gws.fyi>
2024-08-07 12:38:40 +00:00
Aspen Smith
59056cf705 feat(tvix/eval): Leak strings (with flag to disable)
Default to always leaking strings, and copying strings by copying
pointers rather than cloning the underlying allocation. This (somewhat
bafflingly) doesn't seem to affect any benchmarks, but paves the way for
some tricks around string allocation that do.

Unfortunately, we can't do this (yet?) for contextful strings, for
reasons I don't currently understand but which I will address later,
when I address contextful strings more holistically.

I've left a flag in here to disable this, both to test the cloning logic
for strings for when/if we decide to bring this back, and to allow
people who care more about memory usage than perf to disable leaking.

Change-Id: Iec44bcbfe9b3d20389d2450b9a551792a79b9b26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12045
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-05 10:49:27 +00:00
Ilan Joselevich
9f10a71ec5 feat(tvix/store): add redb PathInfoService
This provides a PathInfoService implementation using redb
(https://github.com/cberner/redb) as the underlying storage engine.

Both an in-memory variant, as well as a filesystem one is provided,
similar how it's done with the sled implementation.

Supersedes: https://cl.tvl.fyi/c/depot/+/11692
Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc
Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-22 18:49:04 +00:00
Yureka
2547ff2cf6 feat(tvix/store): add grpc healthcheck service to daemon
Change-Id: Ib95fc9352a45d54f9a16c8841c7e8f7cbeeaee8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12019
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 15:57:30 +00:00
Yureka
39c8b6dece fix(store): add toml dependency for feature xp-store-composition
Fixes 'use of undeclared crate or module ' with --features xp-store-composition

Change-Id: I44dce86e656094d180b91a00f385f685d21f3fbd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12021
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 15:56:59 +00:00
Yureka
14a4b4cbc3 feat(tvix): add experimental-store-composition option
Change-Id: I61661fbb0e77ce3c00c2a467dfabdf3fc77d8575
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12011
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 13:36:08 +00:00
Yureka
67335c41b7 refactor(tvix): move service addrs into shared clap struct
Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 13:36:08 +00:00
Yureka
6774d9c59c chore(tvix/tracing): switch tracing-opentelemetry from git to release
Change-Id: Ib830c8b642496a6cdf3d4b9093f0343f5fb80622
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12018
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 13:36:08 +00:00
Yureka
ca8e2b9fbf chore(tvix): upgrade to tonic 0.12 / hyper 1.0
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 11:01:38 +00:00
Simon Hauser
1515a970be feat(tvix/tracing): http propagation for axum
It introduces a new accept_trace function for axum0.7 which can be used
to accept a header trace from a received request. This function can be
used for tonic 0.12 once that version is released, and the specific
`accept_trace` function within `tvix_tracing::propagate::tonic` can then
be removed.

This also integrates http propagation into the nar_bridge crate.

Change-Id: I46dcc797d494bb3977c2633753e7060d88d29129
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11925
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 05:45:19 +00:00
Florian Klink
5d906054da feat(tvix/nar-bridge): support uploading NAR files
This ingests NAR files into the {Blob,Directory}Service, which are
already part of the AppState.

As we then need to correlate the root node to the uploaded PathInfo, we
need to keep a (short-lived) lookup table from NARHash to root node
around. We insert it into a `LruCache` after the NAR is uploaded, and
use `peek()` to do the lookup, which doesn't update the LRU list.

Change-Id: I48a4c6246bacf76559c5a4ccad2a0bc25c1b7900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11986
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:23:18 +00:00
Florian Klink
861cc1f341 feat(tvix/nar-bridge): init
This adds an implementation of nar-bridge in Rust.
Currently, only the GET parts are implemented.

Contrary to the Go variant, this doesn't try to keep a mapping from nar
hashes to root node in memory, it simply encodes the root node itself
(stripped by its basename) into the URL.

This pulls in a more recent version of axum than what we use in
tonic, causing two versions of http and hyper, however dealing with
`Body::from_stream` in axum 0.6 is much more annoying, and
https://github.com/hyperium/tonic/pull/1740 suggests this will be fixed
soon.

Change-Id: Ia4c2dbda7cd3fdbe47a75f3e33544d19eac6e44e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11898
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-20 16:53:23 +00:00
Yureka
1a6b6e3ef3 feat(tvix/castore): add composition module
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748
Signed-off-by: Yureka <tvl@yuka.dev>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-18 10:42:11 +00:00
Aspen Smith
89d204d295 feat(tvix/eval): Bump imbl to 3.0.0
Change-Id: I88053fa2df35dc486c51d015f4b2156541c07af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11962
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 15:05:27 +00:00
Aspen Smith
0ad986169d test(tvix/cli): Make the REPL testable
Juggle around the internals of the tvix-cli crate so that we expose the
Repl as a public type with a `send` method, that sends a string to the
repl and *captures all output* so that it can be subsequently asserted
on in tests. Then, demonstrate that this works with a single (for now)
REPL test using expect-test to assert on the output of a single command
sent to the REPL.

As the REPL gets more complicated, this will allow us to make tests that
cover that complex behavior.

Change-Id: I88175bd72d8760c79faade95ebb1d956f08a7b83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11958
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 14:19:17 +00:00
Aspen Smith
fc63594631 feat(tvix/repl): Allow binding variables at the top-level
Allow binding variables at the REPL's toplevel in the same way the Nix
REPL does, using the syntax <ident> = <expr>. This fully, strictly
evaluates the value and sets it in the repl's "env", which gets passed
in at the toplevel when evaluating expressions.

The laziness behavior differs from Nix's, but I think this is good:

    ❯ nix repl
    Welcome to Nix version 2.3.18. Type :? for help.

    nix-repl> x = builtins.trace "x" 1

    nix-repl> x
    trace: x
    1

    nix-repl> x
    1

vs tvix:

    tvix-repl> x = builtins.trace "x" 1
    trace: "x" :: string
    tvix-repl> x
    => 1 :: int
    tvix-repl> x
    => 1 :: int

Bug: https://b.tvl.fyi/issues/371
Change-Id: Ieb2d626b7195fa87be638c9a4dae2eee45eb9ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11954
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-07-05 16:40:35 +00:00
Simon Hauser
618aacaa61 feat(tvix/tracing): http trace propagation
Introduces a helper function within tvix-tracing that returns a reqwest
tracing middleware that will ingest the traceparent if otlp is enabled.

It is feature flagged in tvix-tracing so not every consumer of that
library automatically has reqwest in its dependencies.

Tested using netcat to verify that the `traceparent` header is there if
otlp is enabled and missing if otlp feature is disabled.

Change-Id: I5abccae777b725f5ff7382e3686165383c477a39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-02 13:43:09 +00:00
Vincent Ambo
4b2f3c5454 chore(3p/sources): bump to OpenSSH vulnerability hotfix
See https://github.com/NixOS/nixpkgs/pull/323753 for details.

Changes:

* git: temporarily comment out dottime patch (it doesn't apply, but it's not critical)
* third-party/cgit: use an older git version where dottime patch still applies
* 3p/crate2nix: remove crate2nix patches included in latest release
* tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs)
* tvix: regenerate Cargo.nix
* tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function
* tvix/nix-compat: remove redundant trait bounds
* tvix/glue: use clone_into() to set drv.{builder,system}
* tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327
* toold/depotfmt: expose treefmt config as passthru
* tools/crate2nix: undo some more hacks in the crate2nix-check drv

Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c
Co-Authored-By: Florian Klink <flokli@flokli.de>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 17:42:30 +00:00
Florian Klink
830fdda8d4 feat(tvix/store): use tokio-listener for tvix-store daemon command
This allows binding on unix sockets, as well as systemd socket
activation.

Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 15:53:53 +00:00
Florian Klink
f6c759de58 chore(tvix): bump bigtable_rs to latest release
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a
while, no need to use our own checkout here.

Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-07-01 15:53:53 +00:00
Florian Klink
afbe995aa9 chore(tvix): bump to data-encoding 2.6.0
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-01 06:57:01 +00:00
Simon Hauser
639a00e2ab feat(tvix/tracing): gRPC trace context propagation
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.

The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.

This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.

Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-20 19:21:01 +00:00
Ilan Joselevich
9fd601e53d chore(tvix): patch crate2nix to drop darwin dontStrip
This patch has already been applied on upstream but there hasn't been an
official release yet so we patch it ourselves.

Upstreamed patch: 0209f258cd

Follow-up of: https://cl.tvl.fyi/c/depot/+/11856

Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-18 14:13:11 +00:00
Florian Klink
28b692fd50 feat(tvix/tvix-store): improve progress bars
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.

Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.

Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.

Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 12:57:34 +00:00
Florian Klink
7e42b4f314 fix(tvix/store/bin): fix shutdown behaviour for FUSE
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.

So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.

Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.

Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.

Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.

Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-06-16 12:02:11 +00:00
Florian Klink
01aadace80 feat(tvix/cli,store): add tracy feature
If compiled with this features, this emits packets compatible with the
[Tracy](https://github.com/wolfpld/tracy) format.

Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 05:24:37 +00:00
Florian Klink
6b6a34065e feat(tvix/tracing): add tracing-tracy support
This introduces another feature flag, "tracy" to the `tvix-tracing` crate.

If enabled (not enabled by default), it'll add an additional layer
emitting packets in a format that https://github.com/wolfpld/tracy can
display.

I had to be a bit tricky with the combinatorial complexity when adding
this, but the resulting code still seems manageable.

Change-Id: Ica824496728fa276ceae3f7a9754be0166e6558f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10952
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-14 19:33:44 +00:00
Simon Hauser
fa7ed39bf4 feat(tvix/tracing): correctly close otlp on exit
Provide a new interface for forcing a flush of otlp traces and use this
interface to shutdown otlp prior to exiting tvix-store, either if the
tool was stopped with a SIGTERM or ended regularly.
This also fixes an issue where traces were not even exported if for
example we just imported 10 paths and never even emitted more than 256
traces. The implementation uses a mpsc channel so a flush can be done
without having to wait for it to complete. If you want to wait for a
flush to complete you can provide a oneshot channel which will receive a
message once flushing is complete.

Because of a otlp bug `force_flush` as well as
`shutdown_tracer_provider` need to be executed using `spawn_blocking`
otherwise the function will deadlock. See
https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335

Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 09:34:51 +00:00
Florian Klink
79bfa931ed feat(tvix/tracing): set release_max_level_debug for tracing
This allows explicitly opting in to get DEBUG-level log lines, by
setting RUST_LOG.

It currently also causes traces to be emitted in all cases, so we might
do some runtime filtering there too, as discussed in cl/11791.

Change-Id: I2865bb06a62465836d63196422f5f734f7165386
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11801
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 16:18:47 +00:00
Florian Klink
29eda2d5b2 feat(tvix/cli): add toplevel progress span
This gives some better feedback something is going on, gives a (named)
root for all the progress children we're drawing, and also counts the
time we are in eval.

Change-Id: Ibe81dcebf0a2b59bb0680da62e206eb5270f9d3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11798
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-06-13 16:18:15 +00:00