Commit graph

20589 commits

Author SHA1 Message Date
Florian Klink
5bd48de418 feat(tvix/nar-bridge): add 404 handler for GET/HEAD /nar/…
We currently send 405, and that confuses `nix copy`.
Send a 404 for now, and add a futurework, as we can actually at least do
something more meaningful in case we still have that nar hash in our
LRU, which would avoid some unnecessary uploads in some cases.

Change-Id: If625e9bd0fd6506cb73b88962d889aa08315fcea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11987
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:38:58 +00:00
Florian Klink
5d906054da feat(tvix/nar-bridge): support uploading NAR files
This ingests NAR files into the {Blob,Directory}Service, which are
already part of the AppState.

As we then need to correlate the root node to the uploaded PathInfo, we
need to keep a (short-lived) lookup table from NARHash to root node
around. We insert it into a `LruCache` after the NAR is uploaded, and
use `peek()` to do the lookup, which doesn't update the LRU list.

Change-Id: I48a4c6246bacf76559c5a4ccad2a0bc25c1b7900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11986
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:23:18 +00:00
Florian Klink
861cc1f341 feat(tvix/nar-bridge): init
This adds an implementation of nar-bridge in Rust.
Currently, only the GET parts are implemented.

Contrary to the Go variant, this doesn't try to keep a mapping from nar
hashes to root node in memory, it simply encodes the root node itself
(stripped by its basename) into the URL.

This pulls in a more recent version of axum than what we use in
tonic, causing two versions of http and hyper, however dealing with
`Body::from_stream` in axum 0.6 is much more annoying, and
https://github.com/hyperium/tonic/pull/1740 suggests this will be fixed
soon.

Change-Id: Ia4c2dbda7cd3fdbe47a75f3e33544d19eac6e44e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11898
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-20 16:53:23 +00:00
Florian Klink
0244ae6eaf feat(tvix/castore/protos): return root node after validation
This allows avoiding a `.node.unwrap()`` after validation.

Change-Id: Ieef1ffebab16cdca94c979ca6831a7ab4f6007da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11989
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-20 16:35:07 +00:00
Florian Klink
c64c028366 fix(tvix/store): add missing nar_info::ca::Hash::TextSha256
Also reorder to how it's ordered in the .proto file.

Change-Id: I87c422feac4d12f6a84f99aa889f9c524a9878b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11985
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 16:35:07 +00:00
Brian Olsen
18da8a205e docs: More documentation of nix daemon protocol
Uses `Set` to mark that the given list is actually a set.
The format on stream is no different from `List` but is nice to
know.

Best effort document string formats.

Documents default values for fields not used in old protocol
versions.

Document handshake, AddMultipleToStore format, Export format and
Import paths format.

Change-Id: I6cbe4c7de3770cc5caafc26229f99a07353ee3e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11984
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Brian Olsen <me@griff.name>
2024-07-20 15:33:04 +00:00
Yureka
4801d6bf85 feat(tvix): different service types in one composition
This will be necessary for the PathInfoService composition, as some
PathInfoService implementations require a BlobService & DirectoryService
to ingest into.

Using the Extend trait for creating compositions allows extending the same
composition with configs of various types e.g. BlobStore, DirectoryStore
Generics are moved from the Composition struct to the functions.The storage of
the InstantiatonStates uses the TypeId in the key and a Box<dyn Any> in the
value, which is downcasted to InstantiatonState<T>.

Change-Id: I2af11f26c535029adfb1c62905e0e7c4aaed7b51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11980
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
2024-07-19 15:13:37 +00:00
Yureka
7ccdf6dad5 test(tvix/castore/ObjectStoreBlobService): test correct chunking
Change-Id: I4a8f5cda2705560a6d52a2eb2be0dde35b784c8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11982
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-19 12:57:41 +00:00
Yureka
3c160731ed fix(tvix/castore/blob/object_store): empty chunk list for single chunk
Change-Id: Ie3609ac6184ecead11c4e03073433c26ce129848
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11981
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-19 10:55:47 +00:00
Ryan Lahfa
f0b0a6572f fix(tvix/nix-compat): use buf.filled() to track read bytes
We were wrongly using `buf.initialized()` which contains more than the
filled portion to compute the number of bytes read during a poll call.

This made us go into the trailer reading state too early and finally
failing due to invalid trailer data.

Fixes b/405.

Co-authored-by: Florian Klink <flokli@flokli.de>
Change-Id: I66ba6e2116389e6b97305f85c4e0922195000e45
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11978
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-07-18 22:13:44 +00:00
Yureka
168e4fda59 refactor(tvix): use composition & registry for from_addr
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
2024-07-18 19:19:26 +00:00
Yureka
79317be214 feat(tvix/castore): add sled to composition registry
Change-Id: I03fa8dfabcee14c5f657380f86bb1a7aa00e08ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11977
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-18 19:19:10 +00:00
Yureka
b8415a94d9 feat(tvix/composition): allow creating a blank CompositionContext
this is useful when oneshot-instantiating a store from a single config

Change-Id: I08538fdee1d0bb26b3ae2da7d3b2339b2e93bc0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11975
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-18 18:47:42 +00:00
Yureka
1a6b6e3ef3 feat(tvix/castore): add composition module
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748
Signed-off-by: Yureka <tvl@yuka.dev>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-18 10:42:11 +00:00
Ilan Joselevich
64fd1d3e56 chore(tvix/tools/weave): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: I79b062eddb13f59c4656a62735df1e527fc91a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11970
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 16:47:56 +00:00
Ilan Joselevich
52b8cf8d1f chore(tvix/tools/turbofetch): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: I48cabe514e0cf1dd1f00870bcb7958a2a49663d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11969
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-09 16:47:56 +00:00
Ilan Joselevich
716f5624fe chore(tvix/tools/crunch-v2): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: I90245bad27f6fee01455557bbccc865301a4f1a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11968
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 16:47:56 +00:00
Ilan Joselevich
f52b142035 chore(tvix/tools/narinfo2parquet): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: Id5225e73ec5894bc324f363660472a7efdd803cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11967
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-09 16:47:56 +00:00
Ilan Joselevich
4f7b19701c fix(tvix/{store,castore,build}): Compile tonic w/o config in build.rs
Previously we had to make a mutable Config instance and set bytes and
other values in it because they were not exposed to the builder pattern
(https://github.com/hyperium/tonic/issues/908) but now they are, so we
just set them through the builder.

Change-Id: I8904c6b93f09173b56586024b1ced59d622bce66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11966
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 13:24:22 +00:00
Luke Granger-Brown
6d7713572f chore(3p/gerrit): downgrade mina SSHd again
This breaks it with "ssh_dispatch_run_fatal: Connection to
2a01:4f8:242:5b21:0:feed:edef:beef port 29418: incorrect signature"

Change-Id: I9bafc5fb6d4743ce4f097158ec14fecc791366ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11965
Tested-by: BuildkiteCI
Autosubmit: lukegb <lukegb@tvl.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2024-07-08 09:41:21 +00:00
Luke Granger-Brown
c05bf02a85 chore(3p/gerrit): create buildBazelPackageNG and migrate gerrit to it
This bumps Gerrit to 3.10.0, and also introduces a new mechanism for
building it that should hopefully have some more stable hashes than the
previous bodgery.

In this world, we only cache what we explicitly want to. There are some
hooks implemented for `rules_java` and `rules_nodejs` (before version
6) that force use of local binaries; this means we can drop the use of
the FHSUserEnv and use the java and nodejs binaries provided by nixpkgs
instead.

detzip is deleted; it hasn't been used in yonks.

We also add https://gerrit-review.googlesource.com/c/gerrit/+/431977,
which bumps the SSHd version so that we can have U2F-based SSH keys.

Change-Id: Ie12a9a33bbb1e4bd96aa252580aca3b8bc4a1205
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11963
Reviewed-by: lukegb <lukegb@tvl.fyi>
Autosubmit: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2024-07-08 00:17:56 +00:00
Florian Klink
d17c3d96b6 refactor(tvix): point SSL_CERT_FILE to /dev/null
reqwest wants to be able to read a file of trust roots when constructed,
but as it doesn't actually do any HTTPS connections inside the nix
build, an empty list of trust roots is totally sufficient.

Thankfully /dev/null provides such a file.

Change-Id: I9bd1619b2c9f8ff2a6640d2ac410d4de5b20c2ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-07-07 15:11:29 +00:00
Aspen Smith
89d204d295 feat(tvix/eval): Bump imbl to 3.0.0
Change-Id: I88053fa2df35dc486c51d015f4b2156541c07af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11962
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 15:05:27 +00:00
Aspen Smith
8821746d6c fix(tvix/repl): Share globals and sourcemap across evaluations
Now that we can bind (potentially lazy, potentially lambda-containing)
values in the REPL and then reference them in subsequent evaluations,
it's important that the values to which we construct shared references
are shared across those subsequent evaluations - otherwise, we get
panics due to unknown source map locations, or dropped weak references
to globals.

This change assigns both the globals and the source map as fields on the
Repl after the first evaluation, and then passes those in (to the
EvaluationBuilder) on subsequent evaluations.

On the EvaluationBuilder side, there's some panicking introduced - this
is intentional, as my intent is for the builder to be configured
statically enough that panicking is the best way to report errors
here (it's always a bug to misconfigure an Evaluation, and we'd never
want to handle it dynamically).

Change-Id: I37225697235c22b683ca48a17d30fa8fedd12d1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11960
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-07-07 15:04:26 +00:00
Aspen Smith
01765c3717 test(tvix/cli): Add some additional REPL tests
A couple of already-passing tests covering REPL behavior

Change-Id: Ie21f4abf68ab12827fd15128a8ef810cd8592d07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11959
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-07-07 14:19:17 +00:00
Aspen Smith
0ad986169d test(tvix/cli): Make the REPL testable
Juggle around the internals of the tvix-cli crate so that we expose the
Repl as a public type with a `send` method, that sends a string to the
repl and *captures all output* so that it can be subsequently asserted
on in tests. Then, demonstrate that this works with a single (for now)
REPL test using expect-test to assert on the output of a single command
sent to the REPL.

As the REPL gets more complicated, this will allow us to make tests that
cover that complex behavior.

Change-Id: I88175bd72d8760c79faade95ebb1d956f08a7b83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11958
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 14:19:17 +00:00
Aspen Smith
3a79f93795 refactor(tvix/eval): Construct globals in EvaluationBuilder::build
Construct the Rc<GlobalsMap> for the evaluation as part of
EvaluiationBuilder::build, rather than deferring it until we actually
compile. This changes nothing functionally, but gets us one step closer
to sharing this globals map across evaluations.

Change-Id: Id92e9fb88d974d763056d4f15ce61962ab776e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11957
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-06 15:24:50 +00:00
Aspen Smith
dfe137786c refactor(tvix/eval): Builderize Evaluation
Make constructing of a new Evaluation use the builder pattern rather
than setting public mutable fields. This is currently a pure
refactor (no functionality has changed) but has a few advantages:

- We've encapsulated the internals of the fields in Evaluation, meaning
  we can change them without too much breakage of clients
- We have type safety that prevents us from ever changing the fields of
  an Evaluation after it's built (which matters more in a world where we
  reuse Evaluations).

More importantly, this paves the road for doing different things with
the construction of an Evaluation - notably, sharing certain things like
the GlobalsMap across subsequent evaluations in eg the REPL.

Fixes: b/262
Change-Id: I4a27116faac14cdd144fc7c992d14ae095a1aca4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11956
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-06 15:03:46 +00:00
Aspen Smith
d5964c1d54 chore(tvix/docs): Cross off a TODO
This is done now, as of fc6359463 (feat(tvix/repl): Allow binding
variables at the top-level, 2024-07-04)

Change-Id: I2126cbadeb66625cda93a7cd7ddb31409ec60fd9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11955
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-07-06 03:06:51 +00:00
Ilan Joselevich
7ca32d9f0b refactor(web/tvixbolt): buildRustPackage -> crate2nix
With the recent changes to crate2nix and buildRustCrate in nixpkgs it is
now possible to build tvixbolt via crate2nix like we do for other tvix
crates. We can reuse a lot of the customizations done in //tvix in
tvixbolt to avoid repeating ourselves.

A script for serving tvixbolt locally for testing purposes is also
available now through the .serve attribute of tvixbolt.

This change supersedes https://cl.tvl.fyi/c/depot/+/11821.

Change-Id: I4864df8b75aec73cf5fee2428924ed4cfbb32902
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11952
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 20:19:39 +00:00
Ilan Joselevich
6a7069904e feat(tvix/utils): Add defaultCrateOverridesForPkgs function
This function can be reused across the rest of the repo to make use of
our overriden defaultCrateOverrides with support for tvix crates.

Change-Id: I8c554dece052bd9dd32acac13dab8114933272a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11951
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 20:19:14 +00:00
Ilan Joselevich
63654fbeb1 fix(tvix/utils): Optionally add support for Cargo in filterRustCrateSrc
Previously we would unconditionally add Cargo.toml to the fileset. We
mostly use buildRustCrate in tvix so it does not make sense to add it by
default, instead I made it so you enable cargoSupport if you want
Cargo.{toml,lock} to be added to the fileset.

Change-Id: I5a6016534fc5599e85ab581fe3d9b81e7a24f940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11950
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-05 20:07:09 +00:00
Aspen Smith
fc63594631 feat(tvix/repl): Allow binding variables at the top-level
Allow binding variables at the REPL's toplevel in the same way the Nix
REPL does, using the syntax <ident> = <expr>. This fully, strictly
evaluates the value and sets it in the repl's "env", which gets passed
in at the toplevel when evaluating expressions.

The laziness behavior differs from Nix's, but I think this is good:

    ❯ nix repl
    Welcome to Nix version 2.3.18. Type :? for help.

    nix-repl> x = builtins.trace "x" 1

    nix-repl> x
    trace: x
    1

    nix-repl> x
    1

vs tvix:

    tvix-repl> x = builtins.trace "x" 1
    trace: "x" :: string
    tvix-repl> x
    => 1 :: int
    tvix-repl> x
    => 1 :: int

Bug: https://b.tvl.fyi/issues/371
Change-Id: Ieb2d626b7195fa87be638c9a4dae2eee45eb9ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11954
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-07-05 16:40:35 +00:00
Aspen Smith
ac3d717944 feat(tvix/eval): Allow passing in an env to evaluation
Allow passing in a top-level env, a map from name to value, to
evaluation. The intent is to support bound identifiers in the REPL just
like upstream nix does.

Getting this working involves mucking around a bit with internals - most
notably, locals now only optionally have a Span (since locals don't have
an easy span we can use) - and getting that working requires propagating
some minor hacks to places where we currently *need* a span (and which
would require too much changing now to make spans optional; my guess is
that that would essentially end up making spans optional throughout the
codebase).

Also, some extra care has to be taken to close out the scope in the case
that we do pass in an env, to avoid breaking our assumptions about the
size of the stack when we return from the toplevel

Change-Id: Ie475b2d3dfc72ccbf298d2a3ea28c63ac877d653
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11953
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 16:39:34 +00:00
Florian Klink
af933c177a docs(tvix): update WASM status
crate2nix can now build WASM, and cl/11859 showcases this now.

Even if it's not in the same cargo workspace, we should still migrate
tvixbolt to a crate2nix build.

Remove the part about Build/Store frontends, it was more of an
explanation why we want WASM builds to be nice, it can be tracked in
separate TODOs once more concrete.

Change-Id: If4f5e0994b55520ba70cabefb4fcef9dc17bc394
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11945
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
2024-07-05 16:26:14 +00:00
Aspen Smith
6037888e18 refactor(tvix/eval): Drop LightSpan entirely
This was made unnecessary in c92d06271 (feat(tvix/eval): drop
LightSpan::Delayed, 2023-12-08) because it didn't improve benchmarks as
much as expected and has been vestigial since; this continues the
cleanup by just removing it altogether

Change-Id: I21ec7ae9b52a5cccd2092696a5a87f658194d672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11949
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 03:53:44 +00:00
toastal
8e31088a01 docs(tvix): introduction
Includes hacky, but workable solution for the pronunciation ‘issue’

Karl comes from: http://ipa-reader.xyz/?text=tvɪks&voice=Karl

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Iad7788ec7295902fd2159766a664016c7b1e2ae9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11908
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-04 14:18:09 +00:00
Florian Klink
9ed967791c test(tvix/store): test listing endpoint too
Change-Id: Ia4035aca43cf9d3f7de982dd154715120ba25496
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11927
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-03 12:35:07 +00:00
Florian Klink
eb9a74d966 fix(tvix/store/pathinfo/bigtable): fix listing endpoint
We were wrongly comparing the raw row_key with the store path
digest, but we hexlower-encode the digest in the row key (via
derive_pathinfo_key).

Update the logic to fix that.

Change-Id: I8916d8de9fb8b25a6986d4158faa91ec97c57347
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11926
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-03 12:35:07 +00:00
Simon Hauser
618aacaa61 feat(tvix/tracing): http trace propagation
Introduces a helper function within tvix-tracing that returns a reqwest
tracing middleware that will ingest the traceparent if otlp is enabled.

It is feature flagged in tvix-tracing so not every consumer of that
library automatically has reqwest in its dependencies.

Tested using netcat to verify that the `traceparent` header is there if
otlp is enabled and missing if otlp feature is disabled.

Change-Id: I5abccae777b725f5ff7382e3686165383c477a39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-02 13:43:09 +00:00
Ilan Joselevich
7f8da5e6a9 feat(users/kranzes): add wasm-hello-world
This can be used as a reference for how to build a wasm project with
crate2nix.

Change-Id: Ib4d0db6bf24d8f1dec4734d5f1e8de19212a54cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11859
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-01 19:25:39 +00:00
Vincent Ambo
4b2f3c5454 chore(3p/sources): bump to OpenSSH vulnerability hotfix
See https://github.com/NixOS/nixpkgs/pull/323753 for details.

Changes:

* git: temporarily comment out dottime patch (it doesn't apply, but it's not critical)
* third-party/cgit: use an older git version where dottime patch still applies
* 3p/crate2nix: remove crate2nix patches included in latest release
* tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs)
* tvix: regenerate Cargo.nix
* tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function
* tvix/nix-compat: remove redundant trait bounds
* tvix/glue: use clone_into() to set drv.{builder,system}
* tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327
* toold/depotfmt: expose treefmt config as passthru
* tools/crate2nix: undo some more hacks in the crate2nix-check drv

Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c
Co-Authored-By: Florian Klink <flokli@flokli.de>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 17:42:30 +00:00
Florian Klink
17bdf9a574 feat(tvix/store): add --remote-path-info-service-addr option
This is allows adding a cache in front of tvix-store daemon, and is less
code duplication than cl/11902, means we can probably land that until we
have proper store composition config.

It can be used to provide a tvix-store daemon interface for a Nix HTTP
Binary cache, saving all calculated PathInfo to another PathInfoService
after ingestion.

Change-Id: If141d718c2635f66aa90d46a80fd79c86c07d9ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11903
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 17:14:19 +00:00
Florian Klink
830fdda8d4 feat(tvix/store): use tokio-listener for tvix-store daemon command
This allows binding on unix sockets, as well as systemd socket
activation.

Change-Id: Icf648c4fd0895468c52607deb6397b8b5928102b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11901
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-07-01 15:53:53 +00:00
Florian Klink
f6c759de58 chore(tvix): bump bigtable_rs to latest release
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a
while, no need to use our own checkout here.

Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-07-01 15:53:53 +00:00
Simon Hauser
6a9a4d56a4 feat(tvix/tracing): expose stdout_writer and stderr_writer
Using std::io::{Stdout,StdErr} directly will clobber the output by an
active progress bar. To resolve this issue the exposed writers should be
prefered over `println!` and `eprintln!`.

Change-Id: Ic79465cd4e8b9dad5a138f6b08c5f0de9dcf54a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11860
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 13:55:02 +00:00
Florian Klink
87f38cad61 fix(tvix/boot): use shrunk kernel image
The aarch64-linux version of this is too big for cloud-hypervisor to
accept.

The correct place for this is in
`${kernel}/${stdenv.hostPlatform.linux-kernel.target}`, the other output
is only useful for debugging purposes.

This gets cloud-hypervisor to start booting up the kernel. It still
doesn't boot successfully, but it's getting further.

Part of b/407.

Change-Id: I3fbe52597fb1a6f5728a7f375dd35b50e00b3b1a
Co-Authored-By: Alyssa Ross <hi@alyssa.is>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11905
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Autosubmit: flokli <flokli@flokli.de>
2024-07-01 07:24:52 +00:00
Florian Klink
afbe995aa9 chore(tvix): bump to data-encoding 2.6.0
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-01 06:57:01 +00:00
Florian Klink
ffdc96ca59 feat(tvix/boot/tests): add closure-nixos-s3
This uses the `preStart` mechanism to boot up a minio instance, which
we point blob/directoryservice to.

Change-Id: I9e67b19006cb08ce82fe63c885cb0178e97389e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11896
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 19:23:57 +00:00
Florian Klink
bba64d8762 fix(tvix/castore/object_store): make query pairs object_store opts
We previously called ObjectStoreBlobService::parse_url, which passes an
empty list of options when constructing the ObjectStore.

This is most likely not what we want. The more reasonable thing to do is
pass along the query string (pairs) as options to
`object_store::parse_url_opts`, and remove them from the plain URL we
pass to object_store itself.

Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-29 19:23:57 +00:00