Commit graph

2799 commits

Author SHA1 Message Date
Florian Klink
50150fa3c8 feat(tvix/glue): add eval-fail-* support to glue tests
Add some eval-fail-fetcht-*.nix test cases.

Change-Id: Ic02beabf120d1de8df6f9c098d5a074127069f81
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11880
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 11:17:30 +00:00
Florian Klink
89361b2a7f fix(tvix/tracing): make cargo check and clippy happy
In case the otlp feature is not enabled, these generate warnings during
`cargo check`.
Fix by moving some imports into their functions, or using the
fully-qualified name (and one #[allow(unused_mut)])

Change-Id: I5afd89dcd4c772b6002cebdd5d0469932eacfdac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11873
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 06:27:15 +00:00
Florian Klink
a0993e7304 feat(tvix/eval): add ErrorKind::UnexpectedArgumentBuiltin
Change-Id: Ieb091b32aad566719fbe8604c4a589f5ccaaf6b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11877
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
970e2a045b refactor(tvix/eval): rename UnexpectedArgument{,Formals}
There's other places where unexpected arguments can be provided, like
in builtins.

Make space for that error type.

Change-Id: Ic831497a3a1dd36a3a184bedadcf1374bf0ae6db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11876
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-26 04:51:31 +00:00
Florian Klink
ea6f511241 refactor(tvix/glue): return a parsed Url in NixFetchArgs
The only two consumers (fetchurl, fetchtarball) of these do try to parse
it as URL, so do it in the helper.

Update url_basename to take a &url::URL, not a &str.

Also update the test to use rstest for the fixtures to reduce some
boilerplate there.

Change-Id: I1f85fe2803060dc4423e673cb7b9f9bf799d09b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11875
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
540e566900 refactor(tvix/glue): take &CAHash, not CAHash
We use a bit less cloning that way.

Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-26 04:51:31 +00:00
Florian Klink
78eb22c54d feat(tvix/glue): handle regular file at builtins.path import
If builtins.path is passed a regular file, no filtering is applied.
We use the just-introduced file_type function in the EvalIO trait for
that.

This means, we don't need to pass through filtered_ingest, and can
assemble the FileNode directly in that specific match case.

This also means, we can explicitly calculate the sha256 flat digest,
and avoid having to pipe through the file contents again (via
blob_to_sha256_hash) to construct the sha256 digest.

Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
080654aaf9 feat(tvix/eval): add file_type method
This allows peeking of the type at a given path.

It's necessary, as an open() might not fail until you try to read()
from it, and generally, stat'ing can be faster in some cases.

Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Ilan Joselevich
04f04cfc27 refactor(tvix/eval): drop placeholder impls of builtins in glue
The builtins `placeholder` and `filterSource` are now implemented in
tvix-glue and so we no longer need their placeholder "implementations" in
tvix-eval.

Change-Id: Ibf161ccf1ad40103e9fbb688e9f6be58e7772af1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11867
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-22 21:50:30 +00:00
toastal
bf541acf29 docs: use admonish for callouts
Change-Id: Icec7321a49c40e7f7bcb1e4b307f13b8db4eac25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11862
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-21 08:08:17 +00:00
Florian Klink
8dde022d31 docs(tvix): drop done TODOs in the O11Y section
OTLP shutdown behaviour was fixed in cl/11803, and tracy landed in
cl/10952 and cl/11815.

Change-Id: I310dfcc41ba62ce5b006e75a35feb839327b0e7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11866
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Florian Klink
7f625e2fc1 refactor(tvix/*store): remove some trait bounds
We don't need to require these things for these impl blocks yet.

Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Simon Hauser
639a00e2ab feat(tvix/tracing): gRPC trace context propagation
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.

The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.

This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.

Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-20 19:21:01 +00:00
Simon Hauser
2b20d8d82d feat(tvix/castore): instrument tokio task with current span
By default tokio::spawn does not instrument the spawned task with the
current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do
this manually for all tokio::spawn functions in functions that are
instrumented.

Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 15:48:14 +00:00
Simon Hauser
bd8d74a3ee feat(tvix/tracing): optional progressbar
Disable the progressbar on default and provide a interface for
optionally enabling the progressbar.

Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 10:28:54 +00:00
Florian Klink
1446e3be99 docs(tvix/docs): document nar renderer optimization potential
Change-Id: I747ab5c8ba3424828b1c88e0353170ad0efab3a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11849
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 10:28:33 +00:00
Florian Klink
8137077a74 feat(tvix/store/utils): detect gRPC NAR calculation service
We were currently always using SimpleRenderer, which would mean the
client would download every blob locally to calculate the checksum,
which of course is very slow.

Detect this special case and create a second instance (and client) for
now.

Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 10:28:26 +00:00
Ilan Joselevich
9fd601e53d chore(tvix): patch crate2nix to drop darwin dontStrip
This patch has already been applied on upstream but there hasn't been an
official release yet so we patch it ourselves.

Upstreamed patch: 0209f258cd

Follow-up of: https://cl.tvl.fyi/c/depot/+/11856

Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-18 14:13:11 +00:00
Ilan Joselevich
92c5433304 chore(tvix): move filterRustCrateSrc to utils.nix
Change-Id: Ib6bae1ea0457d3309a6ec1e08b9e4b320523c161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11857
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 15:26:19 +00:00
Ilan Joselevich
ac51803ad8 chore(tvix): Remove no longer needed libiconv addition for darwin
Since https://github.com/NixOS/nixpkgs/pull/220212 libiconv is now
automatically added for Darwin.

Change-Id: I2aaa266bcbc4d5a8256131aed36b4c2d040a8758
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11856
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-17 15:08:10 +00:00
Ilan Joselevich
210b4277b6 chore(tvix): Use callPackage instead of import on Cargo.nix
Since crate2nix v0.11 the new preferred way of importing Cargo.nix is
with callPackage instead of import.

Change-Id: Ifcc5c21e6c101cb9d0b3f1e8025b5ddb8cf99f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11855
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:46 +00:00
Ilan Joselevich
fbe29615df chore(tvix): Remove no longer needed rlib stripping fix for darwin
This issue was fixed in https://github.com/NixOS/nixpkgs/pull/255900

Change-Id: I4f569099e62c4284bdea18a512387735f139e702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:04 +00:00
Florian Klink
28b692fd50 feat(tvix/tvix-store): improve progress bars
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.

Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.

Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.

Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 12:57:34 +00:00
Florian Klink
cfab953094 feat(tvix/store): display progress for NAR calculation
This is currently still taking a noticeable amount of time, so make sure
we show it is happening.

Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 09:47:54 +00:00
Florian Klink
71a29ceff4 docs(tvix/store/import): update import_path_as_nar_ca docstr
Change-Id: I697172220efe73b2a215185027df09f815d32627
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11846
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 08:49:52 +00:00
Florian Klink
4be250109c refactor(tvix/store): drop FUSE multithread workaround for Darwin
The underlying issue in macFUSE has been fixed in
https://github.com/osxfuse/osxfuse/issues/974.

Bump our `macfuse` in nixpkgs to a version containing the fix.

This can be removed while our nixpkgs pin is bumped past
https://github.com/NixOS/nixpkgs/pull/320197.

Change-Id: Ia0e644fb13198e45018b0a218647ef211acf4df1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11824
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-16 12:02:11 +00:00
Florian Klink
7e42b4f314 fix(tvix/store/bin): fix shutdown behaviour for FUSE
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.

So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.

Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.

Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.

Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.

Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-06-16 12:02:11 +00:00
Yureka
452299dcd2 feat(castore/directory): add cache combinator
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 08:42:53 +00:00
Yureka
30e72d2d52 refactor(castore/directory/objectstore): use new order validation logic
Use the new helper introduced in CL 11708 instead of rolling our own.

Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 08:40:20 +00:00
Yureka
daada1b2fa refactor(castore/directory): separate order logic from ClosureValidator
ClosureValidator was previously only suitable for a very narrow use case:
Validating incoming uploads, which are in leaves-to-root order.
This is because the ordering validation was hard-wired into the add()
function.
This
- Re-name ClosureValidator to DirectoryGraph, which is more suitable
  since it actually stores the Directory structs and is drained in the end.
- Move the ordering-related logic to a separate OrderValidator, which
  can be used independently.
- re-write DirectoryGraph to be a general purpose
  validator which can accept the input in both orders
  and can be drained in both orders as well.

This means the DirectoryGraph and OrderValidator can now serve
multiple new purposes:
- Validating the incoming closure on the client while downloading.
- Validating the incoming closure downloaded in a caching layer from the
  `far` cache, and re-order it for insertion into the `near` cache.

Change-Id: I2b4b226348416912d7a31935bec050e53d911b70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 05:01:07 +00:00
Florian Klink
01aadace80 feat(tvix/cli,store): add tracy feature
If compiled with this features, this emits packets compatible with the
[Tracy](https://github.com/wolfpld/tracy) format.

Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 05:24:37 +00:00
Ilan Joselevich
cd0b92419a docs(tvix/TODO): dibs on crate2nix WASM
Change-Id: Idde9aaa31ecaa9570559ffe3271bc23b7a71133c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11820
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 04:02:33 +00:00
Florian Klink
86928b5e7f docs(tvix/TODO): add builtins.path roundtrip for flat
Change-Id: I8291d2fdfa2ea31ec24c82dcfb447fa9eaa3a961
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11819
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-15 04:02:33 +00:00
Florian Klink
6e321c615c docs(tvix/*store): add some internal links
We can now cross-link between pages, and it actually works.

Change-Id: Id8bd85111672c687118db9adae7f5066f22441f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11817
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 03:56:11 +00:00
Florian Klink
46585bee15 docs(tvix/TODO): drop builtins:fetchurl item
This has been implemented in cl/11786, cl/11790 and cl/10975.

This includes support for the `executable` and `unpack` variants of it.

Change-Id: I5bcad5b8e86b9fd341219a59ba86660b692e66e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11818
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 01:01:04 +00:00
Florian Klink
6b6a34065e feat(tvix/tracing): add tracing-tracy support
This introduces another feature flag, "tracy" to the `tvix-tracing` crate.

If enabled (not enabled by default), it'll add an additional layer
emitting packets in a format that https://github.com/wolfpld/tracy can
display.

I had to be a bit tricky with the combinatorial complexity when adding
this, but the resulting code still seems manageable.

Change-Id: Ica824496728fa276ceae3f7a9754be0166e6558f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10952
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-14 19:33:44 +00:00
Florian Klink
d25f047b9d refactor(tvix/tracing): move otlp setup into helper function
Having all this in the main control flow makes it a bit hard to read.
Moving it into a helper function makes it a bit cleaner.

Change-Id: Ibdb739dbd1e013b4f8c4aaf9b036a6bd556a1871
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11814
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-14 14:52:19 +00:00
Simon Hauser
118e69d81d fix(tvix/tracing): reduce the error logs of otlp if collector is offline
The problem is that opentelemetry_otlp tonic batch exporter tries to
exports if either the `scheduled_delay` or if the
`max_export_batch_size` is reached. Per default the
`max_export_batch_size` is set to 512 spans, which means that we try to
export these spans once that counter is reached. Each export will then
try to connect to the exporter (if that not already happening) and will
result in a `tcp connect error`.
Increasing the max_export_batch_size to 4096 will then ensure that the
export only happens if the `scheduled_delay` is met after the 10
seconds.
`max_queue_size` is also increased, because `max_export_batch_size`
should not be greater than `max_queue_size`, so similar to the default
config its set to `max_export_batch_size * 4`.
This will reduce the amount of tries to otlp if the collector is not
available and otlp enabled.

Change-Id: Ic3430006e8a104fa3b34d274678cae55b3620ce9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11791
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 10:20:31 +00:00
Simon Hauser
a857a2b978 feat(tvix/tracing): apply EnvFilter to all layers
Currently we apply the EnvFilter only to the stderr output writer.
This didn't affect any other layer, like the otlp layer, causing spans
from `h2`, `tokio_util` or other third party crate dependencies to be
always sent out via OTLP.

This changes that behaviour, applying EnvFilter to all exports, leading
to a lot less spans being exported.

Change-Id: I9f3a7233e9d0aeaa81fe08914579f0b3c80d134e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11813
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 10:18:59 +00:00
Simon Hauser
fa7ed39bf4 feat(tvix/tracing): correctly close otlp on exit
Provide a new interface for forcing a flush of otlp traces and use this
interface to shutdown otlp prior to exiting tvix-store, either if the
tool was stopped with a SIGTERM or ended regularly.
This also fixes an issue where traces were not even exported if for
example we just imported 10 paths and never even emitted more than 256
traces. The implementation uses a mpsc channel so a flush can be done
without having to wait for it to complete. If you want to wait for a
flush to complete you can provide a oneshot channel which will receive a
message once flushing is complete.

Because of a otlp bug `force_flush` as well as
`shutdown_tracer_provider` need to be executed using `spawn_blocking`
otherwise the function will deadlock. See
https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335

Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 09:34:51 +00:00
Florian Klink
5077ca70de chore(tvix/eval): move eval docs to tvix/docs
Change-Id: I75b33c43456389de6e521b4f0ad46d68bc9e98f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11809
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-06-14 08:00:34 +00:00
Florian Klink
6947dc4349 chore(tvix/docs): move [ca]store docs to tvix/docs
Change-Id: Idd78ffae34b6ea7b93d13de73b98c61a348869fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11808
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2024-06-14 08:00:34 +00:00
Florian Klink
adc7353bd1 feat(tvix/docs): add d2 support
This provides a nicer syntax to draw some diagrams.

Change-Id: I2ab580e61399a45f01ed422ee29382e860753ffa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11807
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-06-14 07:04:23 +00:00
Florian Klink
4df3284dd0 feat(tvix/docs): add mdbook-admonish
This provides a plugin for callouts.

It needs to have additional CSS file added to `book.toml`,
which can be (re-)generated by `mdbook-admonish install`.

See https://github.com/tommilligan/mdbook-admonish/issues/171#issuecomment-2009166079
for more context.

Use it by adding a warning one to the architecture document.

Change-Id: I75c9a33d00acb603c6da10d3f9ce3485731c1672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11805
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
2024-06-14 07:04:23 +00:00
Florian Klink
9d44da01ee fix(tvix-glue/fetchers/decompression): propagate error with ?
Change-Id: I5e9d4f067f09307cb6f990ed6eca48345e2fb342
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11810
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-13 21:07:40 +00:00
Florian Klink
b992ca49a6 fix(tvix/glue/tvix_store_io): also populate input sources
These also need to be present in the input nodes of the BuildRequest.

Change-Id: Ie9b957805e42f766002581adc6182a6543c5333b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11802
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 19:59:28 +00:00
Florian Klink
c30fded880 chore(tvix/docs): move daemon protocol to its own chapter
… and add an introductionary page at the root.

Change-Id: I89ce69a4221c332b7c0d99ac26b4e26bfd248341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11806
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-13 19:23:20 +00:00
Florian Klink
d947f61d36 fix(tvix/glue/tvix_store_io): disable concurrent fetches for now
We need some shared queue, preventing the same fetches/builds from
getting triggered multiple times unnecessarily.

Change-Id: I7c4a3c66db558f5cccd66865b170242b758e3e02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11800
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-13 16:19:18 +00:00
Florian Klink
79bfa931ed feat(tvix/tracing): set release_max_level_debug for tracing
This allows explicitly opting in to get DEBUG-level log lines, by
setting RUST_LOG.

It currently also causes traces to be emitted in all cases, so we might
do some runtime filtering there too, as discussed in cl/11791.

Change-Id: I2865bb06a62465836d63196422f5f734f7165386
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11801
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 16:18:47 +00:00
Florian Klink
29eda2d5b2 feat(tvix/cli): add toplevel progress span
This gives some better feedback something is going on, gives a (named)
root for all the progress children we're drawing, and also counts the
time we are in eval.

Change-Id: Ibe81dcebf0a2b59bb0680da62e206eb5270f9d3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11798
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-06-13 16:18:15 +00:00
Florian Klink
7f29cab1cc fix(tvix/glue/tvix_store_io): distinguish waiting and building
We immediately reported "Building", even though then populated necessary
inputs, which looked a bit odd. Make it clear we're still waiting, and
update the spinner message once we have all inputs we were waiting for.

In the future, we might want to have separate spans for this, so the
timer gets reset, but that's something for later.

Change-Id: Ic22c9a906d0e7e7179c5ee328162401261efc224
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11799
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 16:18:14 +00:00
Florian Klink
99c5a2e8bc feat(tvix/glue): report progress on all fetches, use progress bars
This should also report progress on fetches which we couldn't delay
until actually having to IO into them, like `builtins.fetchurl` calls
without a upfront-provided hash.

While at it, upgrade the progress spinners to progress bars, which
increment if we know the size of the fetch.

Change-Id: Ic3f332286d8bc2177f3d994ba25b165728d4b702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11797
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-06-13 16:16:42 +00:00
Florian Klink
7ee55c293c fix(tvix/glue/tvix_store_io): use same case for progress messages
"Fetching" was uppercase, "building" was lowercase.
Let's make this consistent.

Change-Id: I11c16f1a7d2057ada4d057e553a4ceaa59597f26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11796
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-13 11:59:58 +00:00
Florian Klink
08f794e7e7 fix(tvix-glue/fetchers): remove unused allow dead code
This isn't unused anymore, we can remove the
`#![allow(dead_code)]`.

Change-Id: Ia2048394a60161fbfd4ac2a2b64d03350526faa4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11795
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-12 23:43:54 +00:00
Florian Klink
99df0e73f4 chore(tvix/castore): drop unused indicatif dependency
Change-Id: I9fbd32596fbffe09a590a50044a4f1d074a424c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11794
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-06-12 23:43:54 +00:00
Florian Klink
ddd88a589b feat(tvix/glue/tvix_store_io): show progress info
In `store_path_to_node`, in case we need to build or fetch something,
render a progress bar, using the spinner for now.
We can upgrade this to a progress *bar* later.

Change-Id: I4a7cf5ef8f639076f176af9b39d276be3f37c8ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11793
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-12 23:43:54 +00:00
Florian Klink
842d6816bf feat(tvix/glue): support builtin:fetchurl
nixpkgs calls <nix/fetchurl.nix> during nixpkgs bootstrap.

This produces a fake derivation with system = builtin
and builder = builtin:fetchurl, and needs to download files from the
internet.

At the end of the Derivation construction, if we have such a derivation,
also synthesize a `Fetch` struct, which we add to the known fetch paths.

This will then cause these fetches to be picked up like all other
fetches in TvixStoreIO.

Change-Id: I72cbca4f85da106b25eda97693a6a6e59911cd57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10975
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-12 22:31:17 +00:00
Florian Klink
5963133a62 feat(tvix/glue/fetchers): add NAR fetching infrastructure
The magic <nix/fetchurl.nix> derivation can cause two other types of
fetch to happen, one that unpacks NAR files, and another one that puts a
file as an executable at the store path root.

This adds the necessary enum type and path calculation logic for it to
the fetcher code.

It also adds code to do the actual NAR fetching. The executable case is
still stubbed out.

Change-Id: I79103fd58c7e22ad7fde34efa5e2d89cad7d5a0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11790
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-12 10:30:40 +00:00
Florian Klink
6073ef9c6f refactor(tvix/glue/fetchers): move store_path() tests here
Move the part asking a fetch for its store_path() to the place where
this function is defined, and add some more test cases.

Change-Id: I96f326d0d56aa5835f23274b8cd1b1afe3724153
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11789
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-12 10:30:40 +00:00
Florian Klink
e1d3fa240a refactor(tvix/glue/fetchers): use named field for structs
This allows giving more self-speaking names, as well as documenting each
field individually.

Change-Id: Ide164d684b7f819aac279cc8e657c02fc24d093f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11786
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-12 10:30:40 +00:00
Florian Klink
a7bf5b975f test(tvix/glue/fetchurl): add output path tests for fetchurl
These are all derived from the FOD representation of the ATerm
serialization, so no new code necessary to calculate these hashes
correctly.

Change-Id: I8c5dacd2a8de3d5042c9dd56910511c19e1b9b31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11788
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-12 10:30:30 +00:00
Florian Klink
ff40332864 refactor(tvix/store/nar/import): add ingest_nar_and_hash
This wraps ingest_nar, but also keeps track of the number of bytes read,
and calculates the sha256 digest of it.

Make use of it in the NixHTTPPathInfoService, where this code is coming
from.

Change-Id: I9c54e93d3ec8ed9ede87aed43e04d114fb06897b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-11 22:47:31 +00:00
binarycat
beb7f57c73 fix(tvix/eval): handle builtins.split matching the empty string
This prevents the following statements from looping endlessly:

```
builtins.split "(.*)" ""
builtins.split "([abc]*)" "abc"
builtins.split "(.*)" "abc"
builtins.split ".*" ""
```

Cover these (and some more examples) in the test suite.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Ibd339f971e0f4e3e5c229816e2be5a8e3836fec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11743
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-11 14:59:02 +00:00
Florian Klink
154e0d71e0 docs(tvix/docs/TODO): document ChunkService split idea
Change-Id: Ie9c88b0d14902c642e2d3d6603265688eef0e10d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11755
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-11 11:59:29 +00:00
Simon Hauser
825d498908 feat(tvix/tracing): introduce common tvix-tracing crate
Introduce a new common crate that contains tracing boilerplate which then
can be used in the cli, tvix-store and tvix-build crates.
It has otlp as an optional feature, which is currently only used by
tvix-store.

Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-10 16:35:08 +00:00
Florian Klink
11a6ff7706 docs(tvix/docs/TODO): document attempt to add more granular errors
Change-Id: I46f99d15a4ce9c98dcdad822d3032bda3b09c12f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11754
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-10 16:31:34 +00:00
Florian Klink
d3bc358bbc docs(tvix/docs/TODO): document crate2nix for WASM attempts
I tried moving web/tvixbolt to tvix/tvixbolt, and adding it to the cargo
workspace.

I then made `crates =` in `default.nix` a function accepting `pkgs`
(so we can pass in another nixpkgs for some invocations), and then
constructed another nixpkgs and crates instance like this:

```
  pkgs-wasm = (import pkgs.path {
    localSystem = localSystem;
    crossSystem = {
      system = localSystem;
      rustc.config = "wasm32-unknown-unknown";
    };
  }
  crates-wasm = (crates pkgs-wasm);
  tvixbolt-test = crates-wasm.workspaceMembers.tvixbolt.build;
```

… leading to the architecture build failures described in the TODO.

Change-Id: I32112d75f8c098d9810ca52b2d07cd76fae8d8d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11777
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-10 12:03:26 +00:00
Ben Webb
23e0973cdf docs(tvix): fix some typos across various documents
Fix some typos found while reading various documents, mostly those
relating to the castore.

Here is a summary of the edits.

- fix broken link between documents in the store and castore directories
- clarify expression in castore's data model document that indicates
  that the *name* of each child node of a directory must be unique
  across all three lists of children
- add missing closing parenthesis in castore's data model document
- replace "how" with "what" in the phrase "unclear how a ... would even
  look like" in castore's why-not-git-trees document
- remove unnecessary articles in castore's blobstore chunking document
- add missing "y" to "optionall" in eval's compilation of bindings
  document

Change-Id: I1997ea91bb4e9c40abcd81e0cde9405968580ba6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11763
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-08 21:17:56 +00:00
sterni
71435d9546 chore(3p/sources): Bump channels & overlays
* agenix has not been updated (https://github.com/ryantm/agenix/pull/241).
* wasm-bindgen bumped to 0.2.92 in Rust WASM projects
* 3p/lisp: port lispPackages from stable channel

  The Lisp package set we are using (`pkgs.lispPackages`) is the "old
  old" package set, whereas we were supposed to have been using
  `pkgs.lispPackages_new` (which is the "old new" package set).

  Either way we missed that train, and now there's a "new new" package
  set, but with a twist: Lisp packages in nixpkgs are now tied to
  their compilers, so the most generic way to access them seems to be
  from `pkgs.sbclPackages`.

  Switching to the packages from the "new new" package set doesn't
  work: Lots of stuff stops building if we just switch the sources
  over, and not everything is trivially fixable.

  For now we stay on the lispPackages from the stable channel. We need
  to look into the migration later.

  Or rewrite panettone.
* tvix: update generated protobuf files
* 3p/nixpkgs: pick trunk from stable channel; newer versions try to
  read files and do network I/O during build, but don't print enough
  details in error messages to figure out why.
* 3p/overlays: remove tdlib override (nixpkgs is currently new enough)
* 3p/overlays: override telega.el sources while updates are lagging in
  nixpkgs
* users/flokli/ipu6-softisp: update firmware paths, which NixOS now
  stores zstd-compressed.

Change-Id: I5a7a6c8b5d0688461bca92b9e6d654356d3a1cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11711
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-08 15:20:48 +00:00
Aspen Smith
d0ab3c8d15 feat(tvix): Switch to jemalloc
Switch tvix to using jemalloc as the default global allocator on
supported (eg, non-msvc) platforms. This gives a pretty reasonable
performance boost basically for free:

int                     time:   [24.361 µs 24.386 µs 24.418 µs]
                        change: [-19.355% -18.859% -18.527%] (p = 0.00 < 0.05)
                        Performance has improved.

merge small attrs       time:   [37.201 µs 37.328 µs 37.442 µs]
                        change: [-24.609% -24.266% -23.982%] (p = 0.00 < 0.05)
                        Performance has improved.

merge large attrs with small attrs
                        time:   [20.030 ms 20.135 ms 20.251 ms]
                        change: [-4.2811% -3.2549% -2.3807%] (p = 0.00 < 0.05)
                        Performance has improved.

hello outpath           time:   [967.91 ms 974.07 ms 983.33 ms]
                        change: [-5.5008% -4.4552% -3.4099%] (p = 0.00 < 0.05)
                        Performance has improved.

Change-Id: I6c6e6e3295ec2fca01ea28dc37bcb201cd811767
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10851
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-06 10:29:33 +00:00
Florian Klink
08e520b7cc fix(tvix/glue): drop tracing-indicatif from deps
We don't use this in tvix-glue currently. A previous version of cl/11747
did, and I forgot to remove it afterwards.

Change-Id: I6191ffc60450f3e79f22a3aca55246e3956ee9cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11756
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
2024-06-06 10:14:20 +00:00
Florian Klink
20513e7a52 feat(tvix/store/bin): add progress bar infrastructure
This adds the tracing-indicatif crate, and configures it as a layer in
our tracing_subscriber pipeline to emit progress for every span that's
configured so.

It also moves from using std::io::stderr to write logs to using their
writer, to avoid clobbering output.

Progress bar styles are defined in a lazy_static, moving this into a
general tracing is left for later.

This adds some usage of this to the `imports` and `copy` commands.

The output can still be improved a bit - we should  probably split each
task up into a smaller (instrumented) helper functions, so we can create
a progress bar for each task.

Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-06 09:43:24 +00:00
Ilan Joselevich
9b77ce9f8f fix(tvix/crate2nix-check): make drv less likely to be cached
The derivation name for the check will now be calculated from the hash
of all Cargo related files (including in subdirs), this makes it less
likely for the drv to be cached and for CI to miss an outdated
Cargo.lock.

Change-Id: I900e9355be3f8a9d6f01162e8ef0da4d8901af30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11753
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-05 19:05:33 +00:00
Ilan Joselevich
34d93f1d96 fix(tvix): make rstest pick up new test case files
Context: https://github.com/la10736/rstest/issues/256

Cargo will now cause a rebuild whenever a new test case file is added.
Previsouly running `cargo test` after adding a new test case resulted in
that case not being picked up and ignored.

Change-Id: Ibfc420b5bfe3f3ee41d3ebd3fb9d248819fa6ed9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-05 17:52:17 +00:00
Aspen Smith
72b9a126b8 feat(tvix/glue): Implement builtins.storePath
This one's relatively simple - we just check if the store path exists,
and if it does we make a new contextful string containing the store path
as its only context element.

Automatic testing seems tricky for this (I think?) so I tested it
manually:

tvix-repl> builtins.storePath /nix/store/yn46i4xx5alh7gs6fpkxk430i34rp2q9-hello-2.12.1
=> "/nix/store/yn46i4xx5alh7gs6fpkxk430i34rp2q9-hello-2.12.1" :: string

Change-Id: I8a0d9726e4102ab872c53c2419679c2c855a5a18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11696
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-05 17:50:15 +00:00
Florian Klink
c3572048d5 docs(tvix/docs/TODO): expand Store Composition config section
Write down some of the thoughts after brainstorming with yuka.

Change-Id: I01c94474dc643b8c4993db80e50d3ec65f5c17f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11749
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-06-05 17:44:12 +00:00
Ilan Joselevich
9371f97449 fix(tvix): update Cargo.nix as it misses a default feature
Change-Id: I30d91a9aca9f5546ae5e1fa5c55ee389695ae085
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11752
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-05 17:04:05 +00:00
Florian Klink
0ea55c767a docs(tvix/docs/TODO): extend O11Y section
Expand on tvix-tracing crate strategy, add some more context regarding
OTLP and span propagation.

Change-Id: Ice55c116c20aaf60531100465192ce11969551ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11750
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-05 08:51:38 +00:00
Ilan Joselevich
1b39d5868a feat(tvix): add CI targets for checking crate features powerset
Closes: https://b.tvl.fyi/issues/401

With this change all crate features (and their combinations) will be built and
tested in CI.

From now on, when adding/removing a Cargo feature for a crate,
you will want to add it to the features power set that gets tested in CI.
For each crate there's a default.nix with a `mkFeaturePowerset` invocation,
modify the list to include/remove the feature.
Note that you don't want to add "collection" features,
such as `fs` for tvix-[ca]store or `default`.

Change-Id: I966dde1413d057770787da3296cce9c1924570e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11717
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-03 16:35:51 +00:00
Ilan Joselevich
c93849a3fc fix(tvix/nix-compat): require async and async,wire for some tests
Makes the following build:
  `cargo test --no-default-features --features async`
  `cargo test --no-default-features --features wire`

Change-Id: I47ba0c944f08895f67ed3b861706ef2e4ba384b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11739
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-03 16:35:51 +00:00
Aspen Smith
c2f649f62e feat(tvix/repl): Add a command to recursively print
Add a command, :p, to evaluate an expression and recursively print the
result, as if `--strict` had been passed on the command line.

Demonstration of this working:

    ❯ cargo r --bin tvix
        Finished dev [unoptimized + debuginfo] target(s) in 0.27s
        Running `target/debug/tvix`
    tvix-repl> { x = (x: x) 1; }
    => { x = <CODE>; } :: set
    tvix-repl> :p { x = (x: x) 1; }
    => { x = 1; } :: set

Change-Id: I1a81d7481160c30d2a4483c6308e25fa45f2dfdf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11738
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-01 09:04:59 +00:00
Aspen Smith
4a489d930c feat(tvix/repl): Add a help command
Add a command to display help for the REPL, which can be either :? or
:h.

Change-Id: Ifdfd8c31130ca5afcde05a4c4276b768eb54c06f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11737
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-01 09:04:59 +00:00
Aspen Smith
55ec06b26b feat(tvix/repl): Implement :q, to quit
Change-Id: Ib3b314b21f4d8c30bfd674e79771179d51d4f2e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11736
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-01 09:04:59 +00:00
Aspen Smith
70c4b512c2 refactor(tvix/repl): Abstract out REPL command handling
Prepare for introducing additional REPL commands by splitting out
the *parsing* of the repl commands, which returns a new ReplCommand
type, from the actual *handling* of the commands.

Change-Id: If81a53c1e2d90204d26ce3bb2ea9eebf7bb3fd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11734
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-01 09:04:59 +00:00
Aspen Smith
23589a1db3 refactor(tvix): Break out REPL into its own module
In preparation for adding some new functionality to the tvix REPL, break
it out into its own module.

Change-Id: I4fb78320e92562e3474a3724536cb22c1d893e57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11733
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-01 09:04:59 +00:00
Aspen Smith
96ce9aea04 feat(tvix/repl): Support multiline input
Transparently support multiline input in the Tvix REPL, by handling the
UnexpectedEOF error returned by the parser and using it to progressively
build up an input expr over multiple iterations of the REPL's outer
loop.

This works quite nicely:

❯ cargo r --bin tvix
   Compiling tvix-cli v0.1.0 (/home/aspen/code/depot/tvix/cli)
     Finished dev [unoptimized + debuginfo] target(s) in 1.72s
     Running `target/debug/tvix`
tvix-repl> { foo
         >   =
         >   1;
         > }
=> { foo = 1; } :: set
<press up arrow>
tvix-repl> { foo
  =
  1;
}
=> { foo = 1; } :: set

Change-Id: Ib0ed4766b13e8231d696cdc27281ac158e20a777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11732
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-01 09:04:59 +00:00
Ilan Joselevich
be624daf13 fix(tvix/eval): nix_tests.rs's eval_test requires impure flag
This fixes the following command:
'cargo test --no-default-features --features nix_tests'

Change-Id: I9883c39e1e428c72a0e7e0b75a73c8ed734abd3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11740
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-30 21:52:16 +00:00
Florian Klink
5405ed9dc6 fix(tvix/eval): proptests require arbitrary feature
`cargo test --no-default-features` fails, if we don't conditionalize
this on the `arbitrary` feature too.

Change-Id: I81a277810119fed0cfc37c942c422f731aa14b2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11726
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-30 16:18:30 +00:00
Florian Klink
baa39d9d09 fix(tvix/eval/tests/one_offs): test_source_builtin can be pure
`Evaluation::new_impure()` would require the test to be impure, but
there's nothing in this test specifically requiring us to make use of
impure features.

Change-Id: Idb24981195d1a94f51053ae04403eb5f0e27f3d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11725
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-30 16:18:30 +00:00
Florian Klink
c83f2e765c fix(tvix/eval/tests/nix_oracle): tests are impure-only
This uses StdIO, which is only available when the `impure` feature is
enabled.

Change-Id: I039b1f45f6619dd099fa943e58322ff521482dfa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-30 16:18:30 +00:00
Florian Klink
2c628f6e1c refactor(tvix/eval): move nix_tests to separate module
Gate this behind the nix_tests feature flag (which wasn't applied
consistently).

Also, at least right now all of these use StdIO internally, either by
creating it on their own, or through the `eval_test` helper.

Once we have some proper classification system for tests we can probably
split this up further, but for now only run them if the impure feature
is enabled.

Change-Id: I3236cf09b3391585df99073367c4e4832c5e7898
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11723
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-30 16:18:30 +00:00
Florian Klink
543c103903 fix(tvix/eval/vm/generators): allow unused rq_{path_exists,read_dir}
These VM requests are only emitted by code gated behind the impure feature.

To prevent warning from popping up when building without default
features, allow these to be unused if `impure` is not enabled.

Change-Id: Id871a5215e9a0f09aa78edecdd111369ee7ffe34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11722
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-30 16:18:30 +00:00
Florian Klink
6fb542aae6 fix(tvix/eval/nix_search_path): gate tests on impure feature
These use StdIO, which is only available if the impure feature is
enabled.

Change-Id: I18d8e191a7eba6ba5bd59f43631973eaa796c7bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11721
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-26 19:53:51 +00:00
Florian Klink
db68c104af fix(tvix/eval/io): OsStringExt and std::fs::File import are impure-only
These are only needed when building with the impure feature enabled.

This removes some warnings when building with --no-default-features.

Change-Id: I3139d9133d4846aeb1b1b5f3830c0d078d047292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11720
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-26 19:52:49 +00:00
Florian Klink
8089682bb5 fix(tvix/nix-compat): async nar reader requires wire feature
It uses it internally.

Change-Id: I8cb8fc9f567260d57f3a203407333d83beddf537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11719
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-26 19:52:49 +00:00
Florian Klink
ca542440a5 fix(tvix/build): tonic-reflection feature needs castore with it too
If building tvix-build with the tonic-reflection feature, it needs
to import `tvix_castore::proto::FILE_DESCRIPTOR_SET`, which is only
available if tvix-castore is built with the `tonic-reflection` feature.

Change-Id: I355b4c5b4c1333d5cc56335de47ad5d2f1db6337
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11716
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2024-05-26 19:46:47 +00:00
Florian Klink
9586e5c30d refactor(tvix/castore): move src/fs/test into fuse mod
These tests only interact with the FUSE layer, and
import super::fuse to do its work.

However, this only works if the `fuse` feature is enabled, which we
don't do if we enable the `virtiofs` feature only, causing the tests
to fail:

```
❯ cargo test --no-default-features --features virtiofs
   Compiling tvix-castore v0.1.0 (/home/flokli/dev/nixos/code.tvl.fyi-submit2/tvix/castore)
error[E0432]: unresolved import `super::fuse`
  --> castore/src/fs/tests.rs:14:13
   |
14 | use super::{fuse::FuseDaemon, TvixStoreFs};
   |             ^^^^ could not find `fuse` in `super`
```

We move src/fs/tests.rs to src/fs/fuse/tests.rs
(and src/fs/fuse.rs to src/fs/fuse/mod.rs) to better structure this,
which will automatically cause both tests and code to only be built if
we have the `fuse` feature enabled.

Change-Id: I8fbbad3e4457e326bdfd171aa5c43d25d3187b5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-26 19:46:47 +00:00
Ilan Joselevich
3ebcf30b64 feat(tvix): filter src via lib.fileset
Previously changing any file (including default.nix or README.md) would
cause Nix to compute a different derivation path, resulting in needing
to (very often redundantly) rebuild the crate/package. With this change
you can expect less rebuilds.

Crate2nix currently does its own insufficient src filtering and it does
not expose an API to override the filtering, so instead I created a
function that we can use to override the src to have stricter filtering.

I implemented the filtering for all of the workspace members, if you
want to modify any of them, please read the lib.fileset docs
https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset

Change-Id: I7ab5a0064a76938d14f7f65801be9f3a5c3bad04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11714
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-26 14:27:50 +00:00
edef
edd93b1962 fix(tvix/eval): drop superfluous string context check
cl/11712 simultaneously introduced this check and made it unnecessary,
since NixString::context should never return `Some` for empty contexts
now.

Change-Id: I41a655ff33910e8326cbb7d7526eb91bd19e9585
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11713
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-25 11:33:22 +00:00
edef
49750fa1e7 fix(tvix/eval): disallow empty but allocated string contexts
Both `Some(NixContext::new())` and `None` represent empty contexts,
but the former trips up `NixString::has_context`, and seems likely
to trip up other things.

We could hide the difference in the accessors, but we don't really
*want* the distinction to exist, since heap-allocating a null value
is pretty much always a mistake.

Change-Id: Ie84d26fb0d4b59e68354891ba13bde3bae40ab6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11712
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-25 08:42:32 +00:00
Florian Klink
c4c42c8b6c docs(tvix/docs/TODO): add item for string context rework
Change-Id: I4592490a75fb05093b3fd8649db3b57bff748c0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11707
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-05-23 15:58:22 +00:00
Florian Klink
6ad3780734 docs(tvix/TODO): toXML string context and self-closing tags is done
Change-Id: I83f6b1863a023ea0bf8518c67eb51f740c3ea89a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11706
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2024-05-23 14:58:41 +00:00
Florian Klink
649a862ae1 feat(tvix/eval): rm NixContext::join, add take_context & IntoIterator
In places where we want to extend context with that from another
NixString, use take_context() to split it off, then call .extend(),
making use of IntoIterator to avoid a bunch of clones.

Change-Id: I2460141a3ed776c64c36132b2203b6a1d710b922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11705
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-05-23 14:50:38 +00:00
Florian Klink
ec8d79f3db feat(tvix/eval): teach builtins.toXML context
XmlEmitter gains a NixContext field, and `write_typed_value` extends it
with all context elements present in the passed value.

Once all serialization is done, a into_context() function returns the
collected context, so we can construct a NixString with context.

Tests for this live in tvix-glue, as we use builtins.derivation, which
is not present in the tvix-eval crate.

Fixes b/398.

Change-Id: I85feaaa17b753885f8a017a54e419ec4e602af21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11704
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2024-05-23 14:49:07 +00:00
Florian Klink
a4a313cdd2 feat(tvix/eval): add NixContext::extend
This is a slightly less annoying version of `join`, which does not
consume self. It's more consistent with HashSet::extend().

Change-Id: Ifd0872da36fe8e7b2aa6948674cb8e4023abe9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11703
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-05-23 14:49:07 +00:00
Florian Klink
48e045299d fix(tvix/eval): NixString are bytes
This is not necessarily valid UTF-8.

Change-Id: I72f3157240772eb9c558e5699b4785e44d256fd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11702
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-05-23 09:54:26 +00:00
Profpatsch
5b2ba0efa1 refactor(tvix/eval): rewrite xml emitter to be simple-stupid
In order to be compatible with the nix XML generator, it’s easier to
generate the XML directly, instead of going through a library which we
have to bend to do what we need.

Removes dependency on `xml-rs`, which came with a full XML parser that
we didn’t use. Only takes a tiny bit of code for the XML escaping,
somewhat simplified.

I add a little escaping value, to make sure we have the same behaviour
as nix proper.

Interestingly enough, we never need to escape XML attribute names,
because the `builtins.toXML` format encodes user-defined values as
attribute keys only. So we only escape attribute values.

Fixes: https://b.tvl.fyi/issues/399
Change-Id: If4d407d324864b3bb9aa3160e2ec6889f7727127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11697
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
2024-05-22 10:32:55 +00:00
Connor Brewster
e7be342256 feat(tvix/store): concurrently upload small blobs during nar ingestion
Currently all blobs are uploaded serially when ingesting NARs. If a NAR
contains many, small blobs, ingestion may become slow if there is a lot
of round-trip latency to the blob service.

This makes the NAR ingester use the ConcurrentBlobUploader which allows
for buffering small blobs in memory so they can be uploaded concurrently
to the blob service without blocking further deserialization.

Change-Id: I093a73770232df12d9a11e5d901b99c08505c3cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11694
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-20 15:21:51 +00:00
Connor Brewster
b0aaff25fa refactor(tvix/castore): extract concurrent blob uploader
The archive ingester has a mechanism for concurrently uploading small
blobs to the blob service in order to hide round trip latency with the
blob service when ingesting many small blobs.

Other ingestion sources like NARs also need a similar mechanism, this
extracts the concurrent blob uploading mechanism into its own struct to
make it more reusable.

Change-Id: I05020419ff4b9ad5829fbfb5cd08d36db983b8c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11693
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-20 15:21:46 +00:00
Florian Klink
bc42c355cf refactor(tvix/store/pathinfo): test with PathInfoService directly
Since cl/…, a PathInfoService doesn't need to implement `calculate_nar`
anymore, so most of them don't actually have a handle to a
{Blob,Directory}Service anymore.

This means, we can simplify the construction of them for test cases
a lot.

Change-Id: I100e9e1c9b00a049b4d6136c57aad4cdb04461c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11691
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-20 15:03:10 +00:00
Florian Klink
d4978521b0 fix(tvix/eval): use fake values for __curPos, rather than an error
Have this return the same values as builtins.unsafeGetAttrsPos, rather
than returning a CatchableErrorKind, which crashes the CLI if it bubbles
up.

The environment we're in doesn't allow emitting a warning, as we don't
have `co` in scope, but that's probably OK as a stopgap solution.

Alternative to cl/11665.

Change-Id: I5b2c2530842547c93b6533ed9601ee9b2923b1bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11685
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-20 06:52:35 +00:00
Brian Olsen
7d7d0d150e docs(tvix): Begin documenting the Nix daemon protocol
This adds rough notes documenting the history of the Nix
daemon protocol, how logging works as well as begins
documenting inputs and outputs for all operations.

Change-Id: Id24a9a658c3e4e7c350ca1e4622f63ed96ccef5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11594
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-16 22:44:19 +00:00
Yureka
4c062d5c92 fix(castore/directory/objectstore): fix responses for deduplicated dirs
Using remove_node messed up the extraction of nodes from the graph. Use
into_nodes_edges() instead, to remove the nodes without cloning.

Change-Id: Id76c7935d082d6f26192cc3cd490483594f1d1e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11684
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-05-16 22:06:01 +00:00
Florian Klink
9a704acda5 chore(tvix): drop json feature for tracing-subscriber
Drop the (unused) feature flags here, and get rid of some crate
dependencies.

Change-Id: Id64852b498725467d56abb12b548301cfba6a760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11678
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-16 21:05:51 +00:00
Florian Klink
d65e6df3ab feat(tvix/build): drop json logging
This has already been dropped for tvix-store, drop it here as well.

Change-Id: Ib3aa37bbe9fd4c498b5ba1213f7d922d0c64ffc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11677
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-16 21:05:51 +00:00
Yureka
6aa1d2c4a8 feat(tvix/store): add ObjectStoreDirectoryService
Change-Id: I1636012be2e8ee3ae64f7bc62fd28bfe0cb2bca5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11668
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-16 16:33:06 +00:00
Florian Klink
b080870fd9 chore(tvix): bump to tokio-listener 0.4.1
This gives multi-listener support, as well as listening on named socket
FDs.

Change-Id: I92b441f854e2faf80074463d0ca6bdc23cbd890a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11663
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
2024-05-15 21:31:36 +00:00
Florian Klink
8afef1a3cc chore(tvix/store): bump otlp stack
This aligns the tonic version we use directly in Tvix to the one pulled
in by the OTLP stack.

Change-Id: I658528c8dabb7cd6948f1207ddcdef1984e82037
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11666
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-05-15 21:31:36 +00:00
Florian Klink
1392913e98 chore(tvix/nar-bridge): move to nar-bridge-go
Make some space for the rust implementation.

Change-Id: I924dc1657be10abe5a11951c3b9de50bae06db19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11662
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
2024-05-15 21:31:36 +00:00
Florian Klink
adb42959a3 feat(tvix/store/pathinfo): add a Cache combinator
This allows querying two PathInfoService implementations sequentially,
and inserts into the "near" one if it's not there yet.

There is no negative cache, and put / listing is not implemented (for
now).

Change-Id: I24c3d0e0c3c2f0524a6cc7b2f3cbc33eb20cf92b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11636
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 22:05:59 +00:00
Florian Klink
30995a0990 fix(tvix/castore/grpc): don't use explicit channel
We can just use the `BoxStream` directly, or a `once` with the single
`Directory`.

In the recursive case, we also did not properly close the channel after
the first error.

Change-Id: Ifad56d307fc7861107b6d3cffd28d35631d526e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11635
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-13 22:05:59 +00:00
Florian Klink
b26569028c feat(tvix/castore/directory/get_recursive): add 'static to BoxStream
Necessary to directly use this in the GRPC DirectoryService wrapper
directly.

Change-Id: Ic6a0038a40dc30071d145af5035345fcd93288ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11634
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 22:05:59 +00:00
Florian Klink
ed584b9296 feat(tvix/castore/directory/traverse_directory): simplify
Use try_stream! rather than stream!, and a bit more map_err and ok_err
to make things a bit more concise. Once we have proper error types here,
and impl Froms, a lot of the error mapping would disappear entirely.

Change-Id: I5240a6b0ff7818b94c151322774242b2c142e33b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11633
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-13 22:05:59 +00:00
Florian Klink
7fd4adc129 feat(tvix/castore/directory/traverse): simplify code
Replace the loop manually driving the iterator with a for … in, and some
of the match with ok_or_else.

Change-Id: I6d7b3ef1bf1c7aa128bd6adef09390b54f79479e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 22:05:59 +00:00
Florian Klink
afcbc1d86d chore(tvix/clippy): configure clippy allow block_in_conditions
… in Cargo.toml.

This gets an imperative `cargo clippy` run to pick up that config,
so `-A clippy::blocks_in_conditions` doesn't need to be explicitly
specified anymore.

Change-Id: I32b6cc50c77c22cba0d816d0db508c2f94b2c383
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11659
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-05-13 16:57:03 +00:00
Florian Klink
ed2c0be920 feat(tvix/store/pathinfo/*): add more instrumentation
Add instrumentation to the get() and put() implementations of all
PathInfoService.

Use the nixbase32 representation of the digest, not the base64 one.

Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-12 22:03:15 +00:00
Florian Klink
b22fc8cd9a refactor(tvix/castore/error): drop From<PoisonError<T>> for Error
We don't produce these erorrs anymore, no need to provide a conversion
to it.

Change-Id: I37933e436ad15c5d90b3ac270c4ef5742980513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11614
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 15:09:33 +00:00
Florian Klink
25d97c5b55 refactor(tvix/castore/directoryservice/memory): use tokio RwLock
We don't want to block here, and this also means there's no poisoning to
deal with.

Change-Id: Ic375571970c48beace0005ae2c012135086a4d67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 15:09:33 +00:00
Florian Klink
84114cf02c refactor(tvix/castore/blobservice/memory): use parking_lot RwLock
This one doesn't require us to deal with poisoning, is upgradeable and
the right thing to use when locking access to data, not IO resources.

Change-Id: I78634953a73404500d28f51f1d93a87e215c8149
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 15:09:33 +00:00
Florian Klink
96b8b1a205 refactor(tvix/store/pathinfo/memory): tokio RwLock, improve list()
We don't want to use the std::sync::RwLock here, as it blocks.

This also means we don't need to deal with the error cases anymore.

The list() implementation is updated to use try_stream, which means we
can now avoid collecting everything into a Vec before returning from it.

Change-Id: I9057dcc410dc553e6b1be3f20d5ee830569e8218
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11611
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 15:07:31 +00:00
Florian Klink
11850249c8 docs(tvix/docs): async nar reader is done
Change-Id: I4e8fd03d521b7d30904f84a6afcf842f1505ec7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11628
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 15:07:30 +00:00
Florian Klink
728ceda8a6 docs(tvix/todo): add section on [ca]store errors
Change-Id: I832eed8abb62b7e13972ed58649a96e31b17f033
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11610
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 14:59:57 +00:00
Florian Klink
fd49dfa6ca docs(tvix/todo): update test suite section
nix_oracle is quite misleading, and fooled me into thinking this
validates all .exp files in the repo to match Nix output (which it
doesn't, that's done by verify-lang-tests).

Also, the whole test suite structure thing can use a bit more
explanation.

Change-Id: I2fadcc871843143270ad2ed9ac98de8287280e6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11609
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 14:59:57 +00:00
Florian Klink
bdf4df1ac7 chore(tvix/narinfo2parquet): bump dependencies
Get rid of the RUSTSEC warnings.

Also upgrade tokio to drop the Sized requirement, similar to cl/11608.

Change-Id: Idc6fece23c79eb30fd1dfc5fe64fa2e4c08cd412
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11624
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 14:18:30 +00:00
Florian Klink
d9c2a1994c chore(tvix): update dependencies
Get rid of the RUSTSEC warnings.

Change-Id: I949f5173eb3e807f2de497f2eb94493f7a331e18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11623
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:55:21 +00:00
Florian Klink
fe20ba5ffc feat(tvix/store): add LruPathInfoService
This provides an implementation of PathInfoService storing PathInfo in
memory up to a certain capacity, then evicting these that have been used
the least recently.

Change-Id: I9d738687caf4f181a957f72245f26b92832313cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11622
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:55:21 +00:00
Florian Klink
a49c32ef42 refactor(tvix/store/pathinfo/sled): drop {blob,directory}_service
These are not used anymore.

Change-Id: I9c348391c9600e9319f171faf3eda7175ebf7076
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11621
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 13:55:21 +00:00
Florian Klink
03af6ab725 refactor(tvix/store/pathinfo/memory): drop {blob,directory}_service
These are not used anymore.

Change-Id: I6c16b4d80ddaabcb75fec3ea3e32b923b7719485
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11620
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:55:21 +00:00
Florian Klink
14766cfe1d refactor(tvix/store): drop calculate_nar from PathInfoService
This shouldn't be part of the PathInfoService trait.

Pretty much none of the PathInfoServices do implement it, and requiring
them to implement it means they also cannot make use of this calculation
already being done by other PathInfoServices.

Move it out into its own NarCalculationService trait, defined somewhere
at tvix_store::nar, and have everyone who wants to trigger nar
calculation use nar_calculation_service directly, which now is an
additional field in TvixStoreIO for example.

It being moved outside the PathInfoService trait doesn't prohibit
specific implementations to implement it (like the GRPC client for the
`PathInfoService` does.

This is currently wired together in a bit of a hacky fashion - as of
now, everything uses the naive implementation that traverses blob and
directoryservice, rather than composing it properly. I want to leave
that up to a later CL, dealing with other parts of store composition
too.

Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 13:33:59 +00:00
Florian Klink
944a781354 fix(tvix/store/pathinfo/sled): use spawn_blocking
This does IO, which might take a longer amount of time than what we want
to be blocking the normal executor.

Use spawn_blocking instead. I didn't add it for the constructors, as we
only call these once.

Change-Id: I9a1063099bac9582ca9681043c58c1edc780c5ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11618
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-11 13:33:59 +00:00
Florian Klink
717081ae37 fix(tvix/castore/directory/sled): use spawn_blocking
This does IO, which might take a longer amount of time than what we want
to be blocking the normal executor.

Use spawn_blocking instead. I didn't add it for the constructors, as we
only call these once.

Change-Id: I96231fcff8d10abe90cafde25a099a2db6ea9414
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11617
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:33:59 +00:00
Florian Klink
ebb7b32a2c chore(tvix/castore/blob): remove sled implementation
This never did any chunking, and sled (rightfully) performs really bad
if values get too large.

We switched the default to using the objectstore backend with the local
filesystem a while ago, no need to keep this footgun around anymore.

Change-Id: I2c12672f2ea6a22e40d0cbf9161560baddd73d4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11616
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 13:28:51 +00:00
Florian Klink
73c848a18f refactor(tvix/store/pathinfo/grpc): simplify pingpong
We already have the same code in make_grpc_path_info_service_client.

Change-Id: Ibcd60831af8a061a2c3bb2f960f52a43d06cf6fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11615
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 13:28:51 +00:00
Aspen Smith
cff903caff feat(tvix/cli): Use the same TvixStoreIO for the full runtime
This specifically allows subsequent fetch calls in the REPL to avoid
fetching the same path twice (because they share the same
PathInfoService).

Change-Id: Ieda089080174913a8c014f13d6852cac599a7e17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11484
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-05-09 13:57:02 +00:00
edef
31d73cd443 refactor(nix-compat/nar/reader): reuse prev_name allocation
We reuse the prev_name allocation for Entry, instead of allocating and
returning a separate Vec.

We encode the `prev_name: None` case as an empty vector, since we don't
allow empty names anyway, and the sorting is equivalent.

Change-Id: I975b37ff873805f5ff099bc82128706891052247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 15:30:46 +00:00
edef
17a7dac94f feat(nix-compat/wire/bytes): read bytes into an existing buffer
For small bytestrings (like NAR names), we can read into a preallocated
fixed-size buffer, instead of allocating a Vec every time.

Change-Id: Id8da9e9cea99c814361230c0ec02606b731c79a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11606
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-08 15:30:46 +00:00
edef
368a11ee0a chore(tvix/tools/crunch-v2): upgrade tokio
This drops the Sized requirement on AsyncReadExt.

Change-Id: I2a89c708922ca528cb493aefd0613f477f648e83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11608
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-05-08 10:19:17 +00:00
edef
1eedb88939 refactor(nix-compat/wire/bytes): style fixes
Change-Id: I65c3c43df83e0c364a4b7f1f3054c5b676bd07d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11605
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 08:48:59 +00:00
edef
ca10a8726f fix(nix-compat/store_path): use Box<str>
We don't actually build up names in place here, so we don't need a
capacity field. Saves 8 bytes.

Change-Id: Icb01b45561e28fd525b726612f56d4640bc834c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11604
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 08:48:59 +00:00
edef
51e0f78e93 feat(nix-compat/wire/bytes/reader): support buffered reading
If our underlying reader supports AsyncBufRead, then we can too.

Change-Id: If4b948c983400ca591c1c475bbcf7dc00d562040
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11545
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 06:03:37 +00:00
edef
ebad318ab3 fix(nix-compat/wire/bytes/reader): always read trailer tag data
Even if we have an aligned (or zero) size (and thus have no padding),
we may still have a non-padding trailer.

Since an exact read of only the user data bytes must always read the
trailer bytes as well, this has to happen for aligned sizes too.

For zero-sized trailers, we solve this by reading immediately, since no
read is required of the user at all.

user_len thus becomes a NonZeroU64, and the definition of "body data"
is adjusted to always exclude the final block.

Change-Id: I2c6bb51270fb22ad72a65076a3468673ab56ef68
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11597
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 04:44:01 +00:00
edef
0472b55b20 fix(nix-compat/wire/bytes/reader): drop allow(dead_code)
We're using this in the NAR reader now.

Change-Id: I28f17b1ccedd62ffcaf2fa32b517f16bcd036d94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11603
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-08 04:07:46 +00:00
edef
d8a6cc862d refactor(nix-compat/wire/bytes/reader): drop prev_filled
We specifically structured the code this way to allow using
`this.filled` as-is, so we should use it.

Change-Id: I7e11bddceb4d5f37b1dd4c453a9d53b85fc1f6c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11602
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-08 04:07:46 +00:00
Connor Brewster
da9bc274f3 refactor(tvix): remove usage of async-recursion
Rust 1.77 supports async recursion as long as there is some form of
indirection (ie. `Box::pin`). This removes the need to use the
async-recursion crate.

Change-Id: Ic9613ab7f32016f0103032a861edff92e2fb8b41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11596
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-06 16:05:09 +00:00
Florian Klink
01a4a2399c fix(tvix/store/pathinfo/nix_http): update error message
Autosubmit was too fast and submitted this before my push went through.

Flagged in https://cl.tvl.fyi/c/depot/+/11580/comment/40a56824_7be73334/

Change-Id: I1f835ae60c2c6fd2db6654c1b1c71d90bee141af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11595
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-06 15:35:24 +00:00
Florian Klink
aaf258f61e feat(tvix/store): use async nar reader for ingest_nar
Rename read_nar to ingest_nar, and have it use the async nar reader
version, and the ingest_entries machinery.

This means we can now drop all code dealing with manually assembling
castore nodes.

Update our consumer, NixHTTPPathInfoService to use the new API.
As we now accept an AsyncRead, we don't need to do any blocking here
anymore, and can use the same async-compression crate as in the fetching
logic (and support some more compression formats out of the box).

Change-Id: I8646d20bd8603f8da47b5c84bc9e4ac236eb7f1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11580
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:24:17 +00:00
Florian Klink
bc92f4188e test(tvix-castore/import): add tests for ingest_entries
Change-Id: Ia7906533868fd948509419e0d64b64582575a7fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11591
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:11:39 +00:00
Florian Klink
4aff40fff8 fix(tvix/castore/import): assert end of stream
Once we break out with the root node, there may be no more elements in
the stream.

Change-Id: I6f5fc5662095aa2b2a56bcad506d25520d9ad00c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11592
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-06 15:11:39 +00:00
Florian Klink
75f2a1f97d fix(tvix/castore/import): deal with entry.path() not having a parent
We got away with not properly dealing with this for the archive case,
where everything is contained inside a toplevel dir, but NARs can encode
a single file/symlink.

Properly break if the IngestionEntry path has the ROOT as parent, and
only create filling directories in the other case.

Change-Id: Ib378d0d1040de7c3fe310912a0b0488c55afee83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11590
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:11:39 +00:00
Florian Klink
281bd46a43 feat(tvix-castore/import) have IngestionEntry.path() return &Path
There's no need for this to be a &PathBuf.

Change-Id: I2d4126d57cfd8ddaad5dd327943b70b83d45c749
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11589
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-05 14:54:19 +00:00
edef
08feea4817 feat(nix-compat/nar/reader): async support
This is a first cut at the async NAR reader, with some rough edges.
Poisoning is left unimplemented for now, pending future work.

Change-Id: Ifaafe0581a5e0e165a13357b909fb441f7bd8bab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11524
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-04 21:45:39 +00:00
edef
343e176bec feat(nix-compat/wire/bytes/reader): parametrise on trailer tag
This allows using BytesReader with a custom tag, eg the closing parens
for the NAR reader.

No public constructor is provided for custom-tagged readers, since this
feature isn't public API.

Change-Id: I82e73d064edc4b6783ead1d6fe46a5b35f45c844
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11543
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-04 21:45:39 +00:00
Florian Klink
ba00f0c695 refactor(tvix/*store): use DS: DirectoryService
We implement DirectoryService for Arc<DirectoryService> and
Box<DirectoryService>, this is sufficient.

Change-Id: I0a5a81cbc4782764406b5bca57f908ace6090737
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11586
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-04 21:27:26 +00:00
Vincent Ambo
f2f12d1556 docs(tvix/eval): add document about how binding construction works
Change-Id: I216be1b75eb9f18a58ab2164a77b3c51de8bf784
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11583
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2024-05-03 22:25:40 +00:00
Florian Klink
1c7d319164 refactor(tvix/store/pathinfo/sled): cleanup, add instrumentation
Write this a bit more compact, by using map_err(|e| …) and ?.

Ideally we'd get rid of the error mapping entirely, by using proper
error types, but that's left for a followup.

Change-Id: I68dc72b162ac89c5ff82d8c2bc26e1c808a0affd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11584
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-03 19:49:51 +00:00
Florian Klink
37671d3913 test(tvix/[ca]store): enable bigtable only with "integration" feature
The emulator and bigtable client are quite big. Remove them from the
default //tvix:shell.

Put the tests behind a `integration` feature flag, and add a variant
with that enabled to CI, and drop the bigtable tools from //tvix:shell.

Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-03 08:53:09 +00:00
Florian Klink
b8e82a42ed chore(3p/overlays/tvl): switch to nixpkgs crate2nix
Since a recent nixpkgs bump bringing a version of cargo with
https://github.com/rust-lang/cargo/pull/12914,
crate2nix creates a crate-hashes.json with all crate hashes from
Cargo.lock (and downloads a lot of stuff while producing it).

https://github.com/nix-community/crate2nix/pull/341 prevents this from
happening, but our hardcoded crate2nix pin prevented us from getting the
fix included in 0.14.0, which did land in nixpkgs.

Replace the pin with a simply override, carrying our only leftover patch
on top of it, and link to that PR.

Change-Id: I9503898e15d61fa6a2b1589d141bec1b4ed3d616
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11581
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-05-03 07:57:49 +00:00
Brian Olsen
7a7532d053 chore(nix-compat): Document daemon protocol version
This documents which Nix version uses which daemon protocol version so that
any implementor of the protocol can make informed choices about how far back to
support legacy.

Change-Id: If2fc11702f7f731816d51928efc9ea9d939e1c00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11579
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-02 16:04:13 +00:00
Florian Klink
516c6dc572 refactor(tvix/castore/import): use crate Path[Buf] in IngestionEntry
This explicitly splits ingestion-method-specific path types from the
castore types.

Change-Id: Ia3b16105fadb8d52927a4ed79dc4b34efdf4311b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11563
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-02 15:26:29 +00:00
Florian Klink
abc0553eb8 feat(tvix/castore/directory/traverse): use castore Paths
This switches from using std::path::Path to using castore paths.

We can drop some error handling in descend_to, as absolute (or redundant)
paths are not representable.

We however now need to convert from a std::path::Path to our
representation, and decide to accept .. canonicalization, as paths in
EvalIO might contain this. Dealing .. to hop into another store path, if
we encounter this, should be dealt with in a previous step.

Change-Id: I5e94693808420c5d56587c68731252b54755bf93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11575
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-02 15:26:29 +00:00
Florian Klink
4033d4c50f feat(tvix/castore/path): implement PathBuf::from_host_path
This allows converting from std::path::Path to castore PathBufs.

A flag is present to control .. canonicalization, and the usual caveats
about platform-specific differences apply.

Currently only added for unix, we'll carefully consider other platforms
on a case-by-case basis.

Change-Id: If289a92f75a2e5c3eec132b6a91a28d225fc1989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11577
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-02 13:28:29 +00:00
Florian Klink
c0d5439362 refactor(nix-compat): derivation_or_fod_hash -> hash_derivation_modulo
There's no need for us to come up with our own names for this.
Also update the comments/docstrings a bit, and inline the intermediate
hash_derivation_modulo calculation.

Change-Id: I09dab8ffe1ebfb6601841e98119eee4ff25d8f39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11578
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-02 10:32:00 +00:00
Florian Klink
3a9432f4d8 feat(tvix/castore/path): impl AsRef<Path> for Path
This allows using both Path and PathBuf in a function argument taking
`impl AsRef<Path>`.

Change-Id: Ibd3ba6fac538069d2fe729d1ef399fdef301668f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11574
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-01 19:10:49 +00:00
Florian Klink
ea7b334a62 feat(tvix/castore): expose PathBuf too
Change-Id: I1538725766a73436eba9a92dcb4f3275ace2fa4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11573
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-01 19:10:49 +00:00
edef
f222325339 fix(tvix/castore/path): join, push -> try_{join,push}
These are fallible methods, so they should be named accordingly.

Change-Id: I6dc271c42989dd6500173488190f65381835d6fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11572
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
2513120ff5 feat(tvix/castore/path): add PathBuf::{new, with_capacity, push}
Change-Id: Ia64f4bda80e91adbdb804f4f26cef5ace8f5406a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11571
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
687291cebc feat(tvix/castore/path): more conversions
Change-Id: I3ee510b444848316df520dc8ca445d0f3c7d607f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11567
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
1bb023df91 feat(tvix/castore/path): single-component paths are children of ROOT
The empty path (Path::ROOT) is explicitly a valid path, and "foo" is
simply a child of "". The root itself is the only path without a parent.

Change-Id: Iff00dc8aed89eaf98702b664c0df658bd5a1d88a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11569
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-01 13:40:24 +00:00
edef
aa53338ddb test(tvix/castore/path): test the null byte case
Change-Id: Ia4ceaf56f6cb4d8ed1ad935c77b3898d711da73a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11568
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
2d7f4135ec feat(tvix/castore/path): use proto::validate_node_name
Use the shared code for validating node names, since that is what path
components represent.

Change-Id: I12109c1306b224718faa66cf1f2874c78c1436a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
4b3223a621 chore(tvix/castore/path): drop now-duplicate tests
Since PathBuf doesn't have inherent methods anymore, these just forward
to Path itself.

Change-Id: I30f44adc9994337c367bad985ada0e8fcb98dd6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11570
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
Edwin Mackenzie-Owen
465370c11f feat(tvix/nar-bridge): support HTTP HEAD for narinfo
Change-Id: I4ec8e9108f52d0a84bc1af01a27bd51de4ab493f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11456
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-01 13:11:51 +00:00
edef
d84fd49169 fix(nix-compat/store_path): permit store paths with dots again
Nix 2.4 accidentally permitted this behaviour, but the revert came
too late to beat Hyrum's law. It is now considered permissible.

Link: https://github.com/NixOS/nix/pull/9867
Change-Id: Ie97777af6765fe1c12c8aa593afe1c9b69125775
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11553
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 12:32:39 +00:00
edef
c5b3187002 feat(tvix/castore/path): implement Debug + Display
We implement Debug explicitly, so that we don't just see raw integers.

Change-Id: I11213094728f3e0c674562ee71c092a950041632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11565
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-01 11:27:59 +00:00
edef
ccb93a65a8 feat(tvix/castore/path): make Path unsized, repr-equivalent to [u8]
Change-Id: Ia1e6a10dba0bcda44e06e4cab7d4e52ca88b5859
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11564
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 11:03:12 +00:00
Florian Klink
959c50c4ac feat(tvix/castore/path): add Path{Buf}
This contains Path and PathBuf, representing platform-independent paths
representable by the castore model.

These are always relative, and platform-independent, which distinguishes
them from the ones provided in the standard library.

A subsequent CL will move IngestionEntry (and more) to use them.

Change-Id: Ib85857f4159ebc2f3c00192c95d4e5b54ffd4fcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11558
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-05-01 11:01:30 +00:00
Florian Klink
360d80f62e feat(tvix/store): use local object_store for tvix-store daemon blobs
The sled backend doesn't perform very well with blobs in there,
especially as it's not doing any chunking.

Switch to the `objectstore+file://` instead, which does do CDC.

Change-Id: Ic0d8836c6fc811b80c7202e3ee7f44a05a4f8dfa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11554
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-30 18:19:43 +00:00
Florian Klink
c9d3946cb5 refactor(tvix/castore/import): restructure error types
Have ingest_entries return an Error type with only three kinds:

 - Error while uploading a specific Directory
 - Error while finalizing the directory upload
 - Error from the producer

Move all ingestion method-specific errors to the individual
implementations.

Change-Id: I2a015cb7ebc96d084cbe2b809f40d1b53a15daf3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11557
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-30 17:12:39 +00:00
Florian Klink
77546d734e refactor(tvix/castore): remove IngestionEntry::Unknown
We shouldn't try to represent non-representable things in the ingestion
entries (only to throw an error).

It's cleaner to throw the error directly in the part producing the
stream.

Change-Id: I6b6f6d8c2f677425210142a39f1829ddeefec812
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11556
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: firefly <firefly@firefly.nu>
2024-04-30 15:53:58 +00:00
Florian Klink
4c5c810c6f refactor(tvix/castore/import): move upload_blob_at_path into fs mod
This is only useful for when we have access to a filesystem, so it
shouldn't be in the root.

Change-Id: I9923aaed1aef9d3a1e8fad41f58821d51c2eb34b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11555
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: firefly <firefly@firefly.nu>
Tested-by: BuildkiteCI
2024-04-30 15:53:58 +00:00
Florian Klink
5e8cfcfcd6 fix(tvix/castore/import): symlink targets are Vec<u8>
These can be arbitrary bytes in theory. Some of our libraries might
be more strict, or inconsistent w.r.t. their representation of path
separators.

Change-Id: I7981b74fc7d3dd79f5589cf2ef52ced7b71dd003
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11551
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-04-30 13:18:03 +00:00
Florian Klink
ca64881cb3 docs(tvix/castore): fix tvix_castore::import sub-mod docstrings
The one for `fs` was wrong, and ended up being attached to ingest_path,
and the one for `archive` was missing entirely.

Change-Id: I8a4c32fb5293badb1ea0764c278a88e4ca33c018
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11552
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-04-30 10:06:17 +00:00
edef
06f94a21bd fix(tvix/nix-compat/wire): RangeInclusive<usize> for memory size bounds
u64 is an inappropriate type for something memory-sized, and most
callers end up with off-by-ones when using `..` rather than `..=`,
including the tests for the module itself.

Change-Id: If3b7bea27eb0a6c01e0a5d7e64966acbbb664268
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11550
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 09:55:51 +00:00
edef
095f715a80 refactor(nix-compat/wire): drop primitive functions
These may as well be inlined, and hardly need tests, since they just
alias AsyncReadExt::read_u64_le / AsyncWriteExt::write_u64_le.

Boolean reading is worth making explicit, since callers may differ on
how they want to handle values other than 0 and 1.

Boolean writing simplifies to `.write_u64_le(x as u64)`, which is also
fine to inline.

Change-Id: Ief9722fe886688693feb924ff0306b5bc68dd7a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11549
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-30 09:55:51 +00:00
edef
b3305ea6e2 refactor(nix-compat/wire/bytes): branchless padding computation
Change-Id: Ie07c2516a485c78afa6f9a3c8256e9708c4c42c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11548
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 09:16:40 +00:00
edef
88f49c8351 fix(tvix/nix-compat/wire/bytes): return UnexpectedEof for unexpected EOF
Change-Id: I8cb79f93742cc3a9088701bc8c2795de893eb6a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11547
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-30 09:16:40 +00:00
Alice Carroll
8d49ff3d64 test(tvix): Fix tvix tests on macOS
Prior to this, some tests would not build
or would fail in an obscure way.

Change-Id: I68587cc7592492ebfd71ca02fc7ccc9ff7c0196f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11544
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 00:55:34 +00:00
Florian Klink
c192cd04b8 feat(tvix/glue/tests): configure <nix NIX_PATH
This is also needed to make `<nix/fetchurl.nix>` available in the
testsuite.

At some point, we might want to classify different types of tests
excepting a different featureset, but for now, enabling this for all of
glue is fine.

Change-Id: Ided450bbb1f8eb7b66d454bd28bd19b17eb318e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11546
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 22:02:03 +00:00
Florian Klink
fab9a07274 feat(tvix/glue/tests): wrap TvixStoreIO with TvixIO
This makes `<nix/fetchurl.nix>` available in the testsuite.

Change-Id: I04657b227c539edec86e5b6033e1016adcfce244
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11537
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 21:13:55 +00:00
Florian Klink
499075a84f refactor(tvix/glue/fetchers): rename "nixhash" to just "hash"
Change-Id: I2e40b1bcbd9932469ec2af9bb9ce20bb267566ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11536
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-29 17:58:25 +00:00
edef
97e4600078 refactor(nix-compat/wire/bytes/reader): move size reading to async ctor
with_size only existed to allow separating the phases of reading size
and reading data, and similarly the Size state only existed to allow
folding size reading into poll_read.

Neither of these are necessary if we make the constructor async,
and handle the size reading there.

As a bonus, this makes BytesReader::len sensibly typed, and permits
implementing is_empty, as Clippy demands of us.

Change-Id: I72173ec06d60b7998d16a3ecfc5e6ac5424bbed3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11542
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 16:45:23 +00:00
Florian Klink
298c6876cc docs(tvix/glue/fetchers): update docstring on Tarball
The tarball can also not be compressed.

Change-Id: Idbf1b7168cc9a43826355e3cb8b4140f5f53d780
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11535
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 15:44:26 +00:00
edef
3548514d4a refactor(nix-compat/wire/bytes/reader): use fn for PhantomData
We don't actually hold a Tag, we just want to bind the type.

Change-Id: Ida67c026f852ed54c3f18df914cf5c31e6227fd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11541
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 15:33:37 +00:00
edef
84b27760d0 refactor(tvix/nix-compat/wire/bytes): use RangeInclusive for limits
The (min, max) pair is already a RangeInclusive in essence, so we might
as well represent it that way.

Change-Id: I2f67f3c47dc36b87e866ff5dc2e0cd28f01fbb04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11540
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 15:33:30 +00:00
edef
fdecf52a52 refactor(nix-compat/wire/bytes): fold TrailerReader into BytesReader
The TrailerReader has no purpose separate from BytesReader, and the
code gets a fair bit simpler this way.

EOF handling is simplified, since we just rely on the implicit
behaviour of the existing case.

Change-Id: Id9b9f022c7c89fbc47968a96032fc43553af8290
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11539
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 15:33:30 +00:00
edef
44bd9543a6 feat(nix-compat/wire/bytes/reader): expose the remaining data length
The API is a bit odd here, because we don't have a distinct type for a
known-length reader.

Change-Id: I4a1dd07fbed0a400004dbe4aa2095c51898ad3bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11538
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-29 15:33:30 +00:00
edef
5070f9eff7 refactor(nix-compat/nar/reader): always enable poisoning *API*
The poisoning API is now always available, whether debug_assertions is
enabled or not. When debug assertions are not enabled, it is equivalent
to a unit struct, and is always considered ready and unpoisoned.

Change-Id: I950237f4816d480330d9acab32661ed4f1663049
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11525
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 13:32:46 +00:00
Florian Klink
98b85b4580 refactor(tvix/glue): clone less
Prepare the NixAttr to return without an intermediate
Vec<(String, NixString)>, and without into_iter(), and send off the
unmoved Derivation struct to known_paths without having to clone it.

Change-Id: Icdb9f78938e998a27d0313c5d9ab15b93af5821d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11531
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-04-29 13:19:41 +00:00
Florian Klink
26b77b2cf3 refactor(tvix/glue): move decompression into fetchers/ subdir
This is specifically used for the fetcher code (only).
Moving it to there for now.

Change-Id: I1e1d0541b85340ef4ff3a4c6b3fa99b51853f539
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11532
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-29 13:11:36 +00:00
sterni
69e4a78818 chore(3p/sources): Bump channels & overlays
- //tvix: address new clippy lints

- //users/tazjin: Satisfy gonic module's new need for a playlist folder.

- //users/aspen/games: adjust for changed location of df's default
  init.txt and d_init.txt.

Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-04-28 16:39:26 +00:00
Florian Klink
9401cccf84 feat(tvix/cli): change log format to compact, add RUST_LOG support
This brings feature parity with the tvix-store CLI command, at least as
far as logging (not tracing) is concerned.

Alternative to cl/11482, RUST_LOG seems to be the more canonical way to
influence this, and is consistent with how it's done in tvix-store.

Change-Id: I923a0b0ae55dc49af7efdacdcf5b1f24e561b3c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11527
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 15:10:58 +00:00
Florian Klink
88922bb207 feat(tvix/store/bin): disable --json option, set log to compact
This wasn't really used - to ingest logs in meachine-readable form, the
OTLP infrastructure is more suitable to provide structured logs than
parsing JSON from std{err}, as it also captures span information.

Also, the non-JSON output is a bit too spammy, as remarked in cl/11483 -
change it to `compact`.

Change-Id: I48007b84ba076ab566abbb6131a02868fe0eb397
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11526
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 15:08:15 +00:00
edef
cab9c774b7 chore(nix-compat/nar/wire): space Tag::Buf docs properly
Change-Id: If15b14b5328b9e7f85ea3fa7644d327dfb50e1a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11523
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-26 08:22:58 +00:00
edef
6e41e0917e feat(nix-compat/wire/bytes/reader): split out reading the trailer block
We separate ingesting the trailer block into a Future of its own,
parametrised on the specifics of the trailer pattern.

This is intended to be used for future work on an async NAR reader,
which needs to read a terminating parenthesis as well as the regular
padding.

Thanks to @griff for suggesting separating the ingestion into its own
Future.

Change-Id: I36c2503baa67937046a63e9bf0cfc38201394025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11522
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-26 08:22:58 +00:00
edef
b2dc135d92 refactor(nix-compat/wire/bytes/reader): drop random whitespace
Change-Id: Ic683eab435576acc8f7e03f5684767ffa468851a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11521
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-26 08:22:58 +00:00
edef
a39894c158 refactor(nix-compat/wire/bytes/reader): drop Pin::get_mut
Pin<&mut T> is DerefMut when T: Unpin, so we don't actually need to
explicitly call get_mut.

Change-Id: Iaa312ec49c87100010e09c94f319e57e31da0cd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11520
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-26 06:56:18 +00:00
edef
89316b22bc refactor(nix-compat/wire): move BytesPacketPosition into writer
We don't use it in the reader anymore.

Change-Id: I98fe204a747711464e9e7ca17df06fa9854eb344
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11519
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-25 23:47:49 +00:00
edef
ba46b1818a fix(nix-compat/bytes): make BytesReader less hazardous
We now *never* return the final bytes until we've read the padding
in full, so read_exact is safe to use.

This is implemented by TrailerReader, which splits the phases of
reading (and validating) the final 8-byte block, and providing
the contained payload bytes to the caller.

Change-Id: I0d05946a8af9c260a18d71d2b763ba7a68b3c27f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11518
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 23:47:49 +00:00
edef
70c679eac4 feat(nix-compat/wire/bytes): allow specifying a pre-read size
Change-Id: I9c94239c308cfbc2e6dae871ba77fb33507433c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11517
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 23:47:49 +00:00
edef
859bfcb68b refactor(nix-compat/wire/bytes): drop pin_project, clean up
We already require R: Unpin in the constructor, so there's not much use
to pin projection.

Change-Id: Ia7bf734dc3aa86ffa6d1d5de778939baa9676bb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11516
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 23:47:49 +00:00
edef
652413c97d refactor(nix-compat/wire/bytes): don't parametrise on RangeBounds
This is semantically a RangeInclusive, since we can only have
0..=u64::MAX at most, and monomorphising on the bounds doesn't
buy us anything.

Change-Id: Ib601d7fd77d703d6c8c5ec27ac9e67bb122ce1c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11515
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-25 23:47:49 +00:00
edef
d93633937c fix(tvix): typo
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-25 23:47:49 +00:00
tcmal
671bdff5dc test(tvix/nix-compat): add debug assertions for nar reader
Adds debug assertions to ensure that the reader's variants are upheld.
If any of the following happens, then the currently in use reader must
be abandoned:
  * A directory or file reader encounters an error
  * A directory or file reader is dropped before being fully read from
Additionally, a directory reader must not be read from again after it
has returned None.
These checks are only used when debug_assertions are on, so vanish in
release mode.

Resolves two TODO items added by edef

Change-Id: I27bd9643a632798db5351957506c166b9bd5ca4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11508
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Autosubmit: Aria Shrimpton <me@aria.rip>
Tested-by: BuildkiteCI
2024-04-25 17:29:57 +00:00
Florian Klink
f5bb9b8467 docs(tvix): add a list of TODOs/ideas
This contains a rough collection of ideas on the TODO list, trying to
keep track of it somewhere.

Change-Id: Ifc5b0cf9f7ac38f7a8e56515882bdf70e349544b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11512
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-25 15:07:45 +00:00
Florian Klink
7345986ff1 docs(tvix/nix-compat/wire/bytes/reader): fix typo
This is a ReadBuf, not a BufRead.

Change-Id: Ie80e894f4b24b77cdd60409ddfaa66dae0ffeec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11511
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-04-24 20:06:17 +00:00
Florian Klink
2fd9dc11c2 docs(tvix/nix-compat): add missing reference to BytesReader
Change-Id: Ideed83d191b55e131720e598b7591e8375a26cfd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11510
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2024-04-24 20:06:17 +00:00
Connor Brewster
d2e67f021e refactor(tvix/castore): add separate Error enum for archives
The `Error` enum for the `imports` crate has both filesystem and archive
specific errors and was starting to get messy.

This adds a separate `Error` enum for archive-specific errors and then
keeps a single `Archive` variant in the top-level import `Error` for all
archive errors.

Change-Id: I4cd0746c864e5ec50b1aa68c0630ef9cd05176c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11498
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-24 15:41:38 +00:00
Connor Brewster
79698c470c feat(tvix/castore): upload blobs concurrently when ingesting archives
Ingesting tarballs with a lot of small files is very slow because of the
round trip time to the `BlobService`. To mitigate this, small blobs can
be buffered into memory and uploaded concurrently in the background.

Change-Id: I3376d11bb941ae35377a089b96849294c9c139e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11497
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-04-23 17:02:07 +00:00
Connor Brewster
fa69becf4d refactor(tvix/castore): switch to ingest_entries for tarball ingestion
With `ingest_entries` being more generalized, we can now use it for
ingesting the directory entries generated from tarballs.

Change-Id: Ie1f7a915c456045762e05fcc9af45771f121eb43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11489
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-23 15:31:22 +00:00
Florian Klink
e18bc33529 fix(tvix/glue/tvix_store_io): remove early return
Doing the fetch comes up with the root node, but we still need to
descend from there to the desired subpath.

Move things around to ensure the fetch case also only sets root_node.

This logic should probably be moved into smaller, easier to consume
functions.

Change-Id: I6ab9317df794f53d2504029bbc77859e89fef1ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11507
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-23 14:53:31 +00:00
Florian Klink
72d3f9b914 fix(tvix/glue/fetchers): rename node name for all three types
We also need to rename the node in case it's a directory or symlink at
the root.

Change-Id: I6e9957200f65991645ae3e1755b943200453dfd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11506
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-23 14:41:25 +00:00
Florian Klink
8181817e53 feat(tvix/glue/fetchers): support file:// URLs
Nix supports file:// - URLs for `fetchurl` and `fetchTarball`.

Convert the enums and function arguments to hold a URL type.
reqwest::Url is a re-export of the url crate, but they don't re-export
the parsing errors, and as we want to hold these in our Error types, add
it to Cargo.toml explicitly.

The Fetcher::download function now checks on the scheme, and either
opens the file locally, or does do a HTTP request as before.

Fetch gets its custom debug impl, removing potentially sensitive
username and password out of URLs.

Change-Id: I777db1fe487370e822cbfec4624034aca5e08045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11504
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-23 12:41:17 +00:00
Florian Klink
dfef3d18d1 test(tvix/glue): add tests for fetchurl and fetchTarball
Change-Id: I53a0590ecf4e5fcb1bfd1d127824211338e28256
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11503
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-23 12:41:05 +00:00
Florian Klink
30950833c9 feat(tvix/glue/store_io): have KnownPaths track fetches too
Have fetcher builtins call queue_fetch() whenever they don't need to
fetch something immediately, and teach TvixStoreIO::store_path_to_node
on how to look up (and call ingest_and persist on our Fetcher).

Change-Id: Id4bd9d639fac9e4bee20c0b1c584148740b15c2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11501
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-23 12:40:55 +00:00
Florian Klink
091de12a9a refactor(tvix/glue): move Fetch[er] into its own types, fetch lazily
We actually want to delay fetching until we actually need the file. A
simple evaluation asking for `.outPath` or `.drvPath` should work even
in a pure offline environment.

Before this CL, the fetching logic was quite distributed between
tvix_store_io, and builtins/fetchers.rs.

Rather than having various functions and conversions between structs,
describe a Fetch as an enum type, with the fields describing the fetch.

Define a store_path() function on top of `Fetch` which can be used to
ask for the calculated store path (if the digest has been provided
upfront).

Have a `Fetcher` struct, and give it a `fetch_and_persist` function,
taking a `Fetch` as well as a desired name, and have it deal with all
the logic of persisting the PathInfos. It also returns a StorePathRef,
similar to the `.store_path()` method on a `Fetch` struct.

In a followup CL, we can extend KnownPaths to track fetches AND
derivations, and then use `Fetcher` when we need to do IO into that
store path.

Change-Id: Ib39a96baeb661750a8706b461f8ba4abb342e777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11500
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-23 12:30:48 +00:00
Florian Klink
dc444e55dc feat(tvix/boot): default CH_NUM_CPUS=2, and set num_queues= to it
See https://patchwork.kernel.org/project/qemu-devel/patch/20200706135650.438362-5-stefanha@redhat.com/

Change-Id: I3ba9a1c1f8b0be27f215da6b71cd3a0ded513a75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11502
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-22 21:15:49 +00:00
Florian Klink
ff9e8743f6 fix(tvix/eval): don't impl From<NixString> for String
This caught me by accident in an earlier revision of cl/11500 -
I had a `NixString`, wanted to return it as a `String`, so I was naively
calling `s.into()`.

That unfortunately gave me the `Display` implementation of `NixString`,
which quotes strings, causing an annoying error further up the stack.

NixStrings are bytes, we can keep the impl From<NixString> for BString,
but having a `.into()` suddenly do quoting is more than unexpected.

Change-Id: I5434ba94bfe6c493d0a57e68225ecc22daa4b948
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11505
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-22 20:28:55 +00:00
Florian Klink
4124f0e679 refactor(tvix/cli/default.nix): make eval tests and benches derivations
This makes the eval tests and benchmarks standalone Nix derivations and
readTree targets:

```
nix-build -A tvix.cli.eval-nixpkgs-cross-hello-outpath
nix-build -A tvix.cli.benchmark-nixpkgs-attrnames
```

Even without doing any fetches, We need to set `SSL_CERT_FILE`, so
reqwest is able to load its CA roots.

Change-Id: Ib45282d01044165c7816391adbeeb26334f8e924
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11493
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-22 13:15:22 +00:00
Florian Klink
f3ec974895 feat(tvix/store/bin): upload paths concurrently in copy
We can speedup uploads further by not uploading all store paths
sequentially, but in parallel.

We still don't respect the reference graph, so nothing changed here.

Change-Id: I40edb5725fe7a15170f5b9f8f4c8bec2130ecca3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11492
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-20 18:54:28 +00:00
Florian Klink
5fc403587f refactor(tvix/castore): ingest filesystem entries in parallel
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.

Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.

(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.

We still keep the same order, to avoid shuffling things and violating
the stream order.

This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.

Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-20 18:54:28 +00:00
Connor Brewster
01239a4f6f fix(tvix): fix outdated comment and error in TvixStoreIO::open
This function was originally called `read_to_string` but was changed to
`open` to make it so that file contents aren't always held in memory.
A comment and error message were not updated to reflect the new name of
this method.

Change-Id: I3d86e2f6d7006c2e1513121fc3c62efcb7e7b9bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11495
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 18:49:16 +00:00
Connor Brewster
b0bdeb2e89 feat(tvix/castore): Fix build warnings in release mode
Fixes some build warnings that only happen when building in release mode
which disables `debug_assertions`.

Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-20 16:47:12 +00:00
Connor Brewster
18ab59ed70 fix(tvix/castore): ensure all directories are present during ingestion
`ingest_entries` requires that all directories referenced by entries in
the ingestion stream have an explicit entry in the stream.

For example, if the stream contains a file with path `foo/bar`, there
must be an entry that comes later in the stream for the directory `foo`.

Change-Id: I61b4fbbb73ea7278715e04271d8073b484e05e61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11488
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-20 16:46:41 +00:00
Aspen Smith
3107961428 feat(tvix/eval): Implement builtins.fetchTarball
Implement a first pass at the fetchTarball builtin.

This uses much of the same machinery as fetchUrl, but has the extra
complexity that tarballs have to be extracted and imported as store
paths (into the directory- and blob-services) before hashing. That's
reasonably involved due to the structure of those two services.

This is (unfortunately) not easy to test in an automated way, but I've
tested it manually for now and it seems to work:

    tvix-repl> (import ../. {}).third_party.nixpkgs.hello.outPath
    => "/nix/store/dbghhbq1x39yxgkv3vkgfwbxrmw9nfzi-hello-2.12.1" :: string

Co-authored-by: Connor Brewster <cbrewster@hey.com>
Change-Id: I57afc6b91bad617a608a35bb357861e782a864c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11020
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:58:04 +00:00
Florian Klink
f34e0fa342 feat(tvix/castore/import): only allow normal components in entry paths
Explicitly document and add a debug assertion for that.

It's up to callers to ensure this doesn't happen.

Change-Id: Ib5d154809c2ad2920258e239993d0b790d846dc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11487
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:14:19 +00:00
Florian Klink
e9db0449e7 refactor(tvix/castore/import): make module, split off fs and error
Move error types and filesystem-specific functions to a separate file,
and keep the fs:: namespace in public exports.

Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-04-20 14:14:19 +00:00
Florian Klink
c4cb099823 refactor(tvix/castore/import): rename ingest_entries arg
This is not a stream of direntries anymore, but a stream of ingestion
entries.

Change-Id: I387f4497b6567066b24c58ca0262e710348180e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11485
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:14:19 +00:00
Connor Brewster
259d7a3cfa refactor(tvix/castore): generalize store ingestion streams
Previously the store ingestion code was coupled to `walkdir::DirEntry`s
produced by the `walkdir` crate which made it impossible to reuse
ingesting from other sources like tarballs or NARs.

This introduces a `IngestionEntry` which carries enough information for
store ingestion and a future for computing the Blake3 digest of files.
This allows the producer to perform file uploads in a way that makes
sense for the source, ie. the filesystem upload could concurrently
upload multiple files at the same time, while the NAR ingestor will need
to ingest the entire blob before yielding the next blob in the stream.
In the future we can buffer small blobs and upload them concurrently,
but the full blob still needs to be read from the NAR before advancing.

Change-Id: I6d144063e2ba5b05e765bac1f27d41b3c8e7b283
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11462
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-19 20:37:05 +00:00
Connor Brewster
150106610e feat(tvix/castore): add convenience add method to Directory
This adds `Directory::add` which is a convenience helper for adding
nodes into a `Directory` while preserving sorted order.

This implements `Ord` and `PartialOrd` for `FileNode`, `SymlinkNode`,
and `DirectoryNode` so `binary_search` can be used.

Change-Id: I94b86bdef5d0da55aa352e098988b9704cafca19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11481
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-19 20:10:14 +00:00
Florian Klink
8a40c92f5d chore(tvix/glue): migrate from test_case to rstest
Change-Id: Id1bac94e7b95960dabd55f095f04bdabfeee10ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11476
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-19 19:22:10 +00:00