Commit graph

20871 commits

Author SHA1 Message Date
Florian Klink
89361b2a7f fix(tvix/tracing): make cargo check and clippy happy
In case the otlp feature is not enabled, these generate warnings during
`cargo check`.
Fix by moving some imports into their functions, or using the
fully-qualified name (and one #[allow(unused_mut)])

Change-Id: I5afd89dcd4c772b6002cebdd5d0469932eacfdac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11873
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 06:27:15 +00:00
Florian Klink
a0993e7304 feat(tvix/eval): add ErrorKind::UnexpectedArgumentBuiltin
Change-Id: Ieb091b32aad566719fbe8604c4a589f5ccaaf6b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11877
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
970e2a045b refactor(tvix/eval): rename UnexpectedArgument{,Formals}
There's other places where unexpected arguments can be provided, like
in builtins.

Make space for that error type.

Change-Id: Ic831497a3a1dd36a3a184bedadcf1374bf0ae6db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11876
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-26 04:51:31 +00:00
Florian Klink
ea6f511241 refactor(tvix/glue): return a parsed Url in NixFetchArgs
The only two consumers (fetchurl, fetchtarball) of these do try to parse
it as URL, so do it in the helper.

Update url_basename to take a &url::URL, not a &str.

Also update the test to use rstest for the fixtures to reduce some
boilerplate there.

Change-Id: I1f85fe2803060dc4423e673cb7b9f9bf799d09b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11875
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
540e566900 refactor(tvix/glue): take &CAHash, not CAHash
We use a bit less cloning that way.

Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-26 04:51:31 +00:00
Florian Klink
78eb22c54d feat(tvix/glue): handle regular file at builtins.path import
If builtins.path is passed a regular file, no filtering is applied.
We use the just-introduced file_type function in the EvalIO trait for
that.

This means, we don't need to pass through filtered_ingest, and can
assemble the FileNode directly in that specific match case.

This also means, we can explicitly calculate the sha256 flat digest,
and avoid having to pipe through the file contents again (via
blob_to_sha256_hash) to construct the sha256 digest.

Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
080654aaf9 feat(tvix/eval): add file_type method
This allows peeking of the type at a given path.

It's necessary, as an open() might not fail until you try to read()
from it, and generally, stat'ing can be faster in some cases.

Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Ilan Joselevich
04f04cfc27 refactor(tvix/eval): drop placeholder impls of builtins in glue
The builtins `placeholder` and `filterSource` are now implemented in
tvix-glue and so we no longer need their placeholder "implementations" in
tvix-eval.

Change-Id: Ibf161ccf1ad40103e9fbb688e9f6be58e7772af1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11867
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-22 21:50:30 +00:00
toastal
bf541acf29 docs: use admonish for callouts
Change-Id: Icec7321a49c40e7f7bcb1e4b307f13b8db4eac25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11862
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-21 08:08:17 +00:00
Florian Klink
8dde022d31 docs(tvix): drop done TODOs in the O11Y section
OTLP shutdown behaviour was fixed in cl/11803, and tracy landed in
cl/10952 and cl/11815.

Change-Id: I310dfcc41ba62ce5b006e75a35feb839327b0e7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11866
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Florian Klink
7f625e2fc1 refactor(tvix/*store): remove some trait bounds
We don't need to require these things for these impl blocks yet.

Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Simon Hauser
639a00e2ab feat(tvix/tracing): gRPC trace context propagation
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.

The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.

This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.

Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-20 19:21:01 +00:00
Simon Hauser
2b20d8d82d feat(tvix/castore): instrument tokio task with current span
By default tokio::spawn does not instrument the spawned task with the
current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do
this manually for all tokio::spawn functions in functions that are
instrumented.

Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 15:48:14 +00:00
Simon Hauser
bd8d74a3ee feat(tvix/tracing): optional progressbar
Disable the progressbar on default and provide a interface for
optionally enabling the progressbar.

Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 10:28:54 +00:00
Florian Klink
1446e3be99 docs(tvix/docs): document nar renderer optimization potential
Change-Id: I747ab5c8ba3424828b1c88e0353170ad0efab3a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11849
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 10:28:33 +00:00
Florian Klink
8137077a74 feat(tvix/store/utils): detect gRPC NAR calculation service
We were currently always using SimpleRenderer, which would mean the
client would download every blob locally to calculate the checksum,
which of course is very slow.

Detect this special case and create a second instance (and client) for
now.

Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 10:28:26 +00:00
Ilan Joselevich
9fd601e53d chore(tvix): patch crate2nix to drop darwin dontStrip
This patch has already been applied on upstream but there hasn't been an
official release yet so we patch it ourselves.

Upstreamed patch: 0209f258cd

Follow-up of: https://cl.tvl.fyi/c/depot/+/11856

Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-18 14:13:11 +00:00
Ilan Joselevich
92c5433304 chore(tvix): move filterRustCrateSrc to utils.nix
Change-Id: Ib6bae1ea0457d3309a6ec1e08b9e4b320523c161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11857
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 15:26:19 +00:00
Ilan Joselevich
ac51803ad8 chore(tvix): Remove no longer needed libiconv addition for darwin
Since https://github.com/NixOS/nixpkgs/pull/220212 libiconv is now
automatically added for Darwin.

Change-Id: I2aaa266bcbc4d5a8256131aed36b4c2d040a8758
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11856
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-17 15:08:10 +00:00
Ilan Joselevich
210b4277b6 chore(tvix): Use callPackage instead of import on Cargo.nix
Since crate2nix v0.11 the new preferred way of importing Cargo.nix is
with callPackage instead of import.

Change-Id: Ifcc5c21e6c101cb9d0b3f1e8025b5ddb8cf99f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11855
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:46 +00:00
Ilan Joselevich
fbe29615df chore(tvix): Remove no longer needed rlib stripping fix for darwin
This issue was fixed in https://github.com/NixOS/nixpkgs/pull/255900

Change-Id: I4f569099e62c4284bdea18a512387735f139e702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:04 +00:00
Florian Klink
28b692fd50 feat(tvix/tvix-store): improve progress bars
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.

Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.

Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.

Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 12:57:34 +00:00
Florian Klink
cfab953094 feat(tvix/store): display progress for NAR calculation
This is currently still taking a noticeable amount of time, so make sure
we show it is happening.

Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 09:47:54 +00:00
Florian Klink
71a29ceff4 docs(tvix/store/import): update import_path_as_nar_ca docstr
Change-Id: I697172220efe73b2a215185027df09f815d32627
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11846
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 08:49:52 +00:00
Florian Klink
4be250109c refactor(tvix/store): drop FUSE multithread workaround for Darwin
The underlying issue in macFUSE has been fixed in
https://github.com/osxfuse/osxfuse/issues/974.

Bump our `macfuse` in nixpkgs to a version containing the fix.

This can be removed while our nixpkgs pin is bumped past
https://github.com/NixOS/nixpkgs/pull/320197.

Change-Id: Ia0e644fb13198e45018b0a218647ef211acf4df1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11824
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-16 12:02:11 +00:00
Florian Klink
7e42b4f314 fix(tvix/store/bin): fix shutdown behaviour for FUSE
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.

So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.

Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.

Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.

Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.

Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-06-16 12:02:11 +00:00
Yureka
452299dcd2 feat(castore/directory): add cache combinator
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 08:42:53 +00:00
Yureka
30e72d2d52 refactor(castore/directory/objectstore): use new order validation logic
Use the new helper introduced in CL 11708 instead of rolling our own.

Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 08:40:20 +00:00
Yureka
daada1b2fa refactor(castore/directory): separate order logic from ClosureValidator
ClosureValidator was previously only suitable for a very narrow use case:
Validating incoming uploads, which are in leaves-to-root order.
This is because the ordering validation was hard-wired into the add()
function.
This
- Re-name ClosureValidator to DirectoryGraph, which is more suitable
  since it actually stores the Directory structs and is drained in the end.
- Move the ordering-related logic to a separate OrderValidator, which
  can be used independently.
- re-write DirectoryGraph to be a general purpose
  validator which can accept the input in both orders
  and can be drained in both orders as well.

This means the DirectoryGraph and OrderValidator can now serve
multiple new purposes:
- Validating the incoming closure on the client while downloading.
- Validating the incoming closure downloaded in a caching layer from the
  `far` cache, and re-order it for insertion into the `near` cache.

Change-Id: I2b4b226348416912d7a31935bec050e53d911b70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 05:01:07 +00:00
toastal
c2a9ad3583 docs: move Registration to the top; minor reword
When trying to follow Review documentation, I got tripped up on the
“Gerrit setup” section that was asking about SSH keys since I hadn’t yet
gotten an account setup but the registration setion was near the bottom
of the document despite being a requirement for setting up Gerrit.

Included words specifying SSO, specifies that the TVL account is an
alternative method, & commit→patch since you send a patch with the
commit.

Change-Id: I465c8a79979b368ab01bcebdc7e5ef16e65dd341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11823
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-15 17:42:16 +00:00
Florian Klink
01aadace80 feat(tvix/cli,store): add tracy feature
If compiled with this features, this emits packets compatible with the
[Tracy](https://github.com/wolfpld/tracy) format.

Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 05:24:37 +00:00
Ilan Joselevich
cd0b92419a docs(tvix/TODO): dibs on crate2nix WASM
Change-Id: Idde9aaa31ecaa9570559ffe3271bc23b7a71133c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11820
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 04:02:33 +00:00
Florian Klink
86928b5e7f docs(tvix/TODO): add builtins.path roundtrip for flat
Change-Id: I8291d2fdfa2ea31ec24c82dcfb447fa9eaa3a961
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11819
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-15 04:02:33 +00:00
Florian Klink
6e321c615c docs(tvix/*store): add some internal links
We can now cross-link between pages, and it actually works.

Change-Id: Id8bd85111672c687118db9adae7f5066f22441f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11817
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 03:56:11 +00:00
Florian Klink
46585bee15 docs(tvix/TODO): drop builtins:fetchurl item
This has been implemented in cl/11786, cl/11790 and cl/10975.

This includes support for the `executable` and `unpack` variants of it.

Change-Id: I5bcad5b8e86b9fd341219a59ba86660b692e66e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11818
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 01:01:04 +00:00
Florian Klink
6b6a34065e feat(tvix/tracing): add tracing-tracy support
This introduces another feature flag, "tracy" to the `tvix-tracing` crate.

If enabled (not enabled by default), it'll add an additional layer
emitting packets in a format that https://github.com/wolfpld/tracy can
display.

I had to be a bit tricky with the combinatorial complexity when adding
this, but the resulting code still seems manageable.

Change-Id: Ica824496728fa276ceae3f7a9754be0166e6558f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10952
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-14 19:33:44 +00:00
Florian Klink
d25f047b9d refactor(tvix/tracing): move otlp setup into helper function
Having all this in the main control flow makes it a bit hard to read.
Moving it into a helper function makes it a bit cleaner.

Change-Id: Ibdb739dbd1e013b4f8c4aaf9b036a6bd556a1871
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11814
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-14 14:52:19 +00:00
Simon Hauser
118e69d81d fix(tvix/tracing): reduce the error logs of otlp if collector is offline
The problem is that opentelemetry_otlp tonic batch exporter tries to
exports if either the `scheduled_delay` or if the
`max_export_batch_size` is reached. Per default the
`max_export_batch_size` is set to 512 spans, which means that we try to
export these spans once that counter is reached. Each export will then
try to connect to the exporter (if that not already happening) and will
result in a `tcp connect error`.
Increasing the max_export_batch_size to 4096 will then ensure that the
export only happens if the `scheduled_delay` is met after the 10
seconds.
`max_queue_size` is also increased, because `max_export_batch_size`
should not be greater than `max_queue_size`, so similar to the default
config its set to `max_export_batch_size * 4`.
This will reduce the amount of tries to otlp if the collector is not
available and otlp enabled.

Change-Id: Ic3430006e8a104fa3b34d274678cae55b3620ce9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11791
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 10:20:31 +00:00
Simon Hauser
a857a2b978 feat(tvix/tracing): apply EnvFilter to all layers
Currently we apply the EnvFilter only to the stderr output writer.
This didn't affect any other layer, like the otlp layer, causing spans
from `h2`, `tokio_util` or other third party crate dependencies to be
always sent out via OTLP.

This changes that behaviour, applying EnvFilter to all exports, leading
to a lot less spans being exported.

Change-Id: I9f3a7233e9d0aeaa81fe08914579f0b3c80d134e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11813
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 10:18:59 +00:00
Simon Hauser
fa7ed39bf4 feat(tvix/tracing): correctly close otlp on exit
Provide a new interface for forcing a flush of otlp traces and use this
interface to shutdown otlp prior to exiting tvix-store, either if the
tool was stopped with a SIGTERM or ended regularly.
This also fixes an issue where traces were not even exported if for
example we just imported 10 paths and never even emitted more than 256
traces. The implementation uses a mpsc channel so a flush can be done
without having to wait for it to complete. If you want to wait for a
flush to complete you can provide a oneshot channel which will receive a
message once flushing is complete.

Because of a otlp bug `force_flush` as well as
`shutdown_tracer_provider` need to be executed using `spawn_blocking`
otherwise the function will deadlock. See
https://github.com/open-telemetry/opentelemetry-rust/issues/1395#issuecomment-1953280335

Change-Id: I0a828391adfb1f72dc8305f62ced8cba0515847c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-14 09:34:51 +00:00
Florian Klink
5077ca70de chore(tvix/eval): move eval docs to tvix/docs
Change-Id: I75b33c43456389de6e521b4f0ad46d68bc9e98f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11809
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-06-14 08:00:34 +00:00
Florian Klink
6947dc4349 chore(tvix/docs): move [ca]store docs to tvix/docs
Change-Id: Idd78ffae34b6ea7b93d13de73b98c61a348869fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11808
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2024-06-14 08:00:34 +00:00
Florian Klink
adc7353bd1 feat(tvix/docs): add d2 support
This provides a nicer syntax to draw some diagrams.

Change-Id: I2ab580e61399a45f01ed422ee29382e860753ffa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11807
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-06-14 07:04:23 +00:00
Florian Klink
4df3284dd0 feat(tvix/docs): add mdbook-admonish
This provides a plugin for callouts.

It needs to have additional CSS file added to `book.toml`,
which can be (re-)generated by `mdbook-admonish install`.

See https://github.com/tommilligan/mdbook-admonish/issues/171#issuecomment-2009166079
for more context.

Use it by adding a warning one to the architecture document.

Change-Id: I75c9a33d00acb603c6da10d3f9ce3485731c1672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11805
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
2024-06-14 07:04:23 +00:00
Florian Klink
9d44da01ee fix(tvix-glue/fetchers/decompression): propagate error with ?
Change-Id: I5e9d4f067f09307cb6f990ed6eca48345e2fb342
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11810
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-13 21:07:40 +00:00
Florian Klink
b992ca49a6 fix(tvix/glue/tvix_store_io): also populate input sources
These also need to be present in the input nodes of the BuildRequest.

Change-Id: Ie9b957805e42f766002581adc6182a6543c5333b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11802
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 19:59:28 +00:00
Florian Klink
c30fded880 chore(tvix/docs): move daemon protocol to its own chapter
… and add an introductionary page at the root.

Change-Id: I89ce69a4221c332b7c0d99ac26b4e26bfd248341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11806
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-13 19:23:20 +00:00
Florian Klink
d947f61d36 fix(tvix/glue/tvix_store_io): disable concurrent fetches for now
We need some shared queue, preventing the same fetches/builds from
getting triggered multiple times unnecessarily.

Change-Id: I7c4a3c66db558f5cccd66865b170242b758e3e02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11800
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-13 16:19:18 +00:00
Florian Klink
79bfa931ed feat(tvix/tracing): set release_max_level_debug for tracing
This allows explicitly opting in to get DEBUG-level log lines, by
setting RUST_LOG.

It currently also causes traces to be emitted in all cases, so we might
do some runtime filtering there too, as discussed in cl/11791.

Change-Id: I2865bb06a62465836d63196422f5f734f7165386
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11801
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 16:18:47 +00:00
Florian Klink
29eda2d5b2 feat(tvix/cli): add toplevel progress span
This gives some better feedback something is going on, gives a (named)
root for all the progress children we're drawing, and also counts the
time we are in eval.

Change-Id: Ibe81dcebf0a2b59bb0680da62e206eb5270f9d3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11798
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-06-13 16:18:15 +00:00