Commit graph

20638 commits

Author SHA1 Message Date
Florian Klink
10c7701ac1 docs(tvix/castore): drop wrong comment
This doesnt panic anymore.

Change-Id: I1987f8b0640b1e5fe33abcbf46164449c766fd0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11883
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-06-29 17:51:33 +00:00
Florian Klink
86ab052fe0 feat(tvix/boot/tests): closure-nixos-bigtable
This uses the `preStart` mechanism to boot up a bigtable emulator, which
we point directory/pathinfoservice to.

Change-Id: Icb383403cb2d1f7e19c71b6298e82b391f9c8d22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11895
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-29 16:56:08 +00:00
Florian Klink
6cf8003fa1 fix(tvix/boot/tests): escape addrs
Otherwise some URL parameters (containing &) might break the shell.

We cannot use `$PWD` anymore, but that's statically known anyways.

Change-Id: I739249111461ecdb33d42a48a4352f7612fce90f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11894
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-29 16:56:08 +00:00
Florian Klink
50ed7f27d0 feat(tvix/boot/tests): add preStart parameter
This allows running some code before tvix-daemon is started up, which is
helpful to spin up some mock services.

Change-Id: Ie7b6eaaf76c40def493f50879ee5255fd1ebd2fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11893
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-06-29 16:56:08 +00:00
Florian Klink
0fd4a1a655 fix(tvix/boot/tests): set SSL_CERT_FILE
This is necessary for http clients to successfully construct.

Change-Id: If0d968e964f5c58d9956b92ee0ab3d97c39391d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11892
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-29 16:56:08 +00:00
Florian Klink
7294a2f9aa feat(tvix/boot): introduce useNarBridge parameter
This is still useful to test the ingestion part of nar-bridge.

Reintroduce the logic removed in cl/11398, but put it behind a
`useNarBridge` parameter, defaulting to false.

Add an additional target to tvix.boot.tests,
`closure-nixos-nar-bridge`, which does the same as `closure-nixos`,
except seeding the contents through nar-bridge.

Change-Id: Iee87d5d9c7b7d02df11ee47fbc3f1fbe951935e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11891
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-06-29 16:38:03 +00:00
Florian Klink
4ded170e97 refactor(tvix/boot/tests): only use exportReferencesGraph when needed
These fields don't need to be set if isClosure is false.

Change-Id: Id29ed034e0757c61cef09885962dfc5507ca106c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11890
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 16:38:03 +00:00
Florian Klink
de8340335e feat(nix-compat/nixcpp/conf): make fields pub
Otherwise these are not accessible after parsing if parsed from another
crate.

Change-Id: Iba7b442722a2eb1d37c2c496a45aa559aa49c1f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11889
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-28 14:04:14 +00:00
Florian Klink
03c9917749 feat(nix-compat/nixcpp): init nix.conf config parsing
This allows parsing files like `/etc/nix/nix.conf` into the `NixConfig`
struct.

Change-Id: I90b25f43c429dd56127500ff5068e83852adee13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11888
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-28 12:23:03 +00:00
Florian Klink
327d115f3f feat(nix-compat/narinfo/pubkey): derive Clone, [Partial]Eq
We want to use this in another struct that we want to compare and clone.

Change-Id: I5c11d589a4922d85e190e1a9199468411016ff1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11887
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-06-28 12:23:03 +00:00
toastal
ade0781416 docs(tvix): create Welcome + Contributing sections
A lot of this content needs to be duplicated from the monorepo’s root
directory. However, some of the content was not relevant or could have
been more condensed in descriptions or split up for consumption that
also matched the styles of some other library documentation I had been
looking at for examples. I wanted to add the new access to Admonish
callouts but also aimed at the readablitiy in one’s text editor for the
user that needs to `cat` the docs.

In the future, a “Introduction” page would be nice to add to “Welcome”,
but I don’t think that is as clear.

Change-Id: Ibfc169c5fc3501d7eeb9a76423c6f64cd68897aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11869
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-27 07:16:23 +00:00
toastal
fbce56b849 docs: prompts in shell sessions shouldn’t be selectable
these just indicate the start of the prompt by convention & if the user
is root or not with `#` & `$` respectively

Change-Id: Id65f5f879b067ee96715b3e7dc63f68c7ad791b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11868
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-27 07:16:23 +00:00
Florian Klink
cd48566173 fix(tvix/glue): reject unknown attrset args for fetch builtins
This now uses UnexpectedArgumentBuiltin in case builtins.fetchurl or
builtins.fetchTarball are called with the wrong arguments:

```
note: while evaluating this Nix code
 --> [code]:1:1
  |
1 | builtins.readDir (builtins.fetchTarball { url = "https://git"; hash = "sha1-NKNeU1csW5YJ4lCeWH3Z/apppNU=";})
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E040]: Unexpected agrument `"hash"` passed to builtin
 --> [code]:1:19
  |
1 | builtins.readDir (builtins.fetchTarball { url = "https://git"; hash = "sha1-NKNeU1csW5YJ4lCeWH3Z/apppNU=";})
  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ while calling this builtin

```

Change-Id: I51124255a46b78d3cf4dc89a1eca9e68750858d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11878
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-26 16:06:52 +00:00
Florian Klink
aa7d125c12 refactor(tvix/eval): prefix ctx iterators in NixString
NixString::iter_plain() didn't make much sense, especially without a docstring.

Rename it to iter_ctx_plain(), and copy the docstring from NixContext.

Do the same for the two other context element types too.

Change-Id: I1bbfcb967d8d9b14487d069bfe3a1f762253ef4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11882
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-26 14:25:20 +00:00
Florian Klink
b757897e97 feat(tvix/glue): implement builtins.toFile
We currently only had a dummy implementation that didn't actually
persist the files in the store(s).

Copy the contents to the BlobService, and do the output path calculation
as part of the upload.
Use the plain context elements to construct the references.

Change-Id: Ibdaf7a645ddc31e847faa4b87a79f2f95116a7ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11879
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-26 14:25:20 +00:00
Florian Klink
888028b674 docs(tvix): use # for toplevel headings
See discussion in cl/11869. Let's use `#` consistently, we also use `##`
for deeper nested subheadings.

Change-Id: Ie25ebda708639fb617d456c275ae5a264fc4ce85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11881
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: toastal <toastal@posteo.net>
2024-06-26 14:22:19 +00:00
Florian Klink
50150fa3c8 feat(tvix/glue): add eval-fail-* support to glue tests
Add some eval-fail-fetcht-*.nix test cases.

Change-Id: Ic02beabf120d1de8df6f9c098d5a074127069f81
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11880
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 11:17:30 +00:00
Florian Klink
89361b2a7f fix(tvix/tracing): make cargo check and clippy happy
In case the otlp feature is not enabled, these generate warnings during
`cargo check`.
Fix by moving some imports into their functions, or using the
fully-qualified name (and one #[allow(unused_mut)])

Change-Id: I5afd89dcd4c772b6002cebdd5d0469932eacfdac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11873
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-26 06:27:15 +00:00
Florian Klink
a0993e7304 feat(tvix/eval): add ErrorKind::UnexpectedArgumentBuiltin
Change-Id: Ieb091b32aad566719fbe8604c4a589f5ccaaf6b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11877
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
970e2a045b refactor(tvix/eval): rename UnexpectedArgument{,Formals}
There's other places where unexpected arguments can be provided, like
in builtins.

Make space for that error type.

Change-Id: Ic831497a3a1dd36a3a184bedadcf1374bf0ae6db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11876
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-26 04:51:31 +00:00
Florian Klink
ea6f511241 refactor(tvix/glue): return a parsed Url in NixFetchArgs
The only two consumers (fetchurl, fetchtarball) of these do try to parse
it as URL, so do it in the helper.

Update url_basename to take a &url::URL, not a &str.

Also update the test to use rstest for the fixtures to reduce some
boilerplate there.

Change-Id: I1f85fe2803060dc4423e673cb7b9f9bf799d09b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11875
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
540e566900 refactor(tvix/glue): take &CAHash, not CAHash
We use a bit less cloning that way.

Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-26 04:51:31 +00:00
Florian Klink
78eb22c54d feat(tvix/glue): handle regular file at builtins.path import
If builtins.path is passed a regular file, no filtering is applied.
We use the just-introduced file_type function in the EvalIO trait for
that.

This means, we don't need to pass through filtered_ingest, and can
assemble the FileNode directly in that specific match case.

This also means, we can explicitly calculate the sha256 flat digest,
and avoid having to pipe through the file contents again (via
blob_to_sha256_hash) to construct the sha256 digest.

Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
080654aaf9 feat(tvix/eval): add file_type method
This allows peeking of the type at a given path.

It's necessary, as an open() might not fail until you try to read()
from it, and generally, stat'ing can be faster in some cases.

Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Ilan Joselevich
04f04cfc27 refactor(tvix/eval): drop placeholder impls of builtins in glue
The builtins `placeholder` and `filterSource` are now implemented in
tvix-glue and so we no longer need their placeholder "implementations" in
tvix-eval.

Change-Id: Ibf161ccf1ad40103e9fbb688e9f6be58e7772af1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11867
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-22 21:50:30 +00:00
toastal
bf541acf29 docs: use admonish for callouts
Change-Id: Icec7321a49c40e7f7bcb1e4b307f13b8db4eac25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11862
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-21 08:08:17 +00:00
Florian Klink
8dde022d31 docs(tvix): drop done TODOs in the O11Y section
OTLP shutdown behaviour was fixed in cl/11803, and tracy landed in
cl/10952 and cl/11815.

Change-Id: I310dfcc41ba62ce5b006e75a35feb839327b0e7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11866
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Florian Klink
7f625e2fc1 refactor(tvix/*store): remove some trait bounds
We don't need to require these things for these impl blocks yet.

Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Simon Hauser
639a00e2ab feat(tvix/tracing): gRPC trace context propagation
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.

The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.

This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.

Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-20 19:21:01 +00:00
Simon Hauser
2b20d8d82d feat(tvix/castore): instrument tokio task with current span
By default tokio::spawn does not instrument the spawned task with the
current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do
this manually for all tokio::spawn functions in functions that are
instrumented.

Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 15:48:14 +00:00
Simon Hauser
bd8d74a3ee feat(tvix/tracing): optional progressbar
Disable the progressbar on default and provide a interface for
optionally enabling the progressbar.

Change-Id: I0e31b1957e80cf64a8dcf65c6ceb3713975b8220
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 10:28:54 +00:00
Florian Klink
1446e3be99 docs(tvix/docs): document nar renderer optimization potential
Change-Id: I747ab5c8ba3424828b1c88e0353170ad0efab3a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11849
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-20 10:28:33 +00:00
Florian Klink
8137077a74 feat(tvix/store/utils): detect gRPC NAR calculation service
We were currently always using SimpleRenderer, which would mean the
client would download every blob locally to calculate the checksum,
which of course is very slow.

Detect this special case and create a second instance (and client) for
now.

Change-Id: If39a862a5311e71c8073ac4e663f6c5dd437072e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11848
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 10:28:26 +00:00
Ilan Joselevich
9fd601e53d chore(tvix): patch crate2nix to drop darwin dontStrip
This patch has already been applied on upstream but there hasn't been an
official release yet so we patch it ourselves.

Upstreamed patch: 0209f258cd

Follow-up of: https://cl.tvl.fyi/c/depot/+/11856

Change-Id: If56071ce5753fd26e2b4c203cd831bbe5d329009
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-18 14:13:11 +00:00
Ilan Joselevich
92c5433304 chore(tvix): move filterRustCrateSrc to utils.nix
Change-Id: Ib6bae1ea0457d3309a6ec1e08b9e4b320523c161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11857
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 15:26:19 +00:00
Ilan Joselevich
ac51803ad8 chore(tvix): Remove no longer needed libiconv addition for darwin
Since https://github.com/NixOS/nixpkgs/pull/220212 libiconv is now
automatically added for Darwin.

Change-Id: I2aaa266bcbc4d5a8256131aed36b4c2d040a8758
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11856
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-17 15:08:10 +00:00
Ilan Joselevich
210b4277b6 chore(tvix): Use callPackage instead of import on Cargo.nix
Since crate2nix v0.11 the new preferred way of importing Cargo.nix is
with callPackage instead of import.

Change-Id: Ifcc5c21e6c101cb9d0b3f1e8025b5ddb8cf99f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11855
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:46 +00:00
Ilan Joselevich
fbe29615df chore(tvix): Remove no longer needed rlib stripping fix for darwin
This issue was fixed in https://github.com/NixOS/nixpkgs/pull/255900

Change-Id: I4f569099e62c4284bdea18a512387735f139e702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:04 +00:00
Florian Klink
28b692fd50 feat(tvix/tvix-store): improve progress bars
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.

Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.

Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.

Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 12:57:34 +00:00
Florian Klink
cfab953094 feat(tvix/store): display progress for NAR calculation
This is currently still taking a noticeable amount of time, so make sure
we show it is happening.

Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 09:47:54 +00:00
Florian Klink
71a29ceff4 docs(tvix/store/import): update import_path_as_nar_ca docstr
Change-Id: I697172220efe73b2a215185027df09f815d32627
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11846
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 08:49:52 +00:00
Florian Klink
4be250109c refactor(tvix/store): drop FUSE multithread workaround for Darwin
The underlying issue in macFUSE has been fixed in
https://github.com/osxfuse/osxfuse/issues/974.

Bump our `macfuse` in nixpkgs to a version containing the fix.

This can be removed while our nixpkgs pin is bumped past
https://github.com/NixOS/nixpkgs/pull/320197.

Change-Id: Ia0e644fb13198e45018b0a218647ef211acf4df1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11824
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-16 12:02:11 +00:00
Florian Klink
7e42b4f314 fix(tvix/store/bin): fix shutdown behaviour for FUSE
Both umounts happening from another process, as well as tvix-store
itself calling umount() on FuseDaemon will cause the FUSE worker threads
to terminate.

So far there was no nice way to wait on these threads to be terminated
from multiple places, causing the `tvix-store mount` command to only be
terminated if interrupted via ctrl-c, not via an external umount.

Update FuseDaemon to use a ThreadPool, which gives us a join primitive
over all threads, that can also be called from multiple places.

Await on a join() from there to end the program, not the ctrl-c signal
handler as it was before.

Using FuseDaemon from multiple tasks requires Arc<>-ing both the
ThreadPool as well as the inner FuseSession (which also needs to be
inside a Mutex if we want to unmount), but now we can clone FuseDaemon
around and use it in two places. We could probably also have used an
Option and drop the FuseSession after the first umount, but this looks
cleaner.

Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-06-16 12:02:11 +00:00
Yureka
452299dcd2 feat(castore/directory): add cache combinator
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 08:42:53 +00:00
Yureka
30e72d2d52 refactor(castore/directory/objectstore): use new order validation logic
Use the new helper introduced in CL 11708 instead of rolling our own.

Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 08:40:20 +00:00
Yureka
daada1b2fa refactor(castore/directory): separate order logic from ClosureValidator
ClosureValidator was previously only suitable for a very narrow use case:
Validating incoming uploads, which are in leaves-to-root order.
This is because the ordering validation was hard-wired into the add()
function.
This
- Re-name ClosureValidator to DirectoryGraph, which is more suitable
  since it actually stores the Directory structs and is drained in the end.
- Move the ordering-related logic to a separate OrderValidator, which
  can be used independently.
- re-write DirectoryGraph to be a general purpose
  validator which can accept the input in both orders
  and can be drained in both orders as well.

This means the DirectoryGraph and OrderValidator can now serve
multiple new purposes:
- Validating the incoming closure on the client while downloading.
- Validating the incoming closure downloaded in a caching layer from the
  `far` cache, and re-order it for insertion into the `near` cache.

Change-Id: I2b4b226348416912d7a31935bec050e53d911b70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 05:01:07 +00:00
toastal
c2a9ad3583 docs: move Registration to the top; minor reword
When trying to follow Review documentation, I got tripped up on the
“Gerrit setup” section that was asking about SSH keys since I hadn’t yet
gotten an account setup but the registration setion was near the bottom
of the document despite being a requirement for setting up Gerrit.

Included words specifying SSO, specifies that the TVL account is an
alternative method, & commit→patch since you send a patch with the
commit.

Change-Id: I465c8a79979b368ab01bcebdc7e5ef16e65dd341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11823
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-15 17:42:16 +00:00
Florian Klink
01aadace80 feat(tvix/cli,store): add tracy feature
If compiled with this features, this emits packets compatible with the
[Tracy](https://github.com/wolfpld/tracy) format.

Change-Id: I330f5d85ab290abe51f2df38dc55464f3ccfc6cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11815
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-15 05:24:37 +00:00
Ilan Joselevich
cd0b92419a docs(tvix/TODO): dibs on crate2nix WASM
Change-Id: Idde9aaa31ecaa9570559ffe3271bc23b7a71133c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11820
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-15 04:02:33 +00:00
Florian Klink
86928b5e7f docs(tvix/TODO): add builtins.path roundtrip for flat
Change-Id: I8291d2fdfa2ea31ec24c82dcfb447fa9eaa3a961
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11819
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-15 04:02:33 +00:00