Commit graph

2669 commits

Author SHA1 Message Date
Ryan Lahfa
f71bb351d2 feat(tvix/glue): introduce test suite for context strings
This is an additional test suite on the top of the Nix ones
for context strings matters.

It already smoked out multiple mistakes and potential bugs and non-deterministic result from the evaluator.

It uses a similar technology as the one in the tvix-eval albeit we instantiate a fully fledged evaluator
with in-memory store.

We copy the files instead of symlinking them because crates are built in
isolation, so symlinks cannot work.

Change-Id: I63ae225ce4f83c6e2c8ccd60d779c2f8eb9d08fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10619
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 17:31:07 +00:00
Ryan Lahfa
75cc52ddb1 fix(tvix/eval): getContext merges underlying values
Previously, we were assembling very naively an attribute set composed of context we saw.

But it was forgetting that `"${drv}${drv.drvPath}"` would contain 2 contexts with the same key, but
with different values, one with `outputs = [ "out" ];` and `allOutputs = true;`.

Following this reasoning and comparing with what Nix does, we ought to merge underlying values systematically.

Hence, I bring `itertools` to perform a group by on the key and merge everything on the fly, it's not
beautiful but it's the best I could find, notice that I don't use
`group_by` but I talk about group by, that is, because `group_by` is a
`group_by_consecutive`, see
https://github.com/rust-itertools/itertools/issues/374.

Initially, I tried to do it without a `into_grouping_map_by`, it was akin to assemble the final `NixAttrs` directly,
it was less readable and harder to pull out because we don't have a lot of in-place mutable functions on
our data structures.

Change-Id: I9933c9bd88ffe04de50dda14f21879b60d8b8cd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10620
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-17 17:25:24 +00:00
Ryan Lahfa
97e6c39dcd fix(tvix/eval): context-aware dirOf
`dirOf` forgot to accepts contextful strings, e.g. derivations and propagates this context
further.

Change-Id: I6c05944a3ce5073e243e7676c9be56c48407d657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10618
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-17 07:57:43 +00:00
Ryan Lahfa
c2d0e245e2 fix(tvix/eval): context-aware… hasContext
Yes, `hasContext e` should work where `e` is a contextful strings, otherwise, it is really useless.

Change-Id: I5eb071fc257217d6e8a63fe519132ebd98186696
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10617
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 07:57:38 +00:00
Ryan Lahfa
bc8fb825c7 feat(tvix/eval): move away from test_generator to rstest
`test-generator` has not been updated in the past 2 years.
`rstest` has not been updated in the past 5 months.

This is an improvement in the maintenance state… I guess?
We get also new features, it changes the name of the tests with numbers too.

Change-Id: I5376104c7704f525dba7524da78daa09867cc669
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10623
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 07:34:02 +00:00
Florian Klink
850a4bfc7b chore(third_party/crate2nix): run tests in /build/source
Cherry-pick of https://github.com/nix-community/crate2nix/pull/328. This
should fix rstest runs inside crate2nix-generated derivations.

Change-Id: I9d393768f7f764e33c5938bd8fa14bd1bb0b72e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10650
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-17 07:33:12 +00:00
edef
b624edb2ae fix(tvix/eval): lift VM ops over Catchable
We want to handle bottoms in a consistent fashion. Previously this was
handled by repetitive is_catchable checks, which were not consistently
present.

Change-Id: I9614c479cc6297d1f64efba22b620a26e2a96802
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10485
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-16 19:20:28 +00:00
sterni
99615002a7 fix(tvix/cli/ci): prevent unnecessary dep on compared out/drv path
Change-Id: Ieab16fd5fad67a4ed0124f118aef8abbefd51cdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10646
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-16 16:50:59 +00:00
Florian Klink
e2c79e39f0 refactor(nix-compat): use StorePathRef for hash derivation modulo
Rather than passing strings around, use a StorePathRef.

This makes things a bit more typesafe, and more aligned with what we
want to do in b/264.

Change-Id: Ib7080addf27e7f1a9c8da1d8aaa66744468e3b5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10633
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-16 08:37:07 +00:00
Florian Klink
c8114810c9 chore(third_party/overlays): bump crate2nix to 0.13.0
We need to vendor in the package expression, as it's not possible to
override cargoHash.

Change-Id: Ib123647bb9b96d41f4630daa431d020f1cb8d4fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10624
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-15 23:14:21 +00:00
Florian Klink
f0a750bcb7 feat(tvix/build): add CLI entrypoint
This starts a BuildService as a separate process, currently defaulting
to the DummyBuildService.

Change-Id: Ic206f00831641d3ffebaa44883b7dc053700b9ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10631
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-15 18:54:35 +00:00
Florian Klink
170e0cdfad feat(tvix/build): add from_addr method
This allows constructing a BuildService from a URI, similar to how it's
done in tvix-[ca]store.

Change-Id: Ib962b329535c6c7e378ab7ac7f4dd254366497b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10630
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-15 18:49:33 +00:00
Florian Klink
5aa9b29d8c feat(tvix/build): add gRPC client
Change-Id: I0d917a9a308227b13d096def22945db917830d18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10629
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-15 18:41:58 +00:00
Florian Klink
c01ec8ee38 feat(tvix/build): add GRPCBuildServiceWrapper
This produces a gRPC BuildService server for anything implementing our
BuildService trait.

Change-Id: I59c690a432b5e1f59209fd67e2718cb8c935adf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10628
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-15 18:41:58 +00:00
Florian Klink
4fb4fc263d feat(tvix/build): add BuildService trait
Also provide a dummy implementation that just fails on any build that's
requested.

Change-Id: I0df743a730c5331ec9ce6e97a966abe18ce067f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10627
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-15 18:25:19 +00:00
Florian Klink
9fd15ba506 refactor(tvix/glue): have derivation_to_build_request consume inputs
Determining the inputs might trigger additional builds/substitutions,
so answering these lookups via a lambda in a lazy fashion gets
complicated.

You end up assembling the list of input nodes upfront, and the lambda
will just be a dumb lookup into that preassembled list.

Rather than doing that, simply have derivation_to_build_request leave
the work of determining the inputs to the caller.

Change-Id: I75880132916c76b930807c989090da298b6891bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10626
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-15 18:25:19 +00:00
Florian Klink
83291ff51e refactor(tvix/glue/known_paths): drop some unused stuff
This are leftovers from the "reference scanning" approach (which we
didn't end up using).
We still want a concept of known paths, so we can trace IO into
storepaths back to the build recipe that'll produce it, so let's keep
the rest of this struct around.

Change-Id: I73d38e21e5b97950b8fc2a42176cae5f80d371c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10632
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-15 18:25:19 +00:00
Florian Klink
c5e2832cbd feat(tvix/castore): implement Ord for node::Node
This allows assembling BTreeSets of node::Node.

Change-Id: I97b83be5ffc3e891307a8ef2b5fc31e38b747a62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10625
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-15 18:19:15 +00:00
Ryan Lahfa
2750e1e640 fix(tvix/eval): catchable-aware builtins
A bunch of operations in Tvix are not aware of catchable values
and does not propagate them.

In the meantime, as we wait for a better solution, we just offer this
commit for moving the needle.

Change-Id: Ic3f0e1550126b0847b597dfc1402c35e0eeef469
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10473
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-14 03:37:17 +00:00
Ryan Lahfa
61209f0775 feat(tvix/glue): input derivation context tests
`args` was not propagating context, here's a regression test for it.

Change-Id: I8b6a3148508d40df0077128f0bafe68c098a03bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10610
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-12 22:32:22 +00:00
Florian Klink
d516ce56b1 feat(tvix/glue/derivationStrict): support __structuredAttrs
This adds support to handle the __structuredAttrs argument, which can be
passed to builtins.derivationStrict.

If __structuredAttrs is passed, and set to true, most of the arguments
passed to builtins.derivationStrict are not simply coerced to a string
and passed down to "environments", but instead kept in a more structured
fashion.

Inside ATerm, which is what's relevant as far as path calculation is
concerned, a virtual `__json` environment variable is present,
containing these structured values.

Inside Builds, these structured values are not made available as an
environment variable, but a JSON file (and source-able bash script).

This will need to be respected once we start emitting BuildRequests,
and for that we can probably just parse the `__json` key in
Derivation.environment again - or keep this additionally in
non-serialized form around during Evaluation.
No matter what, this is left for a followup CL.

The existing handle_derivation_parameters and populate_outputs helper
function were removed, as __structuredAttrs causes quite a change
in behaviour, and so handling both in the same place makes it more
readable.

There's some open questions w.r.t. string contexts for structured attrs
itself. A TODO is left for this, but at least path calculation for
individual structured attrs derivations are correct now.

Part of b/366.

Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 22:25:35 +00:00
Florian Klink
82540717d6 feat(tvix/eval): make into_json public
Allow other crates (like tvix-glue) to look at a Value in JSON, which is
used by the structured attrs feature.

Change-Id: Iba02ace6e11a74c3f9b19dcbef4b008b76dec046
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10602
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-12 22:25:35 +00:00
Ryan Lahfa
c955560767 docs(tvix/glue): misc reflowing of comments
My OCD could not be stopped.

Change-Id: I2bf504fe0865a5084ad02aee18e6180a8a3e19d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10609
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-12 22:08:34 +00:00
Florian Klink
ce66af09a7 fix(tvix/glue/tvix_build): fn_input_drvs_to_output_nodes
The Derivation input_derivations field contains a list of input
derivations and (a subset of their) output names.

This means, multiple nodes can be returned, so return a Vec.

Also, update the name to better reflect the nodes are the nodes of the
selected outputs, not a node representing the .drv file itself.

Additionally, use a proto::node::Node (the naked enum), rather than
proto::Node, which wraps this in an optional struct field until
realizing the BuildRequest.

Change-Id: Iec5620b5d7ac0462f2c76acac4abcaeea2de0aad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10608
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-12 20:38:38 +00:00
Florian Klink
639ee19101 refactor(tvix/glue/tvix_store_io): async store_path_to_node
Provide a store_path_to_node_sync function which uses the runtime handle
to block on the async function internally, but make store_path_to_node
itself async, so it can call async functions internally.

We'll use that later when triggering builds and waiting on their
results.

Change-Id: Idae9da7aa5b0878e0d3a2eba34ea2623e1ba84b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10607
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 20:38:36 +00:00
Florian Klink
b59df53774 refactor(tvix/store/pathinfoservice): make more generic
We don't need Arcs in most of the cases, we're fine with some container.

Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 20:37:35 +00:00
Florian Klink
7d51193f7d refactor(tvix/store/nar/renderer): don't require Arc, Clone or Sync
To render NARs, we're fine with a simple AsRef to a BlobService and
DirectoryService. We just need to have the function pass back the
references, so we can reuse it after the recursion.

Change-Id: I8a1b899134ddda26cf14aa829a08383986101850
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-12 20:37:03 +00:00
Florian Klink
9cdb5e17a4 fix(tvix/eval): fix JSON error types
The error message is misleading. The errors we return can happen both
during serialization or deserialization, though the messages suggested
the latter only.

Change-Id: I2dafe17ec78ee75cab5937a3a81540fda3175eac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10603
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-12 09:22:37 +00:00
Florian Klink
4d135bcfa2 feat(tvix/castore): implement CombinedBlobService
First attempt on composition of BlobServices.

Change-Id: I6e70248007edfd322a503fd40c1c4b4300cbc30c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10587
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-09 17:32:03 +00:00
Florian Klink
719cbad871 feat(tvix/castore/blobsvc): add chunks method
This adds support to retrieve a list of chunks for a given blob to the
BlobService interface.

While theoretically all chunk-awareness could be kept private inside
each BlobService reader, we'd not be able to resolve individual chunks
from different Blobservices - and due to this, not able to substitute
chunks we already have in a more local store.

This function allows asking a BlobService for the list of chunks,
leaving any actual fetching up to the caller (be it through individual
calls to open_read), or asking another store for it.

Change-Id: I1d33c591195ed494be3aec71a8c804743cbe0dca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10586
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-09 17:31:32 +00:00
Florian Klink
9596c5caff refactor(tvix/castore): do clone inside a scope
Make it clear this is only used inside the scope.

Change-Id: Ie94f88d7f0fb58cd4bf9c2f1176000b272e6f2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10585
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 16:20:19 +00:00
Florian Klink
9de1ebf23e feat(tvix/castore/grpc): instrument some more functions
Change-Id: Icedb148c88c5f4a3b2242ed12df1dd8692af94fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10584
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 16:18:48 +00:00
Florian Klink
99f675ecef refactor(tvix/store/nar/import): use AsRef
We need to be a bit careful and pass the BlobService around (similar to
how we already do with the directory_putter), but that allows getting
rid of a bunch of annoying trait bounds.

We also stop spawning additional tasks where we can just use block_on.

Change-Id: If36de0ee947d2c779d20a384308241d2262d4764
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10580
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-01-09 14:18:26 +00:00
Florian Klink
0009383c07 refactor(tvix/castore/directorysvc): AsRef traverse_to
Change-Id: I641bd4ab3de591a013f03137f1e16295946315f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10579
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-09 14:15:55 +00:00
Florian Klink
b1c556b7e1 refactor(tvix/castore/blobservice/grpc): remove fn pointer hack
It looks like the workaround isn't necessary anymore.

Change-Id: Ifbcef1d631b3f369cac3db25a2c793480043f697
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10583
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:13:24 +00:00
Florian Klink
89882ff9b1 refactor(tvix): use AsRef<dyn …> instead of Deref<Target= …>
Removes some more needs for Arcs.

Change-Id: I9a9f4b81641c271de260e9ffa98313a32944d760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10578
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:08:22 +00:00
Florian Klink
8fbdf72825 feat(tvix/castore/blobsvc/grpc): rm VecDec, fix docstring
The docstrings were not updated once we made the BlobService trait async.
There's no more need to turn things into a sync reader.

Also, rearrange the stream manipulation a bit, and remove the need to
create a new VecDeque for each element in the stream. bytes::Bytes
implements the Buf trait.

Fixes b/289.

Change-Id: Id2bbedca5876b462e630c144b74cc289c3916c4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10582
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:03:21 +00:00
Florian Klink
17eaacb139 feat(tvix/glue): add test for output path calculation
This test serves as a minimal reproducer for output path calculation.

Derivations with the same name and output hash, but different build
recipe should end up with the same outPath.

However derivations with different name should end up with a different
outPath.

Change-Id: I555be59dd87ea675a0816188ed973f96c311e4e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-01-08 11:13:16 +00:00
Florian Klink
986e9b73c3 feat(tvix/build): add BuildRequest validation
Change-Id: I8182e4c4a7e5694c1e6f1f56ce092751c22adf4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10538
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-06 15:33:12 +00:00
Florian Klink
c9c95f4ef3 chore(tvix): bump itertools
Change-Id: I2c798a30d3e69789c7d9120422bfdd67b1560dd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10565
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-06 15:33:12 +00:00
Ryan Lahfa
cbcd078684 chore(tvix/castore): fix the docstring for process_entry
It was a `//` not a `///`.

Change-Id: Iee3e8c116d73b5dd8a41c027153714415a66695f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-06 01:39:41 +00:00
Florian Klink
00aa3260c9 fix(tvix/build): fix inconsistencies around paths
Change-Id: I76ce4e042f6fbe4bc897767b55ce4eb1f7269e5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10541
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-01-05 16:49:10 +00:00
Florian Klink
5f0360c566 refactor(tvix/glue): simplify TvixStoreIO further
We don't need to spawn in all these places, we can just block_on
directly, this is all IO bound.

This also means, we don't need to clone any of the service handles
(except preserving clone-ability of the BlobService).

Change-Id: I7d90f4d6a263a98491caa071ada538a5197a5472
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10540
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-05 16:49:10 +00:00
Florian Klink
4284cd82ef refactor(tvix/glue): simplify store_path_to_[root]_node
This was wrongly named, it returns a specific node at a subpath.

Also, this code can be simplified a lot - we don't need to spawn
additional tasks, and can get rid of some clones too.

This is also where we need a certain build - so add some TODO to block /
fetch here.

Change-Id: Id26d7bd80f7a2095121e642b3f7716de78d6b6a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10539
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-05 16:43:37 +00:00
Florian Klink
5a82736122 chore(tvix): bump test-case dep to 3.3.1
Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-01-05 16:43:34 +00:00
Florian Klink
70e40f5ce8 docs(tvix/build): document requirements for paths and sorting
Change-Id: I466fac025b9b5b2279a6188fa35cb1f7d8ab7bc5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10536
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-05 16:43:34 +00:00
Florian Klink
8e794478bf feat(tvix/store/pathinfosvc): call validate
Change-Id: I557ed67a04112ba3e363303637d1988309250a93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-05 16:43:34 +00:00
Florian Klink
6b42aef88d fix(tvix/castore): validate Option<Node>
Extend our validation function to also check for the None case.

Change-Id: Ib75f880646d7fb3d66588f1988e61ec18be816a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10534
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-05 16:43:34 +00:00
Florian Klink
f20969de9b refactor(tvix/castore): relax trait bounds for DS
Make this an `AsRef<dyn DirectoryService>`.

This helps dropping some Clone requirements.

Unfortunately, we can't thread this through to TvixStoreIO just yet.

Change-Id: I3f07eb28d6c793d3313fe21506ada84d5a8aa3ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-05 16:43:34 +00:00
Florian Klink
3297843bf1 chore(tvix): run clippy with --{benches,examples}
While doing the obvious thing, ensuring clippy doesn't complain on our
benchmarks and examples, this will also flag build failures on them,
and contrary to cl/10301 not pollute the /nix/store with stuff we can't
execute anyways.

Change-Id: I7ee1c2babbc67dac9794274d05bb9d6e1251ed01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10544
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-05 16:37:30 +00:00
Aspen Smith
e66dcba195 fix(tvix/eval): Update eval benches for new API
cl/10475 updated the API of tvix_eval::Evaluation, but didn't update
this benchmark to use that new API.

Also, fixes the docstring to no longer specify that there is a "given
snippet".

Change-Id: Ibb8285731849dbeec814e2585bbaa36f22368afe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10542
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2024-01-05 16:30:51 +00:00
Ryan Lahfa
54ad5a19e7 feat(tvix/cli): test firefox (drv|out)Path
With context strings, we are now able to reproduce up to firefox.

Until a more problematic example can be found, we should at least
lean on this particular example.

Change-Id: Ibeaf799d26e1f160b6c93f8ccd978702a2a7e3e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10460
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:48:03 +00:00
Ryan Lahfa
de6c8b7546 feat(tvix/cli): context-aware raw printing
Raw printing will transform the result into a string
and print it.

In case of a contextful string, this will fail by default, as expected.

Change-Id: I5e564329e7b001adc57a77a9153b4425cb332bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10457
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:48:03 +00:00
Ryan Lahfa
f44ac2a594 feat(tvix/eval): context-aware split
Nix does something like:

```cpp
        NixStringContext context;
        const auto str = state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.split");
```

And then do nothing with that context, therefore, we follow them and
make `split` aware of the context but still do nothing with it.

Change-Id: I4fee1936600ce86d99d00893ca3f64013213935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10428
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 23:24:07 +00:00
Ryan Lahfa
556e52c9cb feat(tvix/eval): impl unsafeDiscardStringContext
Change-Id: I7f0cc42cbebfe5cd27bf6d4f58a4af927b83646a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10423
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:24:07 +00:00
Ryan Lahfa
099ca6b7c0 feat(tvix/glue): contextful derivation
We calculate the input context by performing the union of context
over all input of the derivation.

Then, we just pass the rest of it to the remaining machinery.

Finally, we re-emit an `outPath` and a `drvPath` containing the expected
contexts.

Change-Id: I74905fb258b5bee8b08d1208c9eb87f51b92a890
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10436
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:23:33 +00:00
Ryan Lahfa
20c894e232 feat(tvix/glue): context-aware toFile
This removes the reference tracking and uses instead the context for
references and returns some.

Change-Id: Ic359ca6b903b63f1a9c679c566004c617b792442
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10435
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 19:43:10 +00:00
Ryan Lahfa
cc098b9aaa feat(tvix/eval): contextful coercion of files
In the past reference tracking system, `tvix-io` glue was appending
plain paths in the known path state.

Now, we make up for this by just making contextful coercion of file
imports.

Change-Id: Ieb9b04dd83302c77909252d5f7733857ac3cf8fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10443
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:50:29 +00:00
Ryan Lahfa
951854defc feat(tvix/eval): contextful string coercion
String with contexts are always coerced to a string with the same
context.

Change-Id: I224814febd9cad196bb28876793e76bed564dc72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10440
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-01-03 18:49:57 +00:00
Ryan Lahfa
9d43c26576 feat(tvix/eval): contextful == of derivations
Otherwise, you just fail because they are not... contextless strings!

Change-Id: I0b8f63a18cd89c3841b613d41c12ec4ee336f953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10442
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-03 18:49:57 +00:00
Ryan Lahfa
207f6fed46 feat(tvix/eval): match DO NOT propagate context
`match` silently ignore the input context and do not propagate
it and successful matches.

The why is unclear but nixpkgs does rely implicitly on this behavior
because dynamic attribute selection cannot be done with contextful
strings.

Change-Id: I5167fa9b2c2db8ecab0c2fb3e9895c9cfce6eeb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10441
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:49:57 +00:00
Ryan Lahfa
09ec8b6fcf feat(tvix/eval): implement getContext primop
Change-Id: I2c5068a28f9883a01b0ff80a5e5ab32ba18bfc1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10437
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-03 18:20:43 +00:00
Ryan Lahfa
96d06031af feat(tvix/eval): context-aware replaceStrings
And it also preserve the original context if it exists.

Change-Id: I904f7c13b7f003a267aace6301723780fccaafb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10434
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 18:16:09 +00:00
Ryan Lahfa
cbd22af2b5 chore(tvix/eval): note on context-aware hashString
It must propagate context too.

Change-Id: If57c22c9723ea02aa013f69d3dcf96054476d8de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10433
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:15:36 +00:00
Ryan Lahfa
e372b1d1a5 feat(tvix/eval): ${} propagates contexts
We just perform union of contexts of every pieces.

Change-Id: Ief925c1818cd8bbec0503e9c625b0630feebfdda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10432
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 18:15:36 +00:00
Ryan Lahfa
0e6285ec8b feat(tvix/eval): context-aware concatStringsSep
Change-Id: Id0e169084d26dc598091d157563c4d959b66279b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10431
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:15:04 +00:00
Ryan Lahfa
88fcab68d4 chore(tvix/eval): note on context-aware toString
Change-Id: Ie26ebd16e95e6a7b6f81051d8269169842978058
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10430
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 18:14:02 +00:00
Ryan Lahfa
c7d6d08799 feat(tvix/eval): context-aware throw
Change-Id: Ie552dabe4cf93cc396c883268a3bee67796dbbd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10429
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:09:30 +00:00
Ryan Lahfa
7dcb37fc52 feat(tvix/eval): context-aware substring
`substring` has a very funny behavior when it comes to empty strings,
it propagates the context too, this is used in nixpkgs to attach context
to strings without using any builtin: `lib.addContextFrom`.

Change-Id: Id655356799b3485f7519b3d1914c630f9d8416c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10448
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 18:08:58 +00:00
Ryan Lahfa
375f7eaa59 feat(tvix/eval): context-aware abort
Change-Id: Id5a435961ce3a2a2240b3936ea48515650d445d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10427
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 18:07:56 +00:00
Ryan Lahfa
e259a5c42b feat(tvix/eval): context-aware dirOf
Change-Id: If73a82a7106de9b479c950741efb70bffabd470a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10424
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-03 16:57:35 +00:00
Ryan Lahfa
db748741da feat(tvix/eval): context-aware baseNameOf
Change-Id: I9f0a8143070805b85276f721bdfbdf7ede2cf615
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10421
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-03 16:54:33 +00:00
Ryan Lahfa
e15b6580cc chore(tvix/eval): notes on coercion and contexts
We make a case for adding a `reject_context` `CoercionKind` here.
It does happen during concatenation actually for path concats.

Change-Id: I0c196aad917550b9bcd0896cd2127a94f8181ffb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10444
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 16:54:00 +00:00
Ryan Lahfa
743c362049 feat(tvix/eval): context-aware coerce_to_string
I am still undecided whether we need a CoercionKind to control
the coerced context, here's a simple attempt.

Change-Id: Ibe59d09ef26c519a6acfdfe392014446646dd6d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10426
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 16:45:26 +00:00
Florian Klink
802f374a90 feat(tvix/glue): handle passAsFile
This extends derivation_to_build_request to handle passAsFile the same
way Nix does, and adds a unit test for it.

I opted to making this function fallible (if passAsFile contains a
non-existent env var), rather than pushing all of this into the
Derivation validate function.

Change-Id: I75b635f1f6c0c78d72b9a8fc7824f77e97b69951
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10522
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-03 14:15:15 +00:00
Florian Klink
a82214b3ad fix(tvix/store): set service.name
While this can be influenced with `OTEL_SERVICE_NAME` or
`OTEL_RESOURCE_ATTRIBUTES`, "unknown_service" is an annoying name.

Let's pick tvix.store as a default.

Change-Id: I9b5c45703d9546775bb9266b29ab54daf7c54a9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10524
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 13:29:35 +00:00
Florian Klink
9abe735061 refactor(tvix/glue): create environment_vars Vec at the end
Prevent code in this function from accessing the Vec, realize it only
when returning the BuildRequest struct.

Change-Id: I385c1adca552c9cb261cd70c5ca3ec45f0d7a9c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10521
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 13:19:02 +00:00
Florian Klink
7f030fe265 test(tvix/glue): add test with passAsFile
This already succeeds, as the output path calculation path does not need
to be aware of any builder-specific custom handling for passAsFile.

Change-Id: I9cc9e4e4351cdeaa3ec33ba58ee1569e7a368150
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10520
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-03 13:19:02 +00:00
Florian Klink
e6782bb7c4 feat(tvix/build/protos): add additional_files field
Nix builds also have some other inputs, outside of inputs_dir, used in
derivations setting `passAsFile` and `__structuredAttrs`.

This extends our `BuildRequest` structure to accomodate these usecases.

Change-Id: I8de6516eb467fa48a4961d88933ebd548f131049
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10519
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2024-01-03 13:18:31 +00:00
Florian Klink
6b136dfd23 feat(tvix/glue): emit a warning in case of bad SRI hashes
And include a test to ensure we show the warning.

Change-Id: Ib6a436dbba2592b398b54e44f15a48d1aa345099
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10470
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-03 13:01:02 +00:00
Ryan Lahfa
d5aa75bbcf feat(tvix/eval): emit warnings with kinds and spans
In the past, we had `emit_warning` be no-op and we used `push_warnings` exclusively
but as we have consumers of this function, we need it to work somewhat.

Change-Id: I78a5ece199a473dec9ef5ea1fae60b36e35137b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10477
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-03 13:01:02 +00:00
Florian Klink
7a84a8fe89 fix(nix-compat/nix-hash): relax padding requirements
Nix is quite tolerant when it comes to parsing SRI hashes and their
padding (and only for SRI hashes, it funnily is strict about that in the
non-SRI-hash case).

Nix essentially accepts any number of padding characters, no matter if
it's too much or too little. So we do the only sane thing - simply strip
all padding characters, and parse it with BASE64_NOPAD and the length
the algo uses.

Change-Id: I6a721aa289b06cc36741589792b9dd4c4f930b86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10468
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 13:01:02 +00:00
Florian Klink
3307791855 feat(tvix/store): add opentelemetry
This is behind the otlp feature flag (currently enabled by default).

By default, this will try to push traces to a OTLP collector running at
https://localhost:4317.

You can make one available by running:

```
docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 --rm \
  jaegertracing/all-in-one:1.42
```

Started like that, jaeger brings a web interface at
http://localhost:16686/search

As documented in
https://docs.rs/opentelemetry-otlp/latest/opentelemetry_otlp/, you can
point this to another location by setting `OTEL_EXPORTER_OTLP_ENDPOINT`.

Change-Id: Id1dca367d70027b2ea98bb70bcf99a68363ec2be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8194
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: aaqaishtyaq <aaqaishtyaq@gmail.com>
2024-01-01 17:27:39 +00:00
Florian Klink
597a6b6205 refactor(tvix/castore/tests): let gen_*_service return Boxes
Only convert to and reuse an Arc<…> where needed.

Change-Id: I2c1bc69cca5a4a3ebd3bdb33d6e28e1f5fb86cb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10514
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 14:45:17 +00:00
Florian Klink
09a92b78d2 refactor(tvix/store/nar/import): be a bit more generic
Change-Id: If9a536949f36f428abea1a893f937fe7063e2f41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10517
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 14:43:07 +00:00
Florian Klink
1b62f82b10 refactor(tvix/castore/blobsvc/grpc/wrapper): don't require Arc<_>
Change-Id: I9655f5588c7dc98427de6af47d74b4ab7ce22071
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10516
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 14:42:36 +00:00
Florian Klink
96aa220dcf refactor(tvix/castore/directorysvc/grpc/wrapper): no Arc<_>
We can also drop the Clone requirement. Because the trait is async since
some time, there's no need to clone before moving into an async closure,
allowing us to simplify the code a bit.

Change-Id: I9b0a0e10077d8c548d218207b908bfd92c5b8de0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10515
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-01 14:40:35 +00:00
Florian Klink
54fe97e725 refactor(tvix/castore): make directorysvc more generic
This works on Box<dyn DirectoryService> too.

Change-Id: Ib869f0f4d963ef4dbaeab22db03ff6afb71ede04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10513
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-01 02:09:20 +00:00
Florian Klink
2c2fdfedc6 refactor(tvix/glue/tvix_store_io): use utils::import_path
This is doing the exact same thing.

Change-Id: Iadc5e13dd192efc91cc3d36b2bdf4b8b99a312b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10512
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 01:55:46 +00:00
Florian Klink
52a61e353b feat(tvix/store/bin): factor out import
While at it, make it a bit more generic.

Change-Id: Ic4caefda93aca3ffb656a09f8b4d648b41415532
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10511
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-01 01:55:46 +00:00
Florian Klink
ddae4860c2 feat(tvix/castore/import): generalize ingest_path
We don't actually care if it's an Arc<dyn BlobService>, or something
else, as long as we can Deref to a BlobService and clone.

Change-Id: I0852aaf723f51c5e6b820be8db1199d17309ab08
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10510
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-01 00:54:14 +00:00
Florian Klink
fdd7817aad feat(nix-compat/store_path): build_ca_path may fail
Change-Id: Ia74ee870f38b7966501458bace541092256c3213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10509
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-01 00:52:41 +00:00
Florian Klink
20c98a51bf refactor(tvix/store/bin): don't reconstruct StorePath twice
We already have that data in output_path.

Change-Id: Iadf615551f0b44d539098446ff5926994deb606d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10508
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 00:42:38 +00:00
Florian Klink
fa335aaa68 fix(tvix/store/bin): don't unwrap in case of invalid paths
Instead, return an error, and move the entire check before starting to
ingest the data underneath.

Change-Id: Idcfba115cb7d599f5fc72a156aaad9d4d4714fcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10507
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-31 22:23:49 +00:00
Florian Klink
694ed7ea1a refactor(tvix/store/bin): condense subscriber setup a bit
We can use cli.json.then_some(…) to create a Some(…), allowing us to
omit the else { None } lines.

Change-Id: I6c8142a08d8cb88d6c8302e5ca7570698fcf2aa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10505
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:23:49 +00:00
Florian Klink
f6d1a56c8c refactor(tvix/store): move construct_services helper here
This takes three URLs, and constructs Arc'ed
{Blob,Directory,PathInfo}Service, allowing to remove some of the
boilerplate.

Change-Id: I40e7c2b551442ef2acdc543dfc87ab97e7c742bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10484
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-31 22:23:49 +00:00
Florian Klink
41935fab70 refactor(tvix/castore/directorysvc): return Box, not Arc
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.

Change-Id: Ia05bb370340792a048e2036be30e285ef1e63870
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10483
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:18:14 +00:00
Florian Klink
9ca1353122 refactor(tvix/castore/blobsvc): return Box, not Arc
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.

Change-Id: I7bd337cd2e4c2d4154b385461eefa62c9b78345d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10482
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:18:14 +00:00
Florian Klink
e2b6c77bfc feat(tvix/cli): allow configuring different services
At some point, tvix-cli needs to talk to the outside world to persist
things into a real store.

Introduce the same CLI options to configure
{Blob,Directory,PathInfo}Service URLs.

We need to be a bit careful with how we set up stores, and make
this separate from setting up TvixStoreIO, as it's holding a
Rc<RefCell<KnonPath>> which not Send.

At some point, we might make this a Arc<RwLock<_>> later anyways, and
then this can be simplified a bit, but for now, this is sufficient.

Change-Id: I87d84ca3a10ce947e194ff985073791469773f35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10474
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 21:55:18 +00:00
Florian Klink
12f9b95a2c feat(tvix/eval): accept impl AsRef<str> for code
We're also happy to consume strings, or other owned stringy types.

Change-Id: I5bead4407976134815d8f879f9f70468e6af1dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10476
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 13:15:24 +00:00
Florian Klink
4fba57c2c9 refactor(tvix/eval): remove code and location from struct
Instead, it's passed in the evaluate/compile_only functions, which feels
more naturally. It lets us set up the Evaluation struct long before
we actually feed it with data to evaluate.

Now that Evaluation::new() would be accepting an empty list of
arguments, we can simply implement Default, making things a bit more
idiomatic.

Change-Id: I4369658634909a0c504fdffa18242a130daa0239
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10475
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-12-31 13:15:24 +00:00
Florian Klink
4d6137964b feat(tvix/build): don't force outputs to be in store_dir
There's no need to require builds to also put their outputs in store_dir.

So rename store_dir to inputs_dir and make outputs consume paths, not
basenames.

Also document the possibility for builds to write to inputs_dir. Let's
see if we want to make this a constraint later.

Change-Id: Ib4df20afcdde2d771c269c422f04c7e95587cd0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10447
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-12-29 22:14:08 +00:00
Ryan Lahfa
3e63d7be42 feat(tvix/eval): context-aware casting to strings
By default, we don't want contextful strings and we almost always want contextless strings.

To this end, we make taking a contextful string a very explicit operation under `to_contextful_str`
and we implement manually the `to_str` cast which requires a `if !s.has_context()` guard that
the macro cannot cover.

Change-Id: I7aae8e57a7d73e547e62b1edb0b1cc7e8c0c69b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10425
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-29 22:00:04 +00:00
Ryan Lahfa
343ee0de57 feat(tvix/eval): implement hasContext primop
`hasContext` is now functional.

Change-Id: I23b128afc9150b833bc0d9b042d31fee35badadb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10422
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-29 21:58:32 +00:00
Ryan Lahfa
5dfb15d2c8 feat(tvix/eval): introduce NixContext
This prepares the data structures to implement string contexts
in Nix.

Change-Id: Idd913c9c881daeb8d446907f4b940e462e730978
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10420
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-29 21:57:30 +00:00
Ryan Lahfa
fc182061c6 docs(tvix/eval): why context strings now
Explain why we had to take the bullet on context strings now
and move the historical approach in a historical section.

Change-Id: Ie3bcc2213b391c6ba06547cc05c850891a41d06b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10446
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-29 21:55:58 +00:00
Adam Joseph
4b3f27ebc9 fix(tvix/eval): propagate catchables through builtins.attrNames
Change-Id: Id14e39543239272aed041998fd9a78465c9cb8b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10359
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
9cb3cd10f5 fix(tvix/eval): propagate catchables through builtins.intersectAttrs
Change-Id: I4ada8cf10611e98519cb1b1da11bfd06815f2932
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10358
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
2883b2d4bd fix(tvix/eval): propagate catchables in string interpolations
Change-Id: I13d7ce0c7328a8e6fbc6d2c4ff5c4fe6095b96ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10357
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
7ddea7340f fix(tvix/eval): catchable in type field of nix_eq()
Change-Id: I165ff77764e272cc94d18cb03ad6cbc9a8ebefde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10348
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
2af8174e2e fix(tvix/eval): builtins.match: propagate catchables
Change-Id: I14c9e625c91369e10d0c00380dca992811ae9059
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10346
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
11e35a77a6 refactor(tvix/eval): let OpCoerceToString select the CoercionKind
Change-Id: I92d58ef216d7e0766af70f019b3dcd445284a95d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10344
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
0c22454bb9 fix(tvix/eval): add stack depth assertion to OpReturn
I'm still trying to work out the exact stack invariants for tvix.
We really should add assertions for them; getting the stack messed
up is no fun.  This commit adds one simple assertion.  It also adds
a missing stack-push (my mistake) in one place, which was uncovered
by the assertion.

Change-Id: I9d8b4bd1702d954e325832c5935b0d7e3eb68422
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10369
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Florian Klink
8a52c7f1c5 feat(tvix/castore/fs): borrow some matches
We only do things with the reference, so we don't need to locally borrow
it.

Change-Id: I6073f7ec7aff717ae3069e28a00b1cb408a50ceb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10455
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-29 17:18:14 +00:00
Florian Klink
5e3683cd53 feat(tvix/shell): add grpcurl to shell
This is a bit more useful than evans when sending over the same message
over and over again.

Change-Id: I600b6b9f591c0c963c5d270005aa1cc84d2a0770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10453
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-12-29 15:57:26 +00:00
Florian Klink
22427dc8d0 refactor(tvix/castore/bin/tvix-store) don't import std::io
Make it more clear this is not tokio::io.

Change-Id: Ic2fa56f0baf1c200b6631098d556388a19629a45
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10452
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-29 15:57:26 +00:00
Florian Klink
46a372d5d7 feat(tvix/castore/fs): instrument FuseDaemon functions
Change-Id: I696b7ab6b4c08004db147c0fda7312bbebaa0eec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10451
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-29 15:57:26 +00:00
Florian Klink
0af445c2f4 feat(tvix/store): emit more structured log messages
Change-Id: If4748cb800276fc42779ba8973d1eef72df0503e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10450
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-29 14:37:58 +00:00
Florian Klink
f6c94430c8 feat(tvix/build/protos): add some missing fields
- directory in which the castore input nodes are mounted
 - working directory for the build command
 - scratch paths
 - network access y/n
 - whether a (static) /bin/sh should be provided

Populate these fields appropriately, and extend the tests in tvix-glue
with a FOD example.

Change-Id: I4f9de1483d6696d74694a09784910c407acb0be0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10412
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-26 10:20:41 +00:00
sterni
d07600dbca fix(tvix/eval/value): correctly emit spaces when coercing lists
r/7176 introduced an incorrect assumption was the benefit of the
nonrecursive coercion algorithm, namely that a coercion operation always
returns a non empty string. This allows to detect whether we are
coercing a list or not by checking if the intermediate result is empty
or not. Unfortunately, coercing null and false yields an empty string,
so we need to explicitly track whether we are coercing a list.

Updated the test case to hopefully catch similar bugs in the future. I'm
not a hundred percent certain I have not introduced a new edge case with
this, so it may be interesting to add a prop test case for this to
nix_oracle down the line. At least lists are the only nested data
structures that can be serialized as nested data structures, so the
problem is kind of limited.

Change-Id: Ia41e904356f1c41a9d35e4e65ec02f2fe5a4100e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10418
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-25 15:06:23 +00:00
Florian Klink
d4dc28b675 chore(tvix/turbofetch): switch to futures 0.3.30
The bugs have been fixed,
https://github.com/rust-lang/futures-rs/pull/2801 and
https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended
up in that release.

Change-Id: I301c0ffc951f04a5b3b7267e922771c837a3f5a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10415
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-24 21:45:04 +00:00
Florian Klink
acbb613e61 chore(tvix): switch to upstream futures 0.3.30
The bugs have been fixed,
https://github.com/rust-lang/futures-rs/pull/2801 and
https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended
up in that release.

Change-Id: Iefd990d2d1719b884504093343e54e9c5258e2e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10414
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-24 21:45:04 +00:00
Florian Klink
a5865ec7fa refactor(tvix/castore/fs/tests): drop unused args
There's no need to pass in an unused directory service into the
populate_blob_* method, and considering we have one or two invocation of
each of these, we don't really gain much from having all these functions
follow the same structure, at least for now.

Also, update some function names to better describe what they're doing.

Change-Id: I92f680745c157fb0a602b07342f8838bfad23ecd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10411
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-24 16:05:52 +00:00
Florian Klink
8d86d2f409 refactor(tvix/castore): add RootNode impl for BTreeMap, mv fs tests
cl/10378 did already move store/fs to castore/fs, but we kept the tests
in tvix-store, as they were populating a PathInfoService to make nodes
appear in the mount root.

Update these tests to now just insert root nodes into a BTreeMap, and
ensure we can use that as a RootNodes too.

Change-Id: Iad7d1ee4f9423eb6e3a1da33f433842c9ae0de1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10410
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-24 15:44:30 +00:00
Aspen Smith
6af17545bf feat(tvix/website): Link to windtunnel results page
Change-Id: If4456c565abf32385b1239467b96e7398190ea4b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10383
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-23 17:12:25 +00:00
Florian Klink
a5167c508c chore(tvix): move store/fs to castore/fs
With the recent introduction of the RootNodes trait, there's nothing in
the fs module pulling in tvix-store dependencies, so it can live in
tvix-castore.

This allows other crates to make use of TvixStoreFS, without having to
pull in tvix-store.

For example, a tvix-build using a fuse mountpoint at /nix/store doesn't
need a PathInfoService to hold the root nodes that should be present,
but just a list.

tvix-store now has a pathinfoservice/fs module, which contains the
necessary glue logic to implement the RootNodes trait for a
PathInfoService.

To satisfy Rust orphan rules for trait implementations, we had to add a
small wrapper struct. It's mostly hidden away by the make_fs helper
function returning a TvixStoreFs.

It can't be entirely private, as its still leaking into the concrete
type of TvixStoreFS.

tvix-store still has `fuse` and `virtiofs` features, but they now simply
enable these features in the `tvix-castore` crate they depend on.

The tests for the fuse functionality stay in tvix-store for now, as
they populate the root nodes through a PathInfoService.

Once above mentioned "list of root nodes" implementation exists, we
might want to shuffle this around one more time.

Fixes b/341.

Change-Id: I989f664827a5a361b23b34368d242d10c157c756
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-22 16:55:18 +00:00
Florian Klink
52cad86195 refactor(tvix/store): remove Arc<> from PathInfoService::from_addr
This makes PathInfoService::from_addr return a Box<dyn PathInfoService>,
rather than an Arc<dyn …>, and leaves it up to the consumers to rewrap
it into an Arc where needed.

This allows us to drop the Arc for the tvix-store daemon subcommand.

Change-Id: Ic83aa2ade6c51912281bd17c7eef7252e152b2d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10409
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-22 16:55:18 +00:00
Florian Klink
93a228b9a4 refactor(tvix/store/bin): move service.clone right before async move
Change-Id: I7a2b951d9c9251a053a0de40f31836bda03a922d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10408
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-22 16:55:11 +00:00
Florian Klink
39cddb95be refactor(tvix/store/bin): remove unneeded clone
Change-Id: Ib94376db3a57853d980112919c2d6fcc0f566883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10407
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-12-22 13:29:04 +00:00
Florian Klink
9627ef15de docs(tvix/castore/protos): remove reference
This is not gonna end up as a interlinked docstring.

Change-Id: I2b0ca106aa75bae0156c0b411da5931da60c725d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10406
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-12-21 16:44:48 +00:00
Florian Klink
3b274f47af docs(nix-compat/nar/reader): fix reference
Change-Id: I7ccdfe3c042f49aa962d1c7ba6337bd656ef05c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10405
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:44:16 +00:00
Florian Klink
12406fe241 docs(nix-compat/derivation): fix docstring
Change-Id: I76200aa474b86cf655308f6710a2298b77fa4b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10404
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:44:16 +00:00
Florian Klink
b04fdeaa10 docs(nix-compat/derivation): fix reference
Change-Id: I0ebe256c62435901c12ca1895f30e380280e9ea8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10403
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:44:16 +00:00
Florian Klink
410abd67f8 docs(nix-compat/aterm/parser): fix docstring
Change-Id: I7a65ce320163c5898df402af3a639f6cee2e17c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10402
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:43:44 +00:00
Florian Klink
783a1e314c docs(tvix/castore): fix reference
Change-Id: I00b1d56d58c4d3779b57ab0056cff1c7e6053b9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10401
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-12-21 16:43:44 +00:00
Florian Klink
3fc4d295d5 docs(tvix/store): fix reference
Change-Id: If1970cb8eb9a21c30011c9303f2fcbff646ad12b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10400
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:43:44 +00:00
Florian Klink
5bee6c6be0 docs(tvix/store): fix reference
Change-Id: Ifb77097755f25aa6536c60365941507a592cac3e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10399
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:43:11 +00:00
Florian Klink
6838dc3fff docs(tvix/glue): fix docstring
This is not an unclosed <html> tag.

Change-Id: I2bd2426fc600de2d96dbab47743f1c7bd5fed35e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10398
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:43:11 +00:00
Florian Klink
ca0a8f5603 docs(nix-compat/nixhash): fix reference
Change-Id: I95c35a1405601c6f79cbc03462ab7c7cd6f08f2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10397
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-12-21 16:43:11 +00:00
Florian Klink
1c3ea9d83d docs(nix-compat/drvfmt): fix reference
Change-Id: Ic0403e028cc968bef4023ef78dbfa0d502da90b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10396
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:39:07 +00:00
Florian Klink
8c83838c40 docs(nix-compat/narinfo): fix reference
Change-Id: Ib3a5414afed738ba6178ec69d38a4169c9b24714
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10395
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:36:34 +00:00
Florian Klink
18702c740e docs(nix-compat/narinfo/fingerprint): fix reference
Change-Id: Icc56aa178247b3985a406421b7d8d449c63f8231
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10394
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-12-21 16:36:34 +00:00
Florian Klink
e4eb0b9bf4 docs(nix-compat/derivation): fix references
Change-Id: If6962931350edfbae206fa739a673e7ccbc3ee89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10393
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:36:34 +00:00
Florian Klink
329a7d30a7 refactor(nix-compat/store_path): centralize self_reference check
self_reference being set to true is only allowed for
`CAHash::Nar(NixHash::Sha256(_))`, so we can handle this in a check at
the front.

Change-Id: Ic363ade4789a7767cbe26a6959b143bb53e50e5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10391
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:36:29 +00:00
Florian Klink
88adaea12b refactor(nix-compat/store_path/utils): restructure build_ca_path
All match cases essentially construct `ty` and `hash`, which is then
passed to the `build_store_path_from_fingerprint_parts` function.

Change-Id: I01dfd219f9b0ac1afe8af7c6e361ea048117a0e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10390
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:28:53 +00:00
Florian Klink
9065089b0c refactor(tvix/nix-compat): have helpers interact with StorePathRef
In most case, we don't actually need an owned `StorePath` struct, but a
`StorePathRef<'_>` is sufficient. The lifetime is only due to it holding
onto the name, but that one is mostly left untouched.

`Derivation::calculate_derivation_path` still needs to return
`StorePath`, as its name has a `.drv` appended.

Change-Id: Ie0d52f369d785711bb0658ea2b0bd2617fd9f45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10389
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:28:53 +00:00
Florian Klink
ea8fdb9304 refactor(nix-compat/narinfo/fingerprint): move to StorePathRef
With the introduction of StorePathRef::to_absolute_path(), there's no
need to convert this StorePathRef to a StorePath first.

Change-Id: I634c977c4b63858e4f329fd21726e0611b99da4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10388
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-12-21 16:28:53 +00:00
Florian Klink
91deb67de9 refactor(nix-compat/store_path): use StorePathRef::to_absolute_path
Keep the method around in StorePath for convenience, but move the
implementation to StorePathRef.

Change-Id: Ie1844fa01ce6529dc1a58907563c95c3112c831d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10387
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:28:41 +00:00
Florian Klink
c7ea57df46 feat(nix-compat/store_path): derive [Partial]Eq
This allows comparing StorePathRef structs.

Change-Id: Ia69967ea9358052e2d6e76042a7e6d394f7f29a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10386
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:19:46 +00:00
Luke Granger-Brown
e20848ecf1 chore(depot): update OWNERS files for aspen
Change-Id: Id94b646a6ea035782298c421d6667530da6fc5b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10384
Tested-by: BuildkiteCI
Owners-Override: lukegb <lukegb@tvl.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2023-12-20 18:35:58 +00:00
Aspen Smith
c236231910 chore(tvix): Run criterion benchmarks for the whole workspace
Run criterion benchmarks (via `cargo bench`) for the *whole tvix
workspace*, not just `//tvix/eval`, and report the results to
windtunnel.

Change-Id: I9235c3c166ed9121f35c5bb4c46d59dc1f4c4055
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10382
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-20 18:04:43 +00:00
Florian Klink
9c1358fe44 refactor(tvix/store): remove more clones on pathinfoservice
These are still leftovers from before the traits being async, where we
had to clone before moving into an async closure.

Change-Id: I1b3937edf61ce3e23bb07803306622c37a3572c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10381
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-12-18 14:10:40 +00:00
Ryan Lahfa
0ae32d45f6 feat(tvix/castore): simple filesystem blob service
The simple filesystem `BlobService` enable a user to write blob store
on an existing filesystem using a prefix-style layout in the provided root directory,
e.g. the two first bytes of the blake3 hashes are used as directories prefixes.

Change-Id: I3451a688a6f39027b9c6517d853b95a87adb3a52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10071
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-17 14:34:13 +00:00
Florian Klink
923a5737e6 refactor(tvix/castore): drop is_closed() from impl DirectoryPutter
This is only used in the gRPC version (GRPCPutter), during the test
automation.

So define it as a method there, behind #[cfg(test)], and remove from
the trait.

Change-Id: Idf170884e3a10be0e96c75d946d9c431171e5e88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10340
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 23:07:31 +00:00
Florian Klink
36cc7b7088 docs(tvix/store/fs): fix comment
There's nothing store-path specific here anymore, it's just a name in
the mountpoint root.

Change-Id: I0f8004491baa03ba560d390053a42678ee81154a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10377
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 21:53:31 +00:00
Florian Klink
7749814953 docs(tvix): refer to futures-rs PR
The PR has since been replaced with another version, using a bit of
unsafe, as rustc doesn't like the lifetimes.
We're still waiting on a backport of that to 0.3.

Change-Id: I8dd344e78162cd956f6e53ff05dc4c02763fbb04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10376
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-16 21:47:01 +00:00
Florian Klink
43ab1543e2 chore(tvix/store): switch back to fuse-backend-rs upstream
The fix mentioned in Cargo.toml did land in
https://github.com/cloud-hypervisor/fuse-backend-rs/pull/162.

Change-Id: I4451f910806f05400f7e0f83581ae786f170b9e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10375
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 21:46:30 +00:00
Florian Klink
97dee277b0 refactor(tvix/store/fs): no explicitly required Arc'ed Blob/DirSvc
Change-Id: Ie55026668cd4a6117e7b07174f5ac6638f93d194
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10374
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 21:46:30 +00:00
Connor Brewster
6815572274 refactor(tvix/store/fs): Decouple TvixStoreFs from Nix concepts
To support tvix builds, we need to be able to use the `TvixStoreFs` to
materialize the sandbox's /nix/store filesystem with just the list of
inputs needed for the build. Currently we'd need to seed an in-memory
`PathInfoService`, which includes more functionality than what is
required for `TvixStoreFs`. Additionally, the `PathInfoService` is
specific to Nix. By decoupling `TvixStoreFs` and `PathInfoService`,
we allow for usage of `TvixStoreFs` with `tvix-castore` without needing
a `PathInfoService`.

This introduces a new `RootNodes` trait which provides a way for the
filesystem to look up CA nodes via their basename in the root directory
of the filesystem. We then implement `RootNodes` for any
`PathInfoService`. Additionally, the filesystem root inode tracker now
stores basenames rather than `StorePath`s since `StorePath`s are
specific to Nix.

As a followup we can rename `TvixStoreFs` to `TvixCaStoreFs` and move
it to the `castore` crate (or its own crate).

b/341

Change-Id: I928372955017c23b1bf2b37190cbc508a4ed10d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10363
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-14 21:46:25 +00:00
Aspen Smith
91b6c13f90 fix(tvix): Remove errant backslash in benchmark script
Change-Id: I485986521fd4a25770f2a398896a0f672628df9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10365
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-14 19:52:31 +00:00
Aspen Smith
1d678bab5f fix(tvix): Fix tvix nixpkgs eval benchmark
Fix the nixpkgs eval hyperfine benchmark in tvix/scripts.bench.sh to:

- Eval both a pinned (never-changing) nixpkgs commit and the nixpkgs
  commit used by the depot
- Not depend on fetchTarball, which is not yet implemented.
- Be called bench-windtunnel.sh

Change-Id: Id2ae18f983ab7327625320b5b16c082ecc369a49
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10364
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2023-12-14 19:34:58 +00:00
Florian Klink
e4d6d20aa7 test(tvix/glue): add nonexisting_path_without_import test
This moves the failing example from cl/10285 into its separate test
case.

There were multiple complications: tvix-[ca]store was panicking in some
places, rather than returning an error. This is now fixed.

It needs to live in tvix-glue, so we actually have a "proper" EvalIO
interface doing something.

> toString ({ line = 42; col = 42; file = /deep/thought; }.file)

Should not cause an error, because it shouldn't trigger an import, but
leave the path as-is, and not care about it not being present.

Change-Id: I76f70b3cb1f73a0fb05870375710fd9f67d5603c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10342
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-14 13:25:25 +00:00
sterni
7165ebc43b fix(tvix/eval): remove incorrect imports when coercing
The default behavior of string coercion in C++ Nix is to weakly coerce
and import to store if necessary. There is a flag to make it strongly
coerce (coerceMore) and a flag that controls whether path values have
the corresponding file/directory imported into the store before
returning the (store) path as a string (copyToStore). We need to
implement our equivalent to the copyToStore (import_paths) flag for the
benefit of weak coercions that don't import into the store (dirOf,
baseNameOf, readFile, ...) and strong coercions that don't import into
the store (toString).

This makes coerce_to_string as well as CoercionKind weirder and more
versatile, but prevents us from reimplementing parts of the coercion
logic constantly as can be seen in the case of baseNameOf.

Note that it is not possible to test this properly in //tvix/eval tests
due to the lack of an appropriate EvalIO implementation being available.
Tests should be added to //tvix/glue down the line.

Change-Id: I8fb8ab99c7fe08e311d2ba1c36960746bf22f566
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10361
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-14 13:15:23 +00:00
sterni
a30dd0905a fix(tvix/eval): determine meaning of + exprs based on first type
This matches the behavior of C++ Nix more closely where the decision is
made based on the first type based to ExprConcatStrings:
1f93fa2ed2/src/libexpr/eval.cc (L1967-L2025)

Note that this doesn't make a difference in any successful
evaluation (at least to my knowledge), but ensures that our error
messages will match C++ Nix more closely, e.g. in the case of
`1 + "string"`.

Change-Id: I8059930788f9c8d98baf98e3d93d8a060ef961f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10360
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-14 02:59:58 +00:00
Florian Klink
8b0047e277 docs(tvix/castore/directorysvc): update comment
This comment didn't make a lot of sense before.

Change-Id: Ie057a133ca4b1a099ed3c885e32316b0d87c5eb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10339
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-12-13 21:11:58 +00:00
Florian Klink
3a32963b78 docs(tvix/castore): document expectations about DirectoryService
Namely, all trait implementations should reject invalid data being fed,
and detect invalid data being returned.

b/355 tracks writing some more tests for this, to ensure we're compliant
with this.

Change-Id: I3b05752932837ce208785efb21ffc21508b4b33a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10338
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2023-12-13 19:57:45 +00:00
Florian Klink
d236b08916 docs(tvix/castore): fix docstrings
There's been some copypasta errors.

Change-Id: I8fcad6cfc951ead6c789e0dce823c798adbfcf97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10337
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-13 19:40:40 +00:00
Florian Klink
9c5c717a99 feat(tvix/glue): add single-file import tests
This ensures importing these paths also behave the same way as Nix.

Change-Id: Icaa507bbe3d9867a301fc7a300c5d2b3f9feb911
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10355
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-13 13:42:21 +00:00
Florian Klink
3cdde8ad5a feat(tvix/glue): add some import tests
This creates a directory with a .keep file inside, and uses
"${path/to/there}" to coerce it to a string (and import it into the
store), ensuring it calculates the same store paths as Nix does.

Change-Id: Ie14ae075104ce278bc4f2cce93aab5762a2734d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10343
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-13 13:41:44 +00:00
Florian Klink
307675f5e7 fix(tvix/glue/tvix_store_io): don't unwrap ingest_path errors
Fixes b/344.

Change-Id: I1446726e3be3a8fc20801d466a964c4d6b8cbc02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10331
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:43 +00:00
Florian Klink
81ef26ba3f fix(tvix/castore/import): don't unwrap entry
If the path specified doesn't exist, construct a proper error instead
of panicking.

Part of b/344.

Change-Id: Id5c6a91248b0a387f3e8f138f8e686e402009e8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10330
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:11 +00:00
Florian Klink
afd09c3290 feat(tvix/castore/import): log returned errors
This will emit a log event / trace in case this function returns an
error-y type.

Change-Id: I48db6807f3e42304357c422a2b6e177cb8b95228
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10329
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:11 +00:00
Florian Klink
30d82efa77 refactor(tvix/castore/blobservice): use io::Result in trait
For all these calls, the caller has enough context about what it did, so
it should be fine to use io::Result here.

We pretty much only constructed crate::Error::StorageError before
anyways, so this conveys *more* information.

Change-Id: I5cabb3769c9c2314bab926d34dda748fda9d3ccc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10328
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-12 18:06:40 +00:00
Adam Joseph
91456c3520 refactor(tvix/eval): vm::add_values(): be less verbose
Change-Id: Icf328649fd70bdf0fc3ba6cd7754ae29735f11f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10035
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 18:00:37 +00:00
Adam Joseph
243a4b5699 fix(tvix/eval): substring: propagate catchables
Change-Id: Ia9b7858c817fbc9c95a3d1c2855b2445f7830e8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10326
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:51:29 +00:00
Adam Joseph
c5f58d4af3 fix(tvix/eval): baseNameOf: propagate catchables
Change-Id: Id8dc772ea8f338dfd243210f4108f79072570c3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10324
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:50:56 +00:00
Adam Joseph
2565a21aa9 fix(tvix/eval): builtins.length: propagate catchables
This commit fixes out builtins.length so it propagates catchables
like cppnix does.

Change-Id: I7670bec5eee1d4cd3f67a04c9a6808979fb56a8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10315
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-12 17:49:54 +00:00
Adam Joseph
1ddfed9825 fix(tvix/eval): builtins.filter: propagate catchables
This commit fixes builtins.filter so it propagates catchables
correctly.

Change-Id: Ib23a383bc5e272e42052205ffd1e94649a0ebc47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10313
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:35:39 +00:00
Adam Joseph
25dd1458b9 feat(tvix/eval): builtins.hashString: add placeholder
This adds an unimplemented placeholder for builtins.hashString.

Change-Id: Ibc770103acf5dbc3ea7589ab5ca23fe6e07bd91a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10311
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 17:35:06 +00:00
Adam Joseph
8c8409c0d2 fix(tvix/eval): builtins.getAttr: propagate catchables
Change-Id: I84b6b8f8568d57614a03aff0d6069e0bc27357bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10310
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 17:34:34 +00:00
Adam Joseph
afba150036 fix(tvix/eval): builtins.elemAt: propagate catchables
This commit fixes builtins.elemAt so it propagates catchables like
cppnix does.

Change-Id: Ieca5e128da17e78af0b14dae4a28a1ff8796e4f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10308
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:34:02 +00:00
Adam Joseph
07ca556a94 feat(tvix/glue): make unimplemented-structuredAttrs catchable
This commit adjusts the error produced by STRUCTURED_ATTRS so that
it is catchable.  This way we are able to enumerate the release
packageset, and the enumeration process will simply treat the few
derivations using structured attributes as being broken, rather than
killing the whole eval session.

Change-Id: I2e17638b8e3227f88543c3718aaf505deaec22ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10306
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 17:21:53 +00:00
Adam Joseph
29464fc65b fix(tvix/eval): propagate catchables through builtins.splitVersion
This fixes our implementation of builtins.splitVersion so it
propagates catchables like cppnix does.

Change-Id: Id5d83ea76229f8c8f202aa42353cb609e67de43f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10305
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:21:53 +00:00
Adam Joseph
012b44244b feat(tvix/eval): builtins.hasContext: placeholder implementation
Currently this just `throw`s a message explaining that it is not
implemented.  This is necessary in order to allow enumerating the
nixpkgs release attrset (afaict only one package uses this builtin).

Change-Id: I45266d46af579ddb5856b192b6be4b481369543c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10302
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:21:53 +00:00
Adam Joseph
531f0c0c42 fix(tvix/eval): baseNameOf should not coerce paths into strings
... since this may import them to the store which changes their
basename.

Fixes b/350.

Change-Id: Iabd08ff4d6a424c66d6d7784d7a96b0c078f0a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10298
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 17:21:21 +00:00
Adam Joseph
8d4aa2c15c fix(tvix/eval): add unimplemented __curPos and builtins.filterSource
This commit adds __curPos (to the global scope, yuck) and
builtins.filterSource.  These are not implemented; forcing them will
produce the same result as `throw "message"`.

Unfortunately these two post-2.3 features are used throughout
nixpkgs.  Since an unresolved indentifier is a catchable error, this
breaks the entire release eval.  With this commit, it simply causes
those broken packages that use these features to appear as they are:
broken.

Change-Id: Ib43dea571f6a9fab4d54869349f80ee4ec5424c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10297
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 17:20:50 +00:00
Adam Joseph
bb1e79e5d1 fix(tvix/eval): propagate catchables through &&
Change-Id: I7bb5ac1ef47b41c47269e64cee0e90eb64c6fcce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10322
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:04:07 +00:00
Adam Joseph
bcc1ea8552 fix(tvix/eval): make || propagate catchables
Change-Id: I42f994d7c9228368d5f6c30c4730c24666f7bc69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10320
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:04:07 +00:00
Adam Joseph
7b9eea71d0 fix(tvix/eval): fix nested assertions b/340
This commit fixes our handling of `throw` within an `assert`
condition.

Fixes: b/340
Change-Id: I40a383639ec266da50a853f16216b1b7868495da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10318
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:03:03 +00:00
Adam Joseph
fc963033ae fix(tvix/eval): ?: propagate catchables
This commit fixes out `?` operator so it correctly propagates
catchables.

Change-Id: Iebaa153a8492101ee3ddd29893c98730ff331547
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10317
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 16:57:55 +00:00
Adam Joseph
990c4e727f feat(tvix/eval): OpAttrsSelect should propagate catchables
Previously, using a catchable as either argument of OpAttrsSelect
would result in an unrecoverable error.  This commit matches cppnix
behavior by propagating the catchable.

Change-Id: I4877f4068ec2b823225f185290693c101d0b9c9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10303
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 16:09:47 +00:00
Adam Joseph
2949ee08f1 fix(tvix/eval): calling a catchable is catchable
When attempting to call a Value, if it is a Value::Catchable we must
not cause an uncatchable failure.  This commit simply reuses the
Value::Catchable as the result of attempting to call it.  This is
safe because nix is designed so that nix code cannot distinguish
between different catchable failures -- they all look the same to
the interpreted code.

This fixes b/351.

Change-Id: Ibf763a08753e541843626182ff59fdbf15ea2959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10300
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 16:09:15 +00:00
Adam Joseph
52b68c0539 fix(tvix/eval): fix catchables in named formals
Fixes b/348.

Change-Id: I5e8d56b5fd26a19eac32ec5e11baf93765691dc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10296
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 15:55:06 +00:00
Adam Joseph
e516046ed2 test(tvix/eval): test catchables in named formals
Relates to b/348.

$ /git/depot/result/bin/tvix -E '(builtins.tryEval (({ fred }: "bob") (throw "3"))).success'
note: while evaluating this Nix code
 --> [code]:1:1
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (force)
 --> [code]:1:1
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (tryEval)
 --> [code]:1:2
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  |  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
 --> [code]:1:20
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E006]: expected value of type 'set', but found a 'internal[catchable]'
 --> [code]:1:21
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  |                     ^^^^^^^^

Change-Id: I730fdd996f7e1b81dbbf83dc1524104a8cad2f78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10295
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 15:54:04 +00:00
Adam Joseph
663362f3df fix(tvix/eval): fix testing catchables for inequality
Fixes b/347.

Change-Id: Icad0251884d4d8adcdf8d690b91385bf4896f9c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10294
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 15:53:32 +00:00
Adam Joseph
1b2a1892cb test(tvix/eval): testing catchable for inequality
Relates to b/347.

    $ /git/depot/result/bin/tvix -E '(builtins.tryEval (throw "bob" != 3)).success'
    note: while evaluating this Nix code
     --> [code]:1:1
      |
    1 | (builtins.tryEval (throw "bob" != 3)).success
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    note: while evaluating this as native code (force)
     --> [code]:1:1
      |
    1 | (builtins.tryEval (throw "bob" != 3)).success
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    note: while evaluating this as native code (tryEval)
     --> [code]:1:2
      |
    1 | (builtins.tryEval (throw "bob" != 3)).success
      |  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    error[E006]: expected value of type 'bool', but found a 'internal[catchable]'
     --> [code]:1:20
      |
    1 | (builtins.tryEval (throw "bob" != 3)).success
      |                    ^^^^^^^^^^^^^^^^

Change-Id: Ide19b3ddaf314ef310efc2fe5ac36667e43011dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10293
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-12 15:49:28 +00:00
Adam Joseph
289663cac4 fix(tvix/eval): handle catchables in attribute set updates
Fixes b/346.

Change-Id: I277121d2363e605ebe09651ed9440fe1bc126c8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10292
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 15:48:25 +00:00
Adam Joseph
e54533518b test(tvix/eval): test for catchable in attribute merges
Relates to b/346.

    $ /git/depot/result/bin/tvix -E '(builtins.tryEval (throw "bob" // { })).success'
    note: while evaluating this Nix code
     --> [code]:1:1
      |
    1 | (builtins.tryEval (throw "bob" // { })).success
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    note: while evaluating this as native code (force)
     --> [code]:1:1
      |
    1 | (builtins.tryEval (throw "bob" // { })).success
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    note: while evaluating this as native code (tryEval)
     --> [code]:1:2
      |
    1 | (builtins.tryEval (throw "bob" // { })).success
      |  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    error[E006]: expected value of type 'set', but found a 'internal[catchable]'
     --> [code]:1:20
      |
    1 | (builtins.tryEval (throw "bob" // { })).success
      |                    ^^^^^^^^^^^^^^^^^^

Change-Id: Ib84c4ec6d2612d4f1f6066e66c3dd1bf04369b6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10291
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 15:48:25 +00:00
Adam Joseph
edf411a86c fix(tvix/eval): fix recovering from throws in implications
This fixes b/345.

Change-Id: Ic0d3b6ffacd2a5e0050d22354d08320b69a4fe13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10290
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 15:47:53 +00:00
Adam Joseph
24ff74d346 test(tvix/eval): test recovering from throw in implications
error[E006]: expected value of type 'bool', but found a 'internal[catchable]'
 --> src/tests/tvix_tests/notyetpassing/eval-okay-test-catchables-in-implications.nix:1:43
  |
1 | (builtins.tryEval (({ foo ? throw "up" }: foo -> true) { })).success
  |                                           ^^^^^^^^^^^

Relates to b/345

Change-Id: Ic331c32ea59bf67ae775f485b444dc6804ca13d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10289
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 15:47:21 +00:00
Adam Joseph
9792920f8c fix(tvix/eval): fix branching on catchable defaults (b/343)
This commit adds Opcode::OpJumpIfCatchable, which can be inserted
ahead of most VM operations which expect a boolean on the stack, in
order to handle catchables in branching position properly.

Other than remembering to patch the jump, no other changes should be
required.

This commit also fixes b/343 by emitting this new opcode when
compiling if-then-else.  There are probably other places where we
need to do the same thing.

Change-Id: I48de3010014c0bbeba15d34fc0d4800e0bb5a1ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10288
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:55:48 +00:00
Adam Joseph
e54eeda0ff test(tvix/eval): test branching on catchable defaults (b/343)
This is a test case for b/343, wherein tvix dies if you try to
branch on an argument whose defaulted value is a catchable.

Change-Id: I891ca825e39ad14dda9f220f06d9591874fcd45d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10287
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-12 14:55:48 +00:00
Adam Joseph
1ac57b0d1c feat(tvix/eval): nonrecursive coerce_to_string()
After this commit, the only non-builtins uses of generators are:

  - coerce_to_string() uses generators::request_enter_lambda()
  - Thunk::force() uses generators::request_enter_lambda()

That's it!  Once those two are taken care of, GenCo can become an
implementation detail of `builtins::BuiltinGen`.  No more crazy
nonlocal flow control within the interpreter: if you've got a GenCo
floating around in your code it's because you're writing a builtin,
which isn't part of the core interpreter.  The interpreter won't
need GenCos to talk to itself anymore.

Technically generators::request_path_import() is also used by
coerce_to_string(), but that's just because the io_handle happens to
be part of the VM.  There's no recursion-depth issue there, so the
call doesn't need to go through the generator mechanism
(request_path_import() doesn't call back to the interpreter!)

Change-Id: I83ce5774d49b88fdafdd61160975b4937a435bb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10256
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:54:46 +00:00
Adam Joseph
3701379745 feat(tvix/eval): nonrecursive deep_force()
This commit implements deep_force() nonrecursively, by maintaining
an explicit stack rather than using the call stack for recursion.

As an added bonus, we don't need to pass around the SharedThunkSet
anymore, and can in fact completely eliminate SharedThunkSet.

Change-Id: I7c4f59f37834d451a28bf6be317eb0a90eac4ee6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10252
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:36:02 +00:00
Adam Joseph
c956138ee6 docs(tvix/eval): clarify difference between ThunkSet and Blackhole
The comment explaining ThunkSet makes it seem like it does the same
think as ThunkRepr::Blackhole.  In fact neither one is a substitute
for the other.  Let's explain the difference.

Change-Id: I89ceaaa9d3c499edbc7d48f70ca5d11f97666c43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10250
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:35:00 +00:00
Adam Joseph
ddba4d4e17 test(tvix/eval): nested assertions (b/340)
Change-Id: I898d7056877a6370d5720b633df416f54e7cf65f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10222
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 14:35:00 +00:00
Adam Joseph
c92d06271d feat(tvix/eval): drop LightSpan::Delayed
LightSpan::Delayed was introduced in commit
bf286a54bc which claimed that "This
reduces the eval time for `builtins.length (builtins.attrNames
(import <nixpkgs> {}))` by *one third*!"

I am unable to reproduce this result.  In fact, dropping the
LightSpan::Delayed variant of the enum makes eval of the same
expression slightly faster!  I also tried a large evaluation
(pkgsCross...hello) and got similar results: slightly faster,
slightly less memory.  See git footers.

I suspect that there was some unrelated horrific inefficiency that
has since been fixed.  The avoided computation in `get_span()` is
nothing more than a binary search!  If this were in fact a major
performance issue we could simply precompute the mapping from
CodeIdx to Span when the Chunk becomes immutable (i.e. at the end of
the compilation process, when compiler backtracking is no longer a
concern).  Since a Span is just 64 bits this is not a space issue,
and since binary search is much simpler than compiling Nix
expressions it isn't a performance issue either.

Technically there is no longer any reason to have LightSpan since it
is now a single-variant enum.  However there is no rush to remove
it, since Rust will optimize its representation into the same thing
you'd get if you replaced LightSpan by Span.

Prev-Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
This-Benchmark: {"nixpkgs-attrnames":{"kbytes":"230192","system":"0.29","user":"2.00"}}
Prev-Benchmark: {"pkgsCross.aarch64-multiplatform.hello.outPath":{"kbytes":"458936","system":"0.73","user":"5.36"}}
This-Benchmark: {"pkgsCross.aarch64-multiplatform.hello.outPath":{"kbytes":"451808","system":"0.53","user":"5.10"}}

Change-Id: Ib9e04806850aa1fc4e66e2a042703986440a7b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10254
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-12 14:34:28 +00:00
Adam Joseph
29878259b3 feat(tvix/cli): add benchmark for bf286a54bc
cl/7558 used this expression as a benchmark to justify the introduction
of LightSpan::Delayed:

  builtins.length (builtins.attrNames (import ${pkgs.path} {}))

Let's add it as a benchmark case so it can be referenced easily.

Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
Change-Id: Idb6c69ddd284605dd3b5fd9ac5c79a69b9a470b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10253
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:28:53 +00:00
Florian Klink
27c07b72c6 refactor(tvix): use io::Result for EvalIO
This is just a alias for Result<_, io::Error>, but shorter.

Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 14:28:50 +00:00
Adam Joseph
ad566999ca fix(tvix/eval): preserve catchables in nix_cmp_ordering(), fix b/338
This commit fixes b/338 by properly propagating catchables through
comparison operations.

Change-Id: I6b0283a40f228ecf9a6398d24c060bdacb1077cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10221
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
ae28dc3ca6 test(tvix/eval): test for b/338 catchable hygiene problem
Commit 05f42519b5 fixed b/281 by
establishing a hygiene regimen to partition *catchable* errors
(i.e. those which tryEval can detect) from all other errors, like
internal VM failures or I/O errors (which Nix must not be allowed to
detect, since these errors are fundamentally impure).

Unfotunately there are still cases where tvix assumes that anything
other than Value::Bool means it should panic!().  I found another
one, and added a test case for it in:

  eval_okay_src_tests_tvix_tests_eval_okay_compare_ordering_catchable_nix

Not yet passing.

Change-Id: I69c62ed9ea5c8f81870e8de5c5fe12dcde849763
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10220
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
edbd5055a1 feat(tvix/eval): nonrecursive nix_cmp_ordering(), fixes b/339
This commit rewrites Value::nix_cmp_ordering() into an equivalent
nonrecursive form.  Except for calls to Thunk::force(), the new form
no longer uses generators, and is async only because of the fact
that it calls Thunk::force().

I originally believed that this commit would make evaluation faster.
In fact it is slightly slower.  I believe this is due to the added
vec![] allocation.  I am investigating.

Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460224","system-seconds":"0.67","user-seconds":"5.84"}
Change-Id: Ic627bc220d9c5aa3c5e68b9b8bf199837cd55af5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10212
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
8a40f75c2d fix(tvix/eval): never use partial_cmp() (partial fix b/338)
This is part of a fix for b/338.

We should never use PartialOrd::partial_cmp().

All Nix types except floats are obviously totally-ordered.  In
addition, it turns out that because Nix treats division by zero
rather than producing a NaN, and because it does not support
"negative zero", even floats are in fact totally ordered in Nix.

Therefore, every call to PartialOrd::partial_cmp() in tvix is an
error.  We have to *implement* this function, but we should never
call it on built-in types.

Moreover, nix_cmp_ordering() currently returns an Option<Ordering>.
I'm not sure what was going on there, since it's impossible for it
to return None.  This commit fixes it to return simply Ordering
rather than Option<Ordering>.

Change-Id: If5c084164cf19cfb38c5a15554c0422faa5f895d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10218
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
19d13eb070 test(tvix/eval): add test case for b/339
Not yet passing.

Change-Id: I1de3f72d8b3f46567fdba010fc3ab4bace3f1699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10219
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
72ece2e518 feat(tvix/eval): nonrecursive nix_eq()
This commit rewrites Value::nix_eq() into an equivalent.  Except for
calls to Thunk::force(), the new form no longer uses generators, and
is async only because of the fact that it calls Thunk::force().

I believed that the nonrecursive form would be faster.  It is, in
fact, slightly slower.  I believe this is due to the vec![]
allocation; I am investigating.

Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"459068","system-seconds":"0.71","user-seconds":"5.39"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
Change-Id: I10f4868891e4b7475df13f0cbc41ec78dd985dd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10118
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
0c15a09b15 feat(tvix/cli): add macrobenchmark
This commit adds a simple MVP benchmark, built on our nix
infrastructure instead of cargo.  It simply runs `tvix-eval` inside
of GNU time, and prints the three essential statistics in a short
JSON blob.

You can run the benchmark with a simple `nix run`, like:

  nix run -f . tvix.cli.benchmark-hello
  nix run -f . tvix.cli.benchmark-firefox
  nix run -f . tvix.cli.benchmark-cross-firefox

Currently these blobs are stored only in the CI logs, which I'm sure
get garbage-collected at some point.  We should be putting them in
the git trailers, but that can wait for a future CL.

I tried using `cargo bench` for this but found it incredibly
frustrating.  Maybe I'm doing it wrong.  It seems to be designed for
microbenchmarks only, and very hard to control.  It kept building
all sorts of unnecessary stuff (like the tests), and unlike
crate2nix it was doing all the builds on only a single machine
instead of using more than one machine.  Worse, for that single
machine it kept picking my laptop instead of my fast servers!  It
seems excessively cargo-flavored for such a straightforward task.

Benchmark: {"hello.outPath":{"kbytes":"244736","system":"0.36","user":"2.76"}}
Benchmark: {"firefox.outPath":{"kbytes":"1506736","system":"2.38","user":"32.01"}}
Benchmark: {"pkgsCross.aarch64-multiplatform.firefox.outPath":{"kbytes":"11334548","system":"10.70","user":"107.07"}}

Change-Id: I85bc046ec551360284d7ecfc81a03914f0085909
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10216
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 11:56:58 +00:00
Florian Klink
8018313b68 feat(tvix/glue): add nixpkgs eval benchmark
This adds a criterion.rs-based testbench into tvix-glue.

It can be invoked by running `cargo bench` from inside the `tvix-glue`
crate.
`target/criterion/report/index.html` contains nice graphs.
It's able to diff against the previous run, so you can invoke `cargo
bench` before and after a certain change to reason about the impact in
evaluation performance.

Currently, we need to create a bunch of Evaluator resources inside the
benchmark loop itself, which is a bit annoying, as it leaks into the
things we benchmark.

This should become better with b/262.

Fixes b/322.

Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 10:27:51 +00:00
Florian Klink
6c87ab960d chore(tvix/glue): allow unused_variables
cl/9364 did introduce a warning here, which is visible when building in
release mode - or invoking `cargo bench` in tvix-glue.

Change-Id: Ia82082a58543f0fdd32866fdfcd37d0a5fdfda9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10261
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 09:01:51 +00:00
Florian Klink
ccaf10b4a6 refactor(tvix/*store/sled): make ::new() more generic
We don't really require the Path to be a PathBuf, we don't even require
it to be a Path, we only need it to be AsRef<Path>>.

This removes some conversion in the from_addr cases, which can just
reuse `url.path()` (a `&str`).

Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 22:46:50 +00:00
Florian Klink
9748543f1c chore(tvix): drop sled compression support
It's been a while since the last sled release, and that one binds to a
pretty old version of zstd, requiring workarounds like cl/10090.

Upstream sled main branch currently has zstd halfway patched out (it's
a no-op, but the feature flag and options are still there), and it's in
that state for a year.

Rather than maintaining our own fork of sled, let's just stop using the
compression feature in sled, dropping the version pin to zstd that way,
removing the need for cl/10090.

This doesn't mean we won't reintroduce per-blob compression - but we
probably just won't let sled take care of the compression, but do it
ourselves - which is necessary for more chunked blob storage anyways.

Even though we do drop the feature flag, we still need to explicitly use
use_compression(false).

Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 22:46:50 +00:00
Florian Klink
6025be423a chore(tvix/proto): drop evaluator.proto
This only contained the (unused) evaluator.proto file.

Considering we're less likely to have the CLI talk to a long-running
evaluator, but instead embed the evaluator inside the CLI, remove this.

If we add a RPC to speak to an evaluator, we can resurrect this from
git history.

Change-Id: I2196aade55221660330dfd32dc3e52c39ec6ed43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10241
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
1cc3e9c1a8 refactor(tvix): use granular proto files
Only pass in the proto files that are actually needed to build that
crate. They are already constructed in depot.tvix.$crate.protos.protos.

Change-Id: If4381e6c3350e420ee4ddce1e0513bfe970678a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10240
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
43943e2623 refactor(tvix/*/protos): separate lint target
Break up the go-bindings derivation. Keep "protos" containing all proto
files (well, and the buf config), and use it for a check phase running
linter and formatter, as well as the existing "go-bindings" attribute

Change-Id: I52cb9d08570bb76452acb831eb711c5b6c0eacfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10239
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
8486f87e3c feat(tvix/build): add derivation_to_build_request
This function converts from a nix_compat::derivation::Derivation to
a BuildRequest.

In addition to the Derivation itself, it needs two lookup functions to
map input paths to their castore nodes.

Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 22:35:39 +00:00
Florian Klink
92bd69aef2 docs(tvix/*store-go): fix README invocation
The command is called `regenerate`, not `generate`.

Change-Id: I18075042ebd461e4dd0718a936e6bbe738a144d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10259
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-11 21:47:43 +00:00
Florian Klink
a1b2dc8aaf feat(tvix/build-go): init
This adds the generated golang bindings for tvix-build.

Change-Id: I2eb0d1cc38bc2fa34afd7c904eea05c5ee192cce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10242
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 21:43:38 +00:00
Florian Klink
fd27d8ddc3 feat(tvix/build): init
This adds the tvix-build crate, currently only containing a
`tvix_build::proto` module, exposing the data structures defined in
tvix/build/protos.

Change-Id: I75f5d9196969ed0877b1fe640cacfecba0fb2e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10235
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 21:43:38 +00:00
Florian Klink
3c0a9a949a chore(tvix/store/protof): buf format
Change-Id: I29b0fb62aa849ee01860dc7fe3dbba9b941ca172
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10238
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 10:29:50 +00:00
Florian Klink
459d9e106f chore(tvix/castore/protof): buf format
Change-Id: Idf11de78b0d6eca69fda34a89f2c57a00ed89ad5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10237
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 10:24:44 +00:00
Aspen Smith
d9ac36d307 fix(tvix): Fix cwd in benchmark script
Change-Id: I6d9435c6710ce086aaedb2c1dc634b73a19f476c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10249
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-11 04:13:41 +00:00
Florian Klink
01a3c87a3a chore(tvix): bump criterion to 0.5
Change-Id: I28904ca23437b4bb745c0eb1f4eb9ae33e09eb5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10244
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 03:59:30 +00:00
Florian Klink
b14053ae03 docs(tvix): fix crate2nix-generate invocation
This got renamed recently, but we forgot to update it here.

Change-Id: I7d713c8a0e6ccca57fe67985d9cb4e7f1eeef3b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10243
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-11 03:59:28 +00:00
Aspen Smith
b2c8c1ef94 feat(tvix): Benchmark nixpkgs eval with hyperfine
Add some hyperfine benchmarks to Tvix's windtunnel benchmark script for
evaluating the outPath of hello and a cross-compiled hello.

Change-Id: I9d76e5ce0a3fd7d9c125c36c5fced675b660a8a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10248
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
2023-12-11 03:59:27 +00:00
Adam Joseph
2b4920c355 fix(tvix/cli): panic on root cause of the fetchurl(non-boot) bug
Currently we produce wrong drvPaths for a large number of packages
that use fetchurl (but not fetchurlBoot, which is what stdenv uses).
A simple reproducer is `pkgs.perl538`.

I debugged this down to the root cause, which is the fact that tvix
doesn't realize that the mapping from FOD-paths to outputHash is
*NOT* a 1:1 mapping.  It is a many-to-one mapping.  You can have
lots of different FODs with the same outputHash or even the same
outPath.  For example, perl538.src and perldevel.src use the same
source tarball but a different `version`.

Anyways, I have found the root cause but have run out of time for a
while, so I've added a panic!() to in the spot where we have a logic
bug in order to call it out.

Change-Id: I9766b39cfe2fe7eafec84945b2ad6cc28f9c4b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9364
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-10 14:16:21 +00:00
Florian Klink
efaff79584 feat(tvix): make clippy stricter
Also run clippy on tests, and enable all features.

Change-Id: Ide9f1bc9f565333072afb918c391c7930b658f41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10234
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
22a669d27e refactor(tvix/castore): address clippy
We match to destructure a single pattern.

Change-Id: I564a3510b4860e90b3315a9639effc48ee88b483
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10233
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
Florian Klink
340242174d refactor(nix-compat/aterm): address clippy
This pushes to a Vec immediately after creation.

Change-Id: I2360b45810475d98ededc1d097fb4cbdeabc576b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10232
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-09 12:58:39 +00:00
Florian Klink
f9ff4fe969 refactor(store/fs): address clippy
Change-Id: I28d60263ef672942940656caa8e0f9d20cc6b7d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10231
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
f4c4e32040 refactor(nix-compat/nixbase32): address clippy
Change-Id: I07b53ead10b0efb31e88c6ae93c124c5f35261d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10230
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
06350445e9 refactor(nix-compat/narinfo): address clippy lints
Change-Id: I00f28624ec0469525e52f408f00caa20d3b701b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10229
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
1bdb90e56d refactor(nix-compat/derivation): address clippy
Change-Id: I5d7057347838f733d7dc1331c7b7ef8d27244412
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10228
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-12-09 12:58:39 +00:00
Florian Klink
d3ecef1a5b refactor(tvix/eval): address clippy lints
Change-Id: Ic2bd4e8291b30ceac9fa0e88a4f56e61ae99b603
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10227
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
Florian Klink
f11c6d610c chore(tvix/nix_cli): drop crate
The only thing this was doing was invoking nix-store --add, which is not
gonna help us populate the tvix-store at all (and we now have
`tvix-store import`).

This is also (rightfully) causing clippy warnings, because of some
fields being unused.

It's more of a skeleton, and rather than shelling out to Nix for some
usecases, we might introduce a "compatible" Nix CLI frontend for a
subset of commands.

Drop this for now, to decrease the noise and confusion.

Change-Id: I2fd399e9320260f08893b685561755af9c7c961c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10226
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
0907420600 refactor(nix-compat/store_path): from_absolute_path to StorePathRef
The only non-test usage was only checking for the error case, and we can
still convert this to an owned StorePath by calling to_owned() on
StorePathRef.

Change-Id: I9f67a759e580c9c429c96896bcdd295392aa5a2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10225
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:35 +00:00
Florian Klink
5855959d87 refactor(nix-compat/derivation): rename input derivation components
Match the naming in parse_input_derivations, call the keys
"input_derivation", and the values "output_names".

Change-Id: I7d1974819028f8ea543dc3ad78afb803ff9db865
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10224
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:20:51 +00:00
Adam Joseph
5fd42a9e70 feat(tvix/eval): impl DoubleEndedIter for OwnedAttrsIterator
Change-Id: I4bd85dbe9c27047f4abbdeff4e2b796e9bcab3a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-09 11:35:19 +00:00
Adam Joseph
36c3e646e1 chore(store/src/tests): make clippy shut up
Change-Id: I6dfceaa32a8c01f8395b7889ae19847fabf95ed3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10214
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 10:51:10 +00:00
Adam Joseph
dfca879535 fix(tvix/glue): do not panic if PathInfoService returns Err
Nixpkgs tries to `import` the value returned by
`builtins.unsafeGetAttrPos`, which in our case is the file
`/deep/thought`.  Since that doesn't exist, tvix-glue panics, but
there's no interpreter backtrace to follow.

Let's return an Err instead of panicking.

------------------------------------------------------------------------------
Before:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at src/tvix_store_io.rs:276:58:
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(41580), ...)
Command exited with non-zero status 101

------------------------------------------------------------------------------
After:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: while evaluating this Nix code
   --> [code]:1:1
    |
1   | (import /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src {}).pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:6:78
    |
6   |   runCommand "rocm-llvm-binutils-${llvm.version}" { preferLocalBuild = true; } ''
    |  ______________________________________________________________________________^
7   | |   mkdir -p $out/bin
8   | |
9   | |   for prog in ${lld}/bin/*; do
...   |
27  | |   ln -s ${lld}/bin/lld $out/bin/ld
28  | | ''
    | |__^
note: while evaluating this as native code (coerce_to_string)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:27:9
    |
27  |   ln -s ${lld}/bin/lld $out/bin/ld
    |         ^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:23
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:30
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                              ^^^^^^^^^
note: while evaluating this as native code (throw)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:262:8
    |
262 |     in handler msg;
    |        ^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |________________________________________________________^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |__________^
error[E029]: I/O error: /deep/thought:42: task panicked
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:255:41
    |
255 |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
    |                                         ^^^^^^^^^^^^^^^

Command exited with non-zero status 1
Benchmark: {"pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath":{"kbytes":"26613180","system":"22.35","user":"140.62"}}

Change-Id: I587b57e9e49d1f3ecdc0fc9cf996d179a3548f34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10223
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 10:47:39 +00:00
Adam Joseph
f4460bd612 chore(nix_cli): make clippy shut up
Change-Id: I0c243baf4deecfd81bd939da86d685e2cbc59d69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10215
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-09 09:25:24 +00:00
Adam Joseph
2132bbcb47 fix(tvix/nar-bridge): fix FOD hash
The golang mothership seems to be monkeying with hashes again.

Change-Id: I7430b4cde84fa51be2b572fba02e3567864bb87a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10209
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 09:24:53 +00:00
Adam Joseph
0e2633048e feat(tvix): run crate2nix generate in CI
This runs `crate2nix generate` in CI and then runs `depotfmt` on the
result to ensure that our machine-generated code is really, really
readable and pretty.  Then it checks that the result of all that
is identical to the committed Cargo.nix.

A self-hashing FOD is used to allow network access.

No magic hashes are involved.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I68ec5003dbc6a40894a5a4d6e902f138c99f6719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10194
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-07 18:43:46 +00:00
Aspen Smith
0b524890e9 refactor(tvix): Make benchmark script directory-agnostic
Allow running the benchmark script from any directory - primarily so
Windtunnel can point to the /tvix josh workspace rather than the depot
overall

Change-Id: Ie5fc3ef995bf8114277298ae5c5010e6a0bf13ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10205
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2023-12-06 16:19:46 +00:00
Adam Joseph
49b34183e3 feat(tvix/eval): rewrite Thunk::force() in nonrecursive form
This commit rewrites Thunk::force() so that it is not (directly)
self-recursive.  It maintains a Vec of all the
previously-encountered thunks which point to the one it is currently
forcing, rather than recursively calling itself.

Benefits:

- Short term:

  This commit saves the cost of a round-trip through the generator
  machinery for the generators::request_force() which is removed by
  this commit.

- Medium term:

  Once a similar transformation has been applied to nix_cmp(),
  nix_add(), nix_eq(), and coerce_to_string(), those four functions,
  along with Thunk::force(), will make non-tail calls only to each
  other.  They can then be merged into a single tail-recursive
  function which does not use the generator machinery at all:

    enum Task { Cmp, Add, Eq, CoerceToString, Force};

    fn Value::walk(task:Task, v1:Value, v2:Value) {
      // ...

- Long term:

  The long-term goal here is to use generators **only for builtins**
  and [Marionette]-style remote control of the VM.  In other words:
  use `async` for things that actually involve concurrency.  Calls
  from the VM to builtins can then be blocking calls, because even
  cppnix will overflow the stack if you make a MAX_STACK_DEPTH-deep
  recursive call which passes through a builtin at every stack frame
  (e.g. `{ func = builtins.sort (a: b: ... func ...) ...}`).

  This way the inner "tight loop" of the interpreter doesn't pay the
  costs of `async` and generators.  These costs manifest in terms
  of: performance, complex nonlocal control flow, and language
  impediments (async Rust is a restricted subset of real Rust, and
  is missing things like traits).

[Marionette]: https://firefox-source-docs.mozilla.org/testing/marionette/Intro.html

Change-Id: I6179b8abb2ea0492180fcb347f37595a14665777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10039
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-06 06:53:01 +00:00
Aspen Smith
8135a8d38c fix(tvix/eval): Return error rather than panicking on bad substring
If builtins.substring is invoked with (byte!!) offsets that aren't at
codepoint boundaries, return an error rather than panicking. This is
still incorrect (see b/337) but pushes the incorrectness forward a step.

Change-Id: I5a4261f2ff250874cd36489ef598dcf886669d04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10199
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-05 23:12:23 +00:00
Florian Klink
41235bf908 refactor(tvix): move src into let binding
Change-Id: Ida2a3ac722fb2445745759323975884dfeef3e87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10193
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-05 21:55:09 +00:00
Aspen Smith
6edcb03cb6 feat(tvix): Add script for running benchmarks in Windtunnel
Currently this just uses a Docker container, which is gross but works
fine for now since we don't have the ability to build benchmarks in nix
as of cl/7538

Change-Id: I48e317f44bc2c73533d7663403786a3a37c7952f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10189
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
2023-12-05 17:46:26 +00:00
Florian Klink
dfb48dcade fix(tvix/nar-bridge): drop pathinfoservice
This now exists in tvix-store directly, as NixHTTPPathInfoService, and
contrary to this version, also validates signatures.

Change-Id: Ib6ca161e40d627b7d9741839fc849f2392f422da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10155
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-28 16:08:59 +00:00
Florian Klink
563886c3de feat(tvix/store/pathinfosvc/nix_http): allow configuring pubkeys
This allows setitng the trusted-public-keys URL parameter to a
(whitespace-separated) list of public keys.

NARInfo files retrieved need to contain a valid signature.

Change-Id: Ifd6580b723cbae3182e9cadfa54f1ca2b41d6599
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10153
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
4297e33d94 feat(tvix/store/pathinfosvc): add signature verification
Introduce an Option<Vec<narinfo::PubKey>>, configurable with a
`set_public_keys` method.

If set, this configures NixHTTPPathInfoService to validate signatures.

Change-Id: I157c5e13c41fc9bfd40b0655381fb4cf33900868
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10152
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-28 16:08:59 +00:00
Florian Klink
5fc737b02e feat(nix-compat/narinfo): add PubKey::verify(fp, sig)
This makes it easy for each PubKey to check if a given Signature is
correct for a given fingerprint.

Change-Id: I56e6211d133f74f390fd1ae3ae799eef12221904
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10151
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
6af67af76e feat(nix-compat/narinfo): add PubKey
This represents a ed25519 public key and "name".
These are normally passed in the `trusted-public-keys` Nix config option,
and consist of a name and base64-encoded ed25519 pubkey, separated by a `:`.

Change-Id: I9ab4b3e0e5821805ea6faf2499626630fc5a3f0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10150
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
e7a86273b1 refactor(nix-compat/narinfo/signature): rename Error type
Just call this Error, we can infer from the package what error this is.

Change-Id: I5df25d2873ec739c49c08804f35562c84c222e06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10149
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
7e8719be91 feat(tvix/store/pathinfosvc/nix_http): check Nar{Size,Hash} matches
Ensure the initially communicated NarHash/NarSize from the NarInfo
matches what we read, and don't return a PathInfo message if there's
a mismatch.

Also move the buffering layer around a bit.

Change-Id: I68c60ecfaf0f9cd5edacea648437ecb0c9729251
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10148
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
fce9c0e99e fix(tvix/store/fs): don't panic on PathInfoService error
An error in the PathInfoService request can appear in case the
underlying request returns an error.

We shouldn't panic and bork the fuse mount, but instead return an IO
error.

Change-Id: I2daeae629e1627d06adcd7b82ddb76c50c602212
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10154
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-28 14:54:56 +00:00
Vincent Ambo
360c98ce38 chore(3p/sources): bump channels & overlays (2023-11-27)
* picked avrdude from stable channel
* removed override for texlive, as the upstream fix is merged
* picked awscli2 from stable channel
* bump tdlib to 1.8.21 (new minimum for telega.el)
* tvix/turbofetch: switch to nixpkgs-native mechanism for
  CARGO_MANIFEST_LINKS (whatever that is)

Change-Id: Ic695721b5ca750b89d21cab7a257e1db682b23c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10083
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-27 21:16:46 +00:00
Florian Klink
0415bc6fd2 fix(nix-compat/narinfo/signature): validate name field
We should restrict this to alphanumeric mostly, and we definitely don't
want newlines.

Not entirely sure about the exact additionally allowed characters
outside of alphanumeric, but this can always be extended further.

Change-Id: I1357e79e553f2df2fa97792889f63f0f35d50ed5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10147
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-27 15:12:13 +00:00
Florian Klink
b7de931cc6 fix(nix-compat/narinfo): don't panic trying to parse signatures
BASE64.decode_mut panics if we're passing data that has the wrong size.
Do the size check first and error out there.

Also update the error, and talk about b64-encoded sizes.

Change-Id: I290f80a37d48526a30bf1df9d1d9fe34865008eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10146
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-11-27 15:12:13 +00:00
Florian Klink
dfaaf41cef refactor(nix-compat): use ed25519_dalek::SIGNATURE_LENGTH
No need to hardcode magic numbers here, we have a constant for that.

Change-Id: I67b671c0c4bb7c3bfb001e9c36499f31873ee717
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10145
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-27 15:11:41 +00:00
Vincent Ambo
e65fa82d74 refactor(tvix/eval): use or_default helper in entry API
This fixes a future clippy lint.

Change-Id: Ic830e94ef23595580c1037f10878c76bbb546dd9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10110
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-11-25 15:48:46 +00:00
Vincent Ambo
5ffb997864 fix(tvix): ensure PartialOrd/Ord agree for StorePath & NixString
This fixes a *future* clippy lint:
https://rust-lang.github.io/rust-clippy/master/index.html#/incorrect_partial_ord_impl_on_ord_type

In essence, because the implementation of *both* Ord and PartialOrd
implies that ordering is not partial, all results of PartialOrd should
simply be those of Ord. This is to avoid subtle bugs in future
refactorings.

Change-Id: I8fc6694010208752dd47746a2aaaeca0c788d574
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10109
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-25 15:48:46 +00:00
Adam Joseph
512346ba0b refactor(tvix/eval): add ThunkRepr::is_forced()
Change-Id: I4eab5c81fb82337da06327248845cd2f3a4490d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10038
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-11-25 02:55:57 +00:00
Adam Joseph
b5a15989cd feat(tvix/eval): add Thunk::unwrap_or_clone()
This commit adds Thunk::unwrap_or_clone(), which uses
Rc::try_unwrap() to avoid cloning the Value out of a an Rc which has
only one strong reference.

Change-Id: Icacefe0c823dcddf046d90c0c5cd5ed59fe976d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10037
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-11-25 02:55:56 +00:00
sterni
875bb26fc3 fix(tvix/castore): correctly flag unreachable code
Change-Id: Id09afa4b77c3c70fb5695f253f6df4aa88b61e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10113
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-24 23:36:15 +00:00
sterni
c5961e7774 docs(tvix/eval): optimization potential for inherit (from) exprs
Change-Id: Ibddaa111a5b7a86c42dbe153ae8e53f9a5601a54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10112
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-11-24 23:36:15 +00:00
Florian Klink
6b685ec4a5 feat(tvix/store): add as_narinfo() for PathInfo
This allows seeing a PathInfo as a nix_compat::narinfo::NarInfo<'_>.

It doesn't allocate any new data, but the NarInfo<'_> view allows us to
access things like signature verification, or rendering out
(alternations of this) as strings.

Change-Id: Id0d8d7feeb626ee02c3d8a4932f24ace77022619
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10108
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
639cca3e22 feat(nix-compat/narinfo/signature): add new() constructor
This is useful when creating a new Signature struct where the individual
elements are already parsed.

Change-Id: Ie33c66287641951e7a030aaa1e7ff0a86b2628ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10111
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
c44cbc852a refactor(tvix/store): impl From<&nar_info::Ca> for nixhash::CAHash
Change-Id: I637a4cff5a5ca29c4d86e0b76a2f20f8741f5628
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10107
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
1e04f60800 refactor(tvix/store): impl From<nixhash::CAHash> for nar_info::Ca
Change-Id: Iaa68044d3b469f15a932aa3b59548505eaa6b8bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10106
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:11:49 +00:00
Florian Klink
671206a63f refactor(tvix/nix-compat): move from_name_and_digest to StorePathRef
We can simply use .to_owned() on that thing afterwards if we want to
construct an owned StorePath.

Change-Id: I0f3e2e4434b99ee522f2a7dbfa391e13a987479c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10105
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:11:49 +00:00
Florian Klink
eb84898c17 feat(nix-compat/narinfo): drop .drv from Narinfo.deriver field
We always know this needs to end with a .drv, and fail parsing if it
doesn't, so there's no need to hang onto these 4 bytes.

This will make it much easier to synthesize a NarInfo<'_> later on from
a PathInfo proto, because we don't have to make this ".drv" appear out
of thin air.

Change-Id: Id95e7fd937d7c9a420a39b5a4bab73985640ca3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10084
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-11-22 18:24:04 +00:00
Florian Klink
ef8a8af0bf refactor(tvix/nix-compat): cleanup parse_{ca,hash} and fmt structs
These were used to format to and parse from strings.

Move this to the CAHash and NixHash structs directly, and be explicit in
the name about which encoding for digests is used.

For output path calculation, nix encodes the nixpaths in hex, but for
writing out NARInfos, it's using nixbase32.

Change-Id: Ia585a76a3811b2609e7ce259fda66a29403b7e07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10079
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-22 17:54:59 +00:00
Florian Klink
a834966efd feat(tvix/nix-compat/narinfo): add fingerprint
This adds support to compute the fingerprint string, which is what's
ed25519-signed in binary caches.

Change-Id: I8947239c609896acfd7261f110450014bedf465a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10080
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 22:34:59 +00:00
Florian Klink
bb18556bf3 feat(tvix/nix-compat): add narinfo::Signature::verify
This adds support for verifying signatures found in NARInfo files.

This still needs to be hooked together with the nix+http[s] backend.

Change-Id: Ic1c8ddbdecfb05cefca2492808388b0f7f3f2637
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10081
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-19 22:06:07 +00:00
Florian Klink
a5749fada5 refactor(nix-compat/narinfo): move signature into separate file
Change-Id: Ic257475e2afebf059c5317c1cc5b04ba63d5d318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10078
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 21:58:02 +00:00
Florian Klink
6039b97b55 refactor(tvix/nix-compat): move narinfo into separate mod
Change-Id: Id85f979e46946da0345483cbbc6de3dd29c94c63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10077
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 21:58:02 +00:00
Florian Klink
be48ba75ab feat(tvix/store/pathinfoservice): implement NixHTTPPathInfoService
NixHTTPPathInfoService acts as a bridge in between the Nix HTTP Binary
cache protocol provided by Nix binary caches such as cache.nixos.org,
and the Tvix Store Model.
It implements the [PathInfoService] trait in an interesting way: Every
[PathInfoService::get] fetches the .narinfo and referred NAR file,
inserting components into a [BlobService] and [DirectoryService], then
returning a [PathInfo] struct with the root.
Due to this being quite a costly operation, clients are expected to
layer this service with store composition, so they're only ingested
once.
The client is expected to be (indirectly) using the same [BlobService]
and [DirectoryService], so able to fetch referred Directories and Blobs.
[PathInfoService::put] and [PathInfoService::nar] are not implemented
and return an error if called.

This behaves very similar to the nar-bridge-pathinfo code in nar-bridge,
except it's now in Rust.

Change-Id: Ia03d4fed9d0657965d100299af97cd917a03f2f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10069
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 13:19:24 +00:00
Florian Klink
4e9e4b19ef refactor(tvix/castore/blobservice): rm AsyncBufRead from BlobReader
There's no need to already require this to be buffered here.

Change-Id: Ib9a11b194e0754d87ab8d2ef0b8cb0f4edc01229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10074
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 12:54:19 +00:00
edef
a11abc02e2 feat(nix-compat/nar/reader): provide passthrough buffered I/O
Allow taking advantage of the buffer of the underlying reader to avoid
unnecessary copies of file data.

We can't easily implement the methods of BufRead directly, since we
have some extra I/O to perform in the final consume() invocation.

That could be resolved at the cost of additional bookkeeping, but this
will suffice for now.

Change-Id: I8100cf0abd79e7469670b8596bd989be5db44a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10089
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 09:53:10 +00:00
edef
785ff80c8b fix(nix-compat/nar/reader): require BufRead
We rely on being able to make small reads cheaply, so this was already
an implicit practical requirement. Requiring it explicitly removes a
performance footgun, and makes further optimisations possible.

Change-Id: I7f65880a41b1d6b5e6bf2e52dfe47d4c49b34bcd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10088
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-19 09:53:10 +00:00
edef
2eaee1d48e fix(nix-compat/store_path): valid names ⊊ UTF-8
We don't need to validate UTF-8 separately, since valid names are
a strict subset of ASCII, and therefore a strict subset of UTF-8.

Change-Id: I3261bf0efe3480b5b315074efafcf5e47a6c5a65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10087
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-19 09:53:10 +00:00
edef
9ad7cc629e fix(tvix): patch futures::AsyncBufReadExt::fill_buf
This fixes EOF handling for buffered readers.

Link: https://github.com/rust-lang/futures-rs/pull/2801
Change-Id: Ie98ca6a3e1de38500b0195e9b62511501acb1d2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10086
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 08:35:38 +00:00
edef
479f26f46c chore(tvix): upgrade futures to 0.3.29
Change-Id: I8fd63be3cbec8766fd6d72cd9271989a19774816
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10085
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 08:35:38 +00:00
Florian Klink
8beb38b50c refactor(tvix/store/fs): simpllify read
We can just use take(size) to restrict reading to that as a max.

Change-Id: I0fbda74e4fb98ffeababae86a325233416029acf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10072
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-18 23:07:24 +00:00
Florian Klink
eda5d4da37 feat(tvix/store): From<&nix_compat::...::NarInfo<'_>> for PathInfo
This allows converting from the NarInfo falling out of the NarInfo
parser (which is a bit annoying to handle due to lifetimes) to the
PathInfo proto struct.

The narinfo field, containing most of the data from the original
NARInfo file, as well as the references (bytes) are populated.
The node field is not populated, because it requires ingesting the NAR
itself to describe the root node.

Change-Id: I9c04dd6ad4cae556b455188a4255e34b4f6443c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10067
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-18 19:38:15 +00:00
Florian Klink
e32c2070e4 refactor(tvix/nix-compat): no impl <StorePathRef<'_>> for StorePath
This suggests it's cheap to convert around, but name actually does
allocate.

Move to a `to_owned(&self) -> StorePath`, to better signal that this
does allocate.

Change-Id: Ifaf7c21599e2a467d06e2b4ae1364228370275db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10066
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-18 19:37:04 +00:00
Ryan Lahfa
df63b719ac feat(tvix/castore): fix tracing instrument in MemoryBlobService
Change-Id: Iedba57e8b3e1a44f14f5baa1e981275d4b02eb56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10070
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-18 19:23:55 +00:00
Florian Klink
68e473ed6b feat(tvix/castore): impl From<std::io::Error> for Error
Make it less annoying to convert from io::Error to this. We already have
one direction, doesn't hurt to have the other too.

Change-Id: I9fe2c6da608c9d54910ee8c397572aadb1d90d99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10068
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-18 19:23:47 +00:00
Florian Klink
ebfe456251 refactor(tvix/castore/tonic): use match in channel_from_url
Having random if blocks and returning from them is error-prone.

Also, turns out we only need the unprefixed scheme in the fallback case,
so move it down to there.

Change-Id: Ifcb09279c963f8a39e0dbabe145990263f3d7cf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10041
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-17 14:01:26 +00:00
Florian Klink
d4d1387409 docs(tvix/glue): fix doc-comment reference
This has been renamed to descend_to in cl/9373.

Change-Id: Ia6201fb81c7d4fa953d311451cfff95373549a50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10045
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-16 21:26:54 +00:00
Florian Klink
4e5165d0a1 refactor(tvix/castore/utils): drop unused DuplexStreamWrapper
This wasn't used at all, let's remove it.

Change-Id: I426e3d93c32ebe65247ae5cf8d05b5bf686be2d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10044
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-11-15 22:04:04 +00:00
Florian Klink
30e0c32066 refactor(tvix/castore/tonic): make async, support wait-connect=?
This moves the sync `channel::from_url` to a async
`tonic::channel_from_url`. It now allows connecting non-lazily if `wait-
connect=1` is set in the URL params.

Also, make the pingpong tests for blobsvc and directorysvc use the wait-
connect=1 codepath.

Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
c83841d3a1 refactor(tvix/castore): remove DirectoryService::from_url
Make directoryservice::from_addr use the more specific constructors.

Change-Id: I9fee2afed77692505988d631d9fe246d9843d25a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10029
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
840bee1e97 refactor(tvix/castore/blobsvc): remove BlobService::from_url
Make blobservice::from_addr use the more specific constructors.

Change-Id: Id9637e279d6910ce6d92ff0086a984be5c65a8c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10028
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
25eca8af32 refactor(tvix/store/pathinfosvc/from_addr): use test_case
All we do is constructing some strings, and checking if from_addr
succeeds or not.

This can be written in a much more concise way using test_case.

Use lazy_static to provide temporary directories.

Also add some more grpc-related test cases.

Change-Id: Ia310dd01f617f7628f1e7e21304ac70da2ab3534
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10027
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-11-15 06:43:31 +00:00
Florian Klink
756290a4c0 refactor(tvix/store/pathinfosvc): inline SledPathInfoSvc::from_url
Change-Id: I0d905228df086a422bb30322add7236ca41e807b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10026
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
f57fd16d6e refactor(tvix/store/pathinfosvc): inline GRPCPathInfoSvc::from_url
Change-Id: Ib53b5525ae13c276e61b7f564673b7c6144ffc0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10025
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
362117fbf5 feat(tvix/castore/src/channel): move from_url tests
These gRPC PathInfoService tests were actually not too useful in here,
what we're mostly testing is the channel construction, so move it to
there.

Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-11-15 06:43:31 +00:00
Florian Klink
9aef3cfc89 refactor(tvix/store/pathinfosvc): inline MemoryPathInfoSvc::from_url
Change-Id: If27eb518d372f4004b7b38fc765a42957f2a6b50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10023
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
8111caebc2 refactor(tvix/store): remove from_url from PathInfoService trait
We don't gain much from making this part of the trait, it's still up to
`tvix_store::pathinfoservice::from_addr` to do most of the construction.

Move it out of the trait and into the specific *Service impls directly.

This allows further refactorings in followup CLs.

Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
edef
8d02928b14 feat(tvix/tools/narinfo2parquet): init
Convert turbofetch output to queryable Parquet.

Change-Id: I076f5a431f8aab8cfe7d973bdc9fe019cebde111
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9989
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:35:04 +00:00
edef
45de341794 chore(nix-compat/store_path): use hex_literal
Change-Id: Id093a0131aa7e3ac532daffbf5a883ca213c83ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9996
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 19:35:04 +00:00
edef
2997c1c304 feat(nix-compat/narinfo): turn flags into bitfields
Change-Id: I8b95723444013e97bc6ec8d282c7135b1aede114
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9987
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
1df53a5fcd feat(nix-compat/narinfo): permit out-of-order references
This appears in the cache.nixos.org dataset.

Change-Id: I2eadafe8441e0132a448828026553da2dc7c12aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9994
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 19:05:36 +00:00
edef
24187cfeec feat(nix-compat/narinfo): accept hex-encoded NarHash
This appears in the cache.nixos.org dataset.

Change-Id: I35921f7ef148f6681081a4e371abb8c9cc98854d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9993
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
4218e4dc0e feat(nix-compat/nixbase32): use data_encoding::DecodeError
Rather than having our own error type, just make decoding errors use
the same common error type.

Change-Id: Ie2c86972f3745c695253adc3214444ac0ab8db6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
7e317cfded feat(nix-compat/narinfo): default compression to bzip2
Change-Id: Ibdebc0a69672c00026c8748b93fcd2a83a0e69a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9992
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 19:05:36 +00:00
edef
407e5a53ed feat(nix-compat/narinfo): track presence of unknown fields
Change-Id: Ia3f8a86209a0045ff98322b56a21ae20220fbe99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9991
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
8694694b74 feat(nix-compat/narinfo): permit non-SHA256 CAHash::Nar
This appears in the cache.nixos.org dataset.

Change-Id: I055b60b9950a1a6a36c1b0576b957e11e1d4264b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9990
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
bdda10a2f5 feat(tvix/tools/turbofetch): init
Change-Id: I2efa6f94f57e812c52371256a4e62d1d54ff5057
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9925
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
Florian Klink
9cd2e92065 refactor(tvix/store/fs): mv sparse -> populated directories
Do this upgrade whenever someone is actually interested in the children
of a directory, but that directory doesn't contain a more detailed
listing. This is much more predictable, and removes a bunch of confusing
code from the inode tracker itself.

Change-Id: Ib3a13694d6d5d22887d2d04ae429592137f39cb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9982
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 20:19:38 +00:00
Florian Klink
a778b855d2 refactor(tvix/store/fs): simplify name_in_root_to_ino_and_data
Have it return libc::ENOENT errors rather than an Option<…>.

Also avoid having to traverse inode_data multiple times, by synthesizing
the Arc<…> on our own in the insert case. In that case, the data is
quite small, so cloning it is faster than traversing a second time.

Change-Id: I7ab14bac8bb23859ed8d166a12070d4f4749b6d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9981
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 14:52:10 +00:00
Florian Klink
3ae48465fa refactor(tvix/store/fs): move code to get_directory_children helper
As already established in the two previous CLs, these two pieces of code
where doing the same.

Move to a get_directory_children helper.

Change-Id: Id6876f0c34f3f40a31a22d59a2cdbfef39e2d8de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9980
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-11-07 14:52:10 +00:00
Florian Klink
9e6d89983a refactor(tvix/store/fs): reduce write lock, return children
Very similar to the previous CL

Change-Id: I0df07ddca742b7b9485d48771c8d295dc3aa7136
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9979
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 14:52:10 +00:00
Florian Klink
2405399580 refactor(tvix/src/fs): reduce write lock, avoid inode_tracker lookup
Code after this big match block only cares about parent_digest and
children, so there's no need to do another inode_tracker.get in there.

This also allows removing another if let block, right after, as we don't
need to destructure parent_data anymore.

Change-Id: I68fbbe3304194670caee5a453722369afa4e77ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9978
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 14:52:10 +00:00
Florian Klink
f3a240bf86 refactor(tvix/store/fs): move inode for store_path lookup to helper
This makes it much harder to keep the read lock around for too long, and
the code a bit easier to understand.

Change-Id: I7d99c85cadd433cad444b8edd34e2c43d7eaf5a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9977
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 14:52:10 +00:00
edef
a8e7f4eadb fix(tvix/castore): B3Digest::{to_vec -> as_slice}
Not a single call site actually makes use of the Vec.

Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-07 12:35:41 +00:00
Florian Klink
9f5b1213f9 feat(tvix/nix-compat): add drvfmt
This small tool formats A-Term in a more readable format. It's a lossy
conversion for non-valid UTF-8 environment values.

Change-Id: I65a51054d7faf528321bc2d9fc4425180a7813f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9970
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-07 11:27:08 +00:00
Vincent Ambo
7c32d85455 docs(tvix/eval): document where EvalIO methods are used
Change-Id: I335f2ba4420973861c2a22125995b45a34d3608c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9969
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-06 09:06:26 +00:00
Florian Klink
c86de1b2c4 chore(tvix/nar-bridge): bump castore-go
Change-Id: I81814ae326fb90ab50201cd06e3d9c0a3b15aec5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9958
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-06 08:34:14 +00:00
Florian Klink
72760b7716 chore(tvix/store-go): bump castore-go
Change-Id: I740251785f4ffb6c9305f430e0f2c3418dfaa15d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9957
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 20:32:34 +00:00
Florian Klink
a9329ff811 feat(tvix): add clippy target in CI
This will invoke `cargo clippy` on the codebase, ensuring we spot things
early on.

Relates to b/321.

Change-Id: Ifba13e5bef995d33b24346c6cee134e84d6566c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9106
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 20:28:53 +00:00
Vincent Ambo
f325456cba chore(tvix): bump proptest dependency
This *might* contain a fix for a clippy lint thrown by that crate.

Relates to b/321.

Change-Id: Ia7ebd3e26e0feb8bcc7a6c811b1e583f9016fd9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9966
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 20:28:38 +00:00
Vincent Ambo
87d3fac676 chore(tvix): add missing clippy attributes & config
For cases where clippy lints don't apply to us, or something is
misfiring, add appropriate configuration.

Relates to b/321.

Change-Id: I0af453910b4a4112bf685b2a8e9a73de10ec87ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9965
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-05 20:28:38 +00:00
Vincent Ambo
2e1399698b refactor(tvix/eval): use IntoIterator trait for owned NixAttrs iter
Uses the standard library IntoIterator trait for the construction of
our iterators. Clippy complains about duplicating this.

While doing this, I opted to rename the `IntoIter` type into something
that is more useful to users, in case somebody ends up working with
these manually.

This fixes a clippy lint, and is related to b/321.

Change-Id: I851fde0d7b8b38d182343a0fd6d9f8dd2a33ee11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9963
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 20:28:38 +00:00
Vincent Ambo
327548d2fd refactor(tvix/nix-compat): check presence with btree_map's entry API
Walking a btree_map twice is more expensive than copying a string,
especially because the cloning only happens in the (non-hot) error
path.

This fixes a clippy lint, so it's related to b/321.

Change-Id: I2ccfd0bc46792a45d277f47564e595b87107d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-05 20:28:37 +00:00
Vincent Ambo
b3b1f649d6 chore(tvix): fix trivial clippy lints
Relates to b/321.

Change-Id: I37284f89b186e469eb432e2bbedb37aa125a6ad4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9961
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 20:28:37 +00:00
Florian Klink
67999f0dcf feat(tvix/castore): extend blobstore protos for verified streaming
This pdates the proto docstrings a bit, especially w.r.t. verified
streaming.
It also adds send_chunks, send_bao fields to StatBlobRequest (renamed
from BlobMeta)

Change-Id: I590cc8646d86b73bca9f38a9b6d9ea15e4df5cb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9951
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 15:13:52 +00:00
Florian Klink
b921e3a7e3 fix(tvix/*store-go): fix depot checks
This wasn't doing anything, because $(regenerate) was a directory:

/nix/store/cxfxvz76zh9vb7x26h3cx98gkv234jz2-pb-go-check: line 2: /nix/store/my1nd1qvg5iis38rfyn1pm2c7ib7myn5-regenerate: Is a directory

Change-Id: I6ebed1d7b84dceb885c1f25527ce62d973146819
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9959
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 13:20:55 +00:00
Florian Klink
231f032eb5 chore(tvix/store-go): regenerate proto bindings
As surfaced in cl/9959, we didn't actually spot having to run the
regeneration in CI.

Change-Id: Iabdb79339d2a220a6cb368f9c5c70166f1ce9eb3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9960
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 13:20:55 +00:00
Florian Klink
cbd72b56b2 feat(tvix/store/nar): add read_nar method
This adds a function that can ingest a NAR file into a given
blob_service and directory_service, returning the root node.

Passing around the directory_putter is a bit annoying, but at least for
this linear "reading through a NAR" case, we don't need it to be clone
or sync.

Change-Id: Ia26d596f5dffc6dbd69aa39a82b59bba41685a9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9915
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-05 12:23:55 +00:00
Florian Klink
6d7e619b8f refactor(tvix/nix-compat): remove unused imports
Change-Id: I64523df2344233d9e424812d94b1c7ed2ecb1a74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9956
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 12:20:19 +00:00
Florian Klink
65e5c31adf refactor(tvix/nix-compat): use matches! macro
Flagged by ``#[warn(clippy::match_like_matches_macro)]`.`

Change-Id: If07109e5ec01b05df898119f9a577196dfe11b37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9955
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 12:20:19 +00:00
Florian Klink
fb7e2a247a refactor(tvix/nix-compat): drop useless try_into().expect()
This already has the right type.

Change-Id: I8f5850a41f9e97f1bc5f2a45ca05cf7439665c9d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9954
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-05 12:20:19 +00:00
Florian Klink
d660c5dec8 refactor(tvix/store/tests/nar_renderer): use castorepb consistently
Change-Id: I40f33044c838a7cfb9b3665fdea8e70e6886343e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9953
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-05 12:20:19 +00:00
Florian Klink
2546446d51 feat(tvix/castore): bump [Directory,File]Node size to u64
Having more than 4GiB files is quite possible (think about the NixOS
graphical installer, and an uncompressed iso of it).

No wire format changes.

Change-Id: Ia78a07e4c554e91b93c5b9f8533266e4bd7f22b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9950
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-05 10:57:01 +00:00
edef
6cec663aa1 fix(nix-compat/nixbase32): mark encode_len/decode_len const
Change-Id: Ib688bbb37cd54cfcd01e5cb3a8c376414ee8311e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9926
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-04 16:29:48 +00:00
Florian Klink
c8cc31e079 refactor(tvix/glue): move builtins into separate directory
Change-Id: I25b7197458dbfbde8623545dc0a0286eb2744f10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9911
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 15:20:46 +00:00
Florian Klink
a72a1044c2 refactor(tvix/glue): move nix_path handling to helper in tvix-glue
Change-Id: I2327560c4cf0d3f90e253e3c2f47cb29c762461e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9910
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-04 15:18:45 +00:00
Florian Klink
619f1fe88d refactor(tvix/glue): use add_derivation_builtins helper
Change-Id: I284c82612f1c8c81a8b7711fe63c19778ff2fdf0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9909
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-04 15:18:44 +00:00
Florian Klink
3196fe0143 refactor(tvix): move tvix glue code into glue crate
There's various bits and pieces in tvix-cli that use both the store and
evaluator, as well as nix-compat. For example, builtins.derivation, as
well as the reference scanning implementation.

This "glue code" currently isn't accessible from anywhere else, but it'd
be very useful if it were.

Move it out into a `glue` crate, and make `tvix-cli` a consumer of it.

All the KnownPaths setup and passing around, as well as NIX_PATH
handling is also something that should probably be moved into the glue
crate as well, but that's something left for a future CL.

Change-Id: I080ed3d1825ab23790666486840f301f00856277
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 15:18:44 +00:00
Vincent Ambo
99f618bcb4 refactor(tvix/eval): delay allocation when comparing attr values
Delays allocation (through cloning) of the values to be compared
until *after* the keys have been compared.

Change-Id: I7d68c27d7a0fbcdcc387db7c092bce50ca4b94ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9900
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-03 09:24:31 +00:00
Vincent Ambo
2dd2b844c7 chore(tvix/eval): add a marker for sorted borrowed attrs iteration
Similar to `into_iter_sorted`, add a marker function for call sites
that want *borrowed* sorted iteration.

Change-Id: I7c6f14e1ac43fdb14b861b3da183eb5d12bba139
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9899
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-03 09:24:31 +00:00
Vincent Ambo
d91fcd4f28 refactor(tvix/eval): more efficiently intersect attributes
builtins.intersectAttrs is used a _lot_ in nixpkgs eval, for whatever
reason. We previously had a very inefficient implementation that would
allocate for each comparison. It stuck out like a sore thumb in perf
analysis.

This moves to a custom algorithm with two iterators, one for the left
and one for the right side, advancing them along the (borrowed) map
keys until a match is found and allocation is required.

I've not made any effort to reduce the verbosity of this code, I don't
think it's worth it.

On my machine this reduces the mean runtime of evaluating
`nixpkgs.emacs.outPath` by ~8%.

Change-Id: Ie506d82cb8d5f45909628f771a6b73e0eca16b27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9898
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-03 09:24:30 +00:00
Florian Klink
b7ea3d7b32 fix(tvix): fix cargo test for nix_oracle
As described in https://b.tvl.fyi/issues/313, the `nix_oracle` tests
currently fail if run without Nix 2.3 in `$PATH`.

This is not a problem for people running Nix 2.3 on their systems, and
neither is it for the tests running inside Nix derivations itself (as we
hardcode Nix 2.3 there), but for interactive `cargo test` runs, either
using `mg shell //tvix:shell`, or `nix-shell` (from the tvix workspace)
it currently is broken.

Adding Nix 2.3 to the shell fixes that.

Change-Id: Idbe01aa92d63eb1a3dab5797b9be5eba1e8d7f7a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9907
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-03 09:15:27 +00:00
Florian Klink
1571d7195e fix(tvix/eval/benches): use black_box properly
The purpose of black_box is to actually prevent the compiler from being
able to optimize computation of the benchmarked function away.

To accomplish this, we need to actually *use* black_box to blackbox the
input data away, rather than the return type.

Change-Id: I5438982f57509fbf7b85034346a2739d76aef1fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9902
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-02 20:39:52 +00:00
Florian Klink
ceca00a866 chore(third_party/crate2nix): apply patches to fix cross
Together with temporarily pointing nixpkgs past
b10994c38c61038970a19fa60bfbec21a61755cc, this now fixes cross-
compilation for tvix.

This incorporates the changes proposed in https://cl.tvl.fyi/9888 and
https://cl.tvl.fyi/9889, but by fixing it in crate2nix, and using the
(re-)generated version of it.

Changes were sent upstream at
https://github.com/nix-community/crate2nix/pull/309,
this pulls in a minimal patch for now.

Change-Id: I70bb6f003bbc3e89de9c4eb4985ea4708ac3a9fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9890
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-02 09:32:54 +00:00
Florian Klink
5a0ae8312c docs(tvix/castore): add notes on verified streaming
These are still a bit WIP, but should already explain the idea better
than not having any documentation about how we intend to implement this
at all.

Let's put this into the repo, as a working document.

Co-Authored-By: edef <edef@edef.eu>
Change-Id: Ic0c5195cdf0a33c8382af1b5cff9064fe6ef9ffb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9894
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-02 09:08:20 +00:00
Florian Klink
beae3a4bf1 chore(tvix/castore): move data model docs to here
These describe the castore data model, so it should live in the castore
crate.
Also, some minor edits to //tvix/store/docs/api.md, to honor the move of
the castore bits to tvix-castore.

Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-02 09:08:20 +00:00
Florian Klink
d545f11819 chore(tvix): regenerate Cargo.nix
Change-Id: I96e01f938a46d12a94da85968caaf190d041b9ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9887
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-31 12:11:19 +00:00
edef
12fb5004d9 fix(nix-compat/narinfo): don't allocate errors early
We also switch the MissingField error to &'static str, since we only 
parse a fixed set of fields.

Together, this makes the performance impact of error handling 
negligible in batch happy-path parsing.

Change-Id: I2bd0ef2f5b35fcaced56b32d238eca75ac199ef1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9867
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-28 17:11:22 +00:00
edef
ae83998381 fix(nix-compat/narinfo): unwrap in the benchmark
We primarily want to measure the speed of the happy path.

Change-Id: Iad0146dde86fc262e2a4b8295bde4eb297b8bf30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9866
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
2023-10-28 17:11:22 +00:00
Florian Klink
8dfd8bce04 refactor(tvix/nix-compat/narinfo): return errors
This provides more info about where a NARInfo failed to parse, rather
than just returning None and leaving a library user to manually debug.

Change-Id: I9a28ddd8e5712101483ebe686fdc474c7bbc8e4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9831
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-28 17:06:49 +00:00
edef
3a77e55d2c refactor(nix-compat/nixbase32): clean up encode
Change-Id: I63da5bd47cd9033e1cc13fbe4b4864514a5343cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9865
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-28 15:41:46 +00:00
edef
2fce356ce0 fix(nix-compat/nar/writer): use ByteSlice closer to call site
Otherwise, it'll generate unused import warnings in release mode.

Change-Id: I0399d2617a8a8b973a2b8c18f4a0a613d70c48f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9864
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 17:07:06 +00:00
edef
9253bf6632 feat(tvix/nix-compat): add narinfo parsing and serialisation
Change-Id: I72c63414794642ca8d85c3f635f49db888420c40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9852
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 16:08:06 +00:00
edef
b1ad94cc9a refactor(nix-compat/store_path): use nixbase32::decode_fixed
Change-Id: I81471ee57920aa8fa889fb00c7903cdc570af9c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 16:08:06 +00:00
edef
50aa74a59c feat(nix-compat/store_path): add StorePathRef
Change-Id: I0d888a55d93e5c22e77cb0264d09757656f731d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9862
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 16:08:06 +00:00
edef
621739037f feat(tvix/castore): carry name in ValidateNodeError::InvalidName
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 13:56:51 +00:00
edef
e525272019 refactor(tvix): turn nullary enum variants into unit variants
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
520c5a191e refactor(nix-compat/store_path): speed up validate_name fast path
Change-Id: Ie50b29145804777f7644952c65cb42519a8565e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9859
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
c19c6c23cb feat(nix-compat/store_path): validate_name takes AsRef<[u8]>
Change-Id: I8819e2a7b63008a68f4f82035a08b960ac480dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
b994f692d3 feat(nix-compat/store_path): validate_name over borrowed data
Change-Id: Ifeb6231f48d4ad267a7acd398b4b3b687ee4d560
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9857
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
6238a05868 refactor(nix-compat/store_path): don't materialise fingerprint
Change-Id: I6a88531ded05c0dfb9232a0343a465fa02fb6989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9856
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
7f7c1ae7be refactor(nix-compat/store_path): make digest and name private
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
36f2b69de5 fix(tvix/nix-compat): validate store path name length
Change-Id: I89ac0ad147a1872c021ab4235ca46ef3f51d0446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 13:56:51 +00:00
edef
99a61def17 fix(tvix/nix-compat): don't box CAHash::Text
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 12:18:23 +00:00
edef
fdc2e90ef2 feat(tvix/nix-compat): add fixed-length base32 decoding
Change-Id: Iea9d51157c1e38e1734dc1ba752eba9b2372c61b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9850
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 12:18:23 +00:00
edef
3c0e42167f feat(tvix/nix-compat): use AsRef<[u8]> for base32 decode input
Change-Id: If2275dda62b852aedb64b4f9915f7dc4173cabd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9851
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 12:18:23 +00:00
edef
6a0a75c8e1 refactor(tvix): condense long bytestrings
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 07:37:26 +00:00
edef
55c37a2871 feat(tvix/nix-compat): decode base32 with a lookup table
This also takes input validation out of the loop, leaving the loop
backedge as the sole branch in the hot path.

Change-Id: Id08e6fb9cf5b074780efa09a7ad389352a601bcc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9847
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 07:31:46 +00:00
edef
67b08469db refactor(tvix/nix-compat): clean up base32 decoding
Change-Id: I8591a3e1075f2281b7dc49f7fa9e0027062a78a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9846
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 07:31:46 +00:00
edef
4d35a56798 refactor(tvix/nix-compat): simplify encode_len
Change-Id: Idfefd3b5b0d1ffbd4e9208e634a2d9989965a291
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9845
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 07:31:46 +00:00
Florian Klink
a89695c4c5 chore(tvix): move buildkite.yml from views/tvix to tvix/
This should remove the ambiguity on where buildkite.yml should show up
at least. We might still want to move workspace.josh into tvix/ too at
some point, but that's blocked on
https://github.com/josh-project/josh/issues/1287

This commit doesn't cause any change in views/tvix, with a previous
version of this commit added to my HEAD, I ran josh-filter to verify it
still points to ce0df88561fee8a4d6052ca9f2a1792f899a202a.

Change-Id: Ic03e92260dc7725bc2fcf01b7c72a097d92c028e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9828
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-25 08:22:57 +00:00
Florian Klink
6a04ba6bc5 fix(tvix/cli/derivation): fix populate_output_configuration
The `if let` wasn't matching `outputHashAlgo` being unset, and didn't
populate it in that case.

Port the remaining commented-out testcases over to nix-lang based tests.

Change-Id: I140b5643b9ed9d29f9522ec65d98d0b12262d728
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9825
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-23 14:57:32 +00:00
Florian Klink
2d99bfc7b7 refactor(tvix/cli): rename a to algo_str
Change-Id: I65d307126647841b50db008c21c155441a6b6cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9818
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-23 14:57:32 +00:00
Florian Klink
6800f1cdb9 test(tvix/cli/derivation): add test cases for output path calc
Get some test coverage with some calls to builtins.derivation.

The expected output paths were calculated with Nix.

Change-Id: I2ce205ea5244e8ef939d9cacb033283fc6f15d17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9817
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
077bf06c1d refactor(tvix/cli): move evaluator instantiation to helper
Have a private `eval` function in the test module that returns an
EvaluationResult, and migrate the existing tests over to use it, rather
than repeating itself.

Change-Id: I879987700c8507248c644ef03b62a8cb8e308139
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9816
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
6fe7684832 fix(nix-compat/src/derivation/validate): remove break
If the output was fixed, we broke out of the for loop too early, before
actually validating individual outputs.

Change-Id: I2259697dfa2a157764358f6d326a1f7f6610647c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9815
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
34fc4637eb refactor(tvix/nix-compat): rename NixHashWithMode -> CAHash
This specific struct is only used to represent content-addressed paths
(in case a Derivation has a fixed-output hash, for example).
Rename `Output`'s `hash_with_mode` to `ca_hash`.

We now also include `CAHash::Text`, and update the `validate` function
of the `Output` struct to reject text hashes there.

This allows cleaning up the various output path calculation functions
inside nix-compat/src/store_path/utils.rs, as they can now match on
the type.

`make_type` is renamed to `make_references_string`,
`build_regular_ca_path` is renamed to `build_ca_path`, and
`build_text_path` has a disclaimer added, because you might not actually
want to use it.

Change-Id: I674d065f2ed5c804012ddfed56e161ac49d23931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9814
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
833957b374 feat(tvix/store/protos): add CA field
This adds support to represent the `CA` field found in some .narinfo
files. As `deriver`, it's also a subfield of the `narinfo` field.

Extending nix-compat with a more accessible data structure that can
take care of formatting, as well as writing validation functions in Rust
+ Golang, and integrating it into nar-bridge is something for a followup
CL.

Change-Id: I71e9c30957bcd03051a491aa54d7baac25b6dd2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9795
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-22 17:04:37 +00:00
Florian Klink
9118dc8a50 refactor(tvix/store): move deriver field into narinfo submessage
This information is present in the .narinfo files, it should have gone
there.

Change-Id: Ib43d0cf30c2795bf1fe77c46646174353ade0458
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9794
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-22 17:04:37 +00:00
Florian Klink
42d3c1a748 test(tvix/nix-compat/nar/writer/sync): add tests
Change-Id: I04c556e7b57f20dd33f71b0379e6104e1702a8ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9785
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-18 12:31:57 +00:00
Florian Klink
05bca22121 feat(tvix/nix-compat/nar/reader): add tests
Change-Id: If0abaf74a3b7d201af8972e54e3e34b20b8c0bc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9764
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-18 12:31:57 +00:00
edef
e64acba454 fix(tvix/nix-compat/nar/wire): correct Node::Some doc comment
It is *eventually* followed by a Node, but there is some stuff in
between.

Change-Id: Ie7c7b462828bd3e066f4a7e774895f30b82763ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9768
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 12:22:47 +00:00
edef
2cae6a02b9 refactor(nix-compat/nar/writer): use clone_into
Equivalent to the existing code, but a little less cryptic.

Change-Id: Ib9b2f9aedddc84d0e79840bba4cce01f92d9bc56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 11:42:28 +00:00
edef
6638f4d4ea feat(tvix/nix-compat): NAR reader
Change-Id: I50d51baf62c0419eaf17f0dc262f728aaff9794d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9688
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 11:40:06 +00:00
edef
08b98b7503 docs(tvix/nix-compat/nar): document the wire format
Change-Id: I6c8e23bad27fa6ada1b8973482b4d99190cf050d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9767
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-18 11:40:06 +00:00
Florian Klink
57b69baf7c refactor(tvix/nix-compat): use hash_with_mode's nix_hash_string
HashWithMode already provides a to_nix_hash_string() method, giving us
exactly the string we need here.

Change-Id: Id2635bf3ea6c2514faf3c26b297866d774f4ff4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9799
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-18 10:32:43 +00:00
Florian Klink
1bb7e84241 docs(tvix/castore/directorysvc): document drop without put and close
As seen in cl/9761, we only use put() and close() in some cases.

Make sure this is documented, so we don't end up having implementations
not able to deal with that.

Change-Id: I406ad23f2f4c9604eca731d898ae41a4663fb846
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9786
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-17 21:12:43 +00:00
Florian Klink
718af22dbb chore(tvix/nar-bridge): update to latest go mod renames
- code.tvl.fyi/tvix/castore/protos -> code.tvl.fyi/tvix/castore-go
 - code.tvl.fyi/tvix/store/protos -> code.tvl.fyi/tvix/store-go

 See cl/9791, cl/9792 for context.

Change-Id: I44614c6ed40b9f52d9dcdea8e61fe2c3c830ce78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9793
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-17 20:46:38 +00:00
Florian Klink
2d51da7f40 chore(tvix/store-go): rename go module
`code.tvl.fyi/tvix/store/protos` now points to a directory that only
contains the `.proto` files, while all golang tooling and .pb.go files
live in tvix/store-go.

As discussed in
https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount
of people currently using this is still small, so rename the go.mod now,
while it doesn't yet hurt.

Also, use code.tvl.fyi/tvix/castore-go instead of code.tvl.fyi/tvix/
castore/protos, to make use of cl/9791.

Change-Id: I9ea89957d7c29dfae4c893b9aae8ac8a0bad2d8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9792
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 20:36:36 +00:00
Florian Klink
6e8fbc830a chore(tvix/castore-go): rename go module
`code.tvl.fyi/tvix/castore/protos` now points to a directory that only
contains the `.proto` files, while all golang tooling and .pb.go files
live in tvix/castore-go.

As discussed in
https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount
of people currently using this is still small, so rename the go.mod now,
while it doesn't yet hurt.

Change-Id: Ib3c6a2dac2923b3806ebb05be00af66d0da9f698
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9791
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-17 20:25:45 +00:00
Florian Klink
45d8a78a8b chore(tvix): move regenerate scripts to passthru
This keeps things isolated a bit more.

Change-Id: I437f2f63ee1567c0cbc02298514ad4d89f1dce85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9790
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-17 20:25:45 +00:00
Florian Klink
1b26bf21e3 chore(tvix): move store golang bindings to tvix/store-go
Similar to the castore-go CL before, this also updates the store-go
bindings to the new layout.

Change-Id: Id73d7ad43f7d70171ab021728e303300c5db71f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9788
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 19:51:24 +00:00
Florian Klink
e38733a955 chore(tvix): move castore golang bindings to tvix/castore-go
Have `tvix/castore/protos` only contain the protos, no go noise.

Make the `.pb.go` file generation a pure Nix build
at `//tvix/castore/protos:go-bindings`, and have a script at
`//tvix:castore-go-generate` (TBD) that copies the results to
`tvix/castore-go`.

`//tvix:castore-go`, with sources in `tvix/castore-go` now contains the
tooling around the generated bindings, and the generated bindings
themselves (So go mod replace workflows still work).

An additional CI step is added from there to ensure idempotenty of
the .pb.go files.

The code.tvl.fyi webserver config is updated to the new source code
path. I'm still unsure if we want to also update the go.mod name. While
being a backwards-incompatible change, it'll probbaly make it easier
where to find these files, and the amount of external consumers is still
low enough.

Part of b/323.

Change-Id: I2edadd118c22ec08e57c693f6cc2ef3261c62489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-17 19:51:24 +00:00
Connor Brewster
0325ae3ba3 fix(tvix/castore): Fix race when ingesting into castore
After finishing the ingestion, the directory putter was not being
closed. This caused a race where the root directory node was accessed
before the directory node had been flushed to the server.

This patch makes it so we close the putter before returning the root
node which should ensure that the root node exists on the directory
service server before the `ingest_path` function returns.

Fixes b/326

Change-Id: Id16cf46bc48962121dde76d3c9c23a845d87d0f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9761
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-17 13:01:29 +00:00
Florian Klink
73a19c249e refactor(tvix/nix-compat/derivation): remove some clones
Change-Id: Ie464467edfafd2050aa9e5fd0e07ff745cb15df8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9753
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-16 16:17:41 +00:00
Florian Klink
7df720855d feat(tvix): add go to shell
That's needed to build nar-bridge, as well as things in tvix/*store/
protos.

Change-Id: If5f71b6c528a27689faa3c5ab3489cc528f9e6e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9759
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-16 16:12:18 +00:00
Florian Klink
e0e3dc08a9 fix(tvix/cli): don't use protobuf from pkgs, but pkgs.buildPackages
We need to invoke protoc at build time (only).

Change-Id: Ib4c101d2ccdbdbb078725fc98c30d6c00086667d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9754
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-16 16:08:04 +00:00
Florian Klink
0f44451919 fix(tvix): fix binary building on Darwin
On Darwin, some crates producing binaries need to be able to link
against security.

Change-Id: I5bdd69247c12729b9efd5c4f18527d361ef99e87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9758
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-16 15:14:21 +00:00
Florian Klink
5ce1a03a3a refactor(tvix): push rarely-used darwin-specific hacks down
Also, make some of them buildInputs, not nativeBuildInputs.

We likely need to link against libiconv, so it shouldn't be a native
build input.

Change-Id: I757bc7f5d8c864ecaa3db26988d00b496c085c8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9757
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-16 15:09:28 +00:00
Florian Klink
6aa8a1a8ea fix(tvix/shell): fix cargo build on MacOS
It looks like we need Security Framework in the shell to be able to
imperatively run `cargo build`.

Change-Id: Ia5df9052bafca3610d4ed235b180b812d33d0d15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9756
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-16 15:09:27 +00:00
Florian Klink
652afd21b3 feat(nix-compat): add bench for aterm parsing
Change-Id: Ie9416f0046b6ee0a02096a7712cdba9cc84db4f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9733
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Gabriel Doriath Döhler <gabriel.doriath.dohler@ens.fr>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-16 12:26:36 +00:00
Florian Klink
c3fad811f4 refactor(nix-compat/derivation): drop non_unicode test
The previous CLs did already absorb all the logic into the common tests,
no need to write this here again.

Change-Id: I7ba84ba86d5445ed247e5d11d5e59b7fa815670e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9732
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-16 12:26:36 +00:00
Florian Klink
398cf8e7ce refactor(nix-compat/derivation): use ATerm for output_paths test
Due to the lack of a ATerm parser, we were previously loading the JSON
fixtures to construct our Derivation structs to run the output path
calculations with.

However, as we now have a ATerm parser, we can load the ATerm
representation directly.
This also means we can test the output path calculation for non-UTF8
Derivations.

Change-Id: I0e53f41a23566b5ad5f0fed12724e02a10b02707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9731
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-16 12:26:36 +00:00
Florian Klink
8934b34489 fix(nix-compat/derivation): handle dups
This now properly checks for duplicate output names in input derivation
output names, and duplicate input sources.

Change-Id: I0053854bfbf504f4f511fb3fe1a77a82b3aa61dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9738
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-16 12:23:00 +00:00
Florian Klink
2410f2292f feat(nix-compat/{aterm,derivation}): init parser
This provides a nom-based parser for Nix derivations in ATerm format,
which can be reached via `Derivation::from_aterm_bytes`.

Some of the lower-level ATerm primitives are moved into a (new) aterm
module, and some more higher-level ones that construct derivation-
specific types.

Also, move the escape_bytes function into there, this is a generic ATerm
thing.

Change-Id: I2b03b8a1461c7ea2fcb8640c2fc3d1fa3ea719fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9730
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-16 12:23:00 +00:00
Florian Klink
c739df3d78 refactor(nix-compat/derivation): fix function name
typo: write_enviroment -> write_environment.
Change-Id: I1a2dc826c0bf71f6ba97fe41c4d44619b5e6beff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9739
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-16 11:32:13 +00:00
Florian Klink
c50967c1f1 refactor(tvix/cli): clippy
Change-Id: I35a28a320c890f62e6a76641dfdf99e33e839c02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9749
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-10-16 11:13:33 +00:00
Florian Klink
72872785b4 refactor(tvix/store): clippy
Change-Id: I31ba268bda21c1f07c63520a269dd62f7b309e99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9748
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-16 11:13:33 +00:00
Florian Klink
1a3c610081 refactor(tvix/nix-compat): clippy
Change-Id: I56282d9754b42e0e6cdbb08e9ae71f5700d28dd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9747
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-16 11:13:33 +00:00
Florian Klink
1d97b4230f refactor(tvix/castore): clippy
Change-Id: I8a792d6b238190a78390c5d8f638669a2681f70f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9746
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-16 11:13:33 +00:00
Florian Klink
869be82e09 refactor(tvix/store/fs): clippy
Change-Id: I11247e24475f0e49dc22f2d10bf3e1e276737899
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9709
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-16 10:45:37 +00:00
Florian Klink
9aafbe8d95 refactor(nix-compat/derivation): make write_environment more generic
We don't actually care if it's a BTreeMap of strings, bstrings or any of
that sort.

The only thing we want to be able to do is get a reference to the bytes
from the keys and values.

Change-Id: I21b85811a9ea47fe06afa3108836ef9295e5d89b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9737
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-15 15:10:19 +00:00
Florian Klink
f337601f69 refactor(nix-compat/derivation): move test fixtures to subdir
We use test_resources and globbing for some of the test cases, so adding
additional files in there will also create new test cases, which we
don't always want.

Move it down one level to make some more space.

Change-Id: I619867dc80a4ced59d45096d0703678663b559cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9729
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-15 15:10:12 +00:00
Florian Klink
8a0fa69c6c feat(tvix/nix-compat/nixhash/NixHashWithMode): from_algo_mode_hash
Change-Id: Ieae628fab1926a498ae5c3eb27df1b722e6151d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9728
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-15 15:10:07 +00:00
Florian Klink
c3446da1c7 refactor(nix-compat/nixhash): add Result type alias
Change-Id: Id0248047e9642d38afc106629957a2e7608f8c78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9727
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-15 02:21:34 +00:00
Florian Klink
cb1a14334a refactor(nix-compat/nixhash): box sha512
Limit the amount of memory consumed on the stack for NixHash. Sha512
isn't used that often, so it's fine if we heap-allocate it.

Change-Id: I4a9eecd20c6184610124dc130c41bfa5d0dc04c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9726
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-15 02:21:34 +00:00
Connor Brewster
e3d72cc4cb refactor(tvix/store): Upgrade tokio-listener to get tonic support
Tonic support was added to tokio-listener upstream which removes the
need for use to have tonic compatibility wrapper types around it.

See: https://github.com/vi/tokio-listener/pull/2

Fixes b/311

Change-Id: I04a2dbb3bc3c8bfe9339583c0b46070c7ec97811
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9721
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-14 18:56:51 +00:00
Florian Klink
cc040a5ad3 refactor(tvix/nix-compat/derivation): fod_digest return [u8; 32]
This is of known length.

Change-Id: Iba48ccc486d5bf9e38ec1a2da6e7b80997d2c6ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9723
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 17:55:08 +00:00
Florian Klink
4ae0f428bd refactor(tvix/nix-compat): make NixHash an enum with fixed-len bytes
Less Vec<u8> passed around.

Change-Id: Ie153a6bfaa084d7490ffa38634efdf5f3c31a768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9722
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-14 17:55:08 +00:00
Florian Klink
786b0324a9 feat(tvix/store/protos): use Validate() function on root node
This updates the code to make use of the new Validate() function defined
on a Node.

Change-Id: I9b6ed694661f41e700f19cc78d53d2224b61852d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9718
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-14 14:21:32 +00:00
Florian Klink
2d2c4322d9 feat(tvix/store/protos): add Deriver field to PathInfo
This uses the newly introduced StorePath message type to add a Deriver
field to the PathInfo message.

Support for validation is added to both the golang and rust
implementation. This includes extending unit tests.

Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-14 13:23:07 +00:00
Florian Klink
5f8eb4eeaa feat(tvix/store/protos): add StorePath message
This encodes a store path a bit more concise, which is used in the
Deriver field.

Change-Id: Ibfb54d3b206917e51970d1d5fe94fcedb901704b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9646
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-14 13:23:07 +00:00
Florian Klink
173641ed37 refactor(tvix/store/fs): make fetch_directory_inode_data async
To make this easier, move it outside of TvixStoreFs, and accept the
DirectoryService as a function argument, so we don't need to worry about
the lifetime of self.

This also aligns with how we spawn async tasks inside the rest of
TvixStoreFs.

Change-Id: I3b95072209d32039f05aed122240f2d6db7ad172
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9713
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-14 12:34:36 +00:00
Florian Klink
9757bf6377 refactor(tvix/*store): helper for channel creation from url
This moves the repetitive code to parse a URL and create a channel
connected to it into `tvix_castore::channel::from_url`.

Part of b/308

Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 12:26:40 +00:00
Florian Klink
3f011d2762 refactor(tvix/*store): reorganize from_url
Move the channel creation depending on the string-based URL into its
own block.

Change-Id: I546b769acd2296b548eb966b62c495f910266df5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-14 12:26:40 +00:00
Florian Klink
199e5e0339 refactor(tvix/*store): remove some grpc_client let bindings
We had to have these all while the traits where sync, and there was a
lot of spawning and moving.

Most of this can now be removed in favor of some inline `.clone()`.

Change-Id: Id5466c32a403100bc3347866b3172e06a792e311
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 12:26:40 +00:00
Florian Klink
0b18be3b57 feat(tvix/castore/protos): add more granular validation methods
Similar to cl/9715, this makes the validation checks more granular,
introducing a Validate on all *Node.

A check for symlink targets is added too.

Once merged, it can also be used from tvix/store/protos.

Change-Id: I0909a89fadcd74b74ef0c9a8a1f22658fccc83b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9716
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-14 12:21:18 +00:00
Florian Klink
532f414da6 refactor(tvix/castore): simplify node validation checks
We can just check the digest length to be correct, and move the symlink
target checks to a single line.

Change-Id: I41d2e3a50e7990ef6c04f02acd754b1e17b43e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9717
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 12:09:10 +00:00
Linus Heckemann
b1ab8075cd docs(tvix/castore): point out use of contents_first
Change-Id: I7620d2abe01675ea7028a478d4f8447e36d5768b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9605
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-13 17:14:48 +00:00
Florian Klink
64beaab163 refactor(tvix/store/protos): remove unused mustBlobDigest function
Change-Id: I79a73de0cb06abcf16a6579ae298876004ab0ba8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9719
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-13 14:43:26 +00:00
Florian Klink
9b7629826f refactor(tvix/castore): factor out node checks
Implement `validate()` on `node::Node`, and call it from PathInfo's
validate() too. Node-related errors are moved to a ValidateNodeError
error type.

This additionally adds some more validations for symlink targets (they
must not be empty, and not contain null bytes).

Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-12 21:05:48 +00:00
Florian Klink
c9e90b4dd7 refactor(tvix/store/fs): use while let statement
Suggested by clippy.

Change-Id: I686b1abcb40fab42017972a3f814fae29fa70c25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9712
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-12 20:46:24 +00:00
Florian Klink
4da906bf34 refactor(tvix/store/proto): merge two match statements into one
Change-Id: I3daca008dff5527169f5916f4845234e8f3263cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9711
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-12 20:46:24 +00:00
Florian Klink
3dd5ba042b refactor(tvix/store): clippy lint
Change-Id: I03aa5c6cb0167ff6cb7312d81d1e43116fcb6d70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9710
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-12 20:46:24 +00:00
Florian Klink
04c1f151f3 refactor(tvix/castore/utils): clippy
Change-Id: Ib0fa1e1fb90d7f315f943a3f28266c0fb3714a31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9708
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-12 20:46:24 +00:00
Florian Klink
a9e5a7cec9 feat(tvix/store): enable tls[-roots] feature in tonic
This teaches tonic to speak https, so `grpc+https` becomes a working
connection string.

Change-Id: I9c550ea665b829725b545a164cf6f78587bd5213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9714
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-12 19:18:59 +00:00
edef
be32ab1eb2 fix(tvix/store/fs): fix typo
Change-Id: I132987dda1073dcc0b30e50e26682d6c8724d988
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9689
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-12 09:51:30 +00:00
Florian Klink
1c5bf366be refactor(tvix/nar-bridge): panic on inval PathInfo.NarInfo.NarSha256
Bump code.tvl.fyi/tvix/store/protos past cl/9649, where Validate()
already ensures the NarSha256 has the correct size.

Change-Id: I774668822f4d9dbd4dea47dde6e4745dc95e8e7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9665
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-12 06:24:07 +00:00
Florian Klink
e64e97ee23 feat(tvix/store): validate nar_sha256
Change-Id: I4c4dcdb75ea7748f2ab01a0bab218596b90b7b58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9648
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-11 15:29:08 +00:00
Florian Klink
8ced43f3c7 feat(tvix/store/protos): validate NarSha256
Change-Id: I016d1b020b04850f0dca68c2ea96643230dada10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9649
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-11 15:29:08 +00:00
Florian Klink
6dab608198 docs(tvix/nar-bridge): remove redundant comments
These really only describe what the code is doing, remove.

Pointed out in
https://cl.tvl.fyi/c/depot/+/9652/comment/987b3b7f_c7b5e47b/

Change-Id: I512a274295244f96744a489871861fed9523bb37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9654
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-11 15:22:22 +00:00
Florian Klink
155ab77cff refactor(tvix/nar-bridge): use mh.SHA2_256
As correctly mentioned in
https://cl.tvl.fyi/c/depot/+/9652/comment/03b9b96e_bbb337fd/,
we shouldn't be using these magic constants, but pull them from where
they're defined.

This already is a dependency of go-nix, and pkg/pathinfosvc/server.go,
so no changes in go.mod.

Change-Id: I0cc41ce040fcbddf4b6171417bc9b0de55af4991
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9653
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 15:22:22 +00:00
Florian Klink
fb04645b0d docs(tvix/store/protos): explain a bit more what we're doing
Explain a bit better what we do in the codeblock below,
and add a newline to make it clear we don't just document the first
branch.

Change-Id: Ifb142d9984e6d2cbca648525c10298f2fcbdfedf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9645
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
2023-10-11 11:43:29 +00:00
Florian Klink
673f5febbb feat(tvix/nar-bridge): stop parsing nixbase32 manually, validate
We have nixhash.FromHashTypeAndDigest now.

Also, run Validate() on the PathInfo received from the remote
PathInfoService.

Change-Id: I14db0d9356c539c084afc9dd712314b56da2587e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9652
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 11:41:50 +00:00
Florian Klink
98c17147c6 refactor(tvix/nar-bridge): have Export return root node
… and nar size / sha256 digest.

Instead of producing sparse PathInfo messages when NARs are sent to
nar-bridge, the nar-bridge http server now keeps a lookup table
(narsha256) -> (rootNode, narSize)

This removes a whole bunch of noise, because we don't need to keep
sparse fields around.

A convenience function
`GenPathInfo(rootNode *castorev1pb.Node, narInfo *narinfo.NarInfo)` is
added, which is used to produce PathInfo messages, either when receiving
a NAR file over http and uploading it to a remote PathInfoService, or to
synthesize the PathInfoMessage to return to the client, if nar-bridge is
acting as a PathInfoService for a remove Nix HTTP Binary cache.

Change-Id: Ibba1ab6238a050816c4fab29cb21ae88877d8613
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9651
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 11:41:50 +00:00
Florian Klink
ceb1674e9f refactor(tvix/nar-bridge): pass root node to storev1pb.Export
Bumps the go module past cl/9604 and update the consumer side.

Change-Id: Id44245017f1dc2f8aac28051cdbb45b83bdc5be3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9650
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-11 11:41:50 +00:00
edef
930edb69d7 fix(tvix/castore): use bstr for formatting names in errors
Much friendlier than either Base64 or raw byte slices.

Change-Id: I9b4cdd57c83ddc76c0be8103da4320207657a72b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9622
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 22:40:55 +00:00
edef
baae5ce473 fix(tvix/castore): handle Directory::size overflow explicitly
We use checked arithmetic for computing the total size, and verify
that size is in-bounds in Directory::validate.

If an out-of-bounds size makes it to the "unchecked" size method,
we either panic (in debug mode), or silently saturate to u32::MAX.

No new panic sites are added, since overflows in debug mode already
panic at the language level.

Change-Id: I95b8c066a42614fa447f08b4f8fe74e16fbe8bf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9616
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 20:33:52 +00:00
edef
e2dba089c4 test(tvix/castore): demonstrate that Directory::size may panic
Change-Id: Iebfa12df51c8c9d39b4d9eb2fccc6ffa312639cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9620
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 20:33:52 +00:00
edef
d38e6502ad fix(third_party/crate2nix): run tests in debug mode
Behave like `cargo test` rather than `cargo test --release`.

Change-Id: Ie013d04ac68d7dec2a3b870fa9f0060a70a9635d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9621
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 20:33:52 +00:00
Florian Klink
d94749ac22 refactor(tvix/store/protos): have Export accept root node
We don't need the full PathInfo message, only the root node.

Change-Id: I667045ed766875dfbf8ac126a50b02baa2df67a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9604
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-10 20:22:47 +00:00
Florian Klink
fe963ae0a3 chore(tvix/nar-bridge): move to Export from storev1pb
This removes the Export method in nar-bridge, and updates all users to
the version now in storev1pb.

It moves the roundtrip test to the importer crate, and some of the
utility functions into a separate util_test.go file.

Change-Id: I81d9e0b35dfd78ef1042bed307281eecd2aaa2a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9603
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-10 19:08:29 +00:00
Florian Klink
e6ba84ea50 feat(tvix/store/protos): add Export
Export will traverse a given PathInfo structure, and write the contents
in NAR format to the passed Writer.
It uses directoryLookupFn and blobLookupFn to resolve references.

This is being moved over from nar-bridge. We need to keep the code there
around until we can bump go.mod to storev1 with this merged, but the
tests can already be moved entirely.

Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
2023-10-10 18:55:49 +00:00
Florian Klink
6fe34b7ba0 refactor(tvix/nar-bridge): use castorev1pb.RenamedNode
We can use the helper to rename the node.

Change-Id: Id8defea7e5ebbd43d7b7a9b2992c62084e1828ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9601
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-10 18:02:38 +00:00
Florian Klink
2699b61052 chore(tvix/nar-bridge): bump google.golang.org/protobuf
… to 1.28.1 as well.

Change-Id: Ieaaf9363a8dd63c2437c34d78a0ab369e7492f19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9600
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 18:02:38 +00:00
edef
d23fe6ee20 feat(tvix/store): use tvix_compat::nar::writer::async
Change-Id: Iad36872244df6f2225a2884f6b20cacd8f918b31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9619
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
2023-10-10 17:33:33 +00:00
edef
8b35d97b4b feat(tvix/nix-compat): add nar::writer::async
Change-Id: Ib962f889836659fd278beec461ee329b00e64ea5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9615
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 17:13:27 +00:00
edef
f38290cfb0 refactor(tvix/nix-compat): move nar::writer to nar::writer::sync
This is preparation for adding an async port.

Change-Id: Id638ec1f6f46e2f3935448184eed51e2233263fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9618
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 17:13:27 +00:00
edef
bb54e04c1b fix(tvix/nix-compat): clean up the debug assertions a bit
Consistent error messages, and slightly nicer code layout. We avoid
printing the input data, since we primarily want to point out the
specific violated invariant. In the one place where we do want to,
we use BStr's Debug implementation, since byte slices don't print
nicely.

Change-Id: I3a9a0c37be270ea5f16cf124922c254608fb849e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9617
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
2023-10-10 16:58:17 +00:00
edef
17d6f4320f fix(tvix/nix-compat): drop unnecessary reference taking
Found by Clippy, which we should probably run in CI.

Change-Id: Id79c30b63f681021ab79358e02d29454d43c0aa6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9614
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 08:57:04 +00:00
Florian Klink
d1bccfe82f chore(tvix/*store): use google.golang.org/protobuf
The other one is deprecated, according to
https://pkg.go.dev/github.com/golang/protobuf/proto.

Change-Id: I502d5bfa7431df0dea3d564f1e0ecfea14c90e10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9597
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:07 +00:00
Florian Klink
abf9c05090 feat(tvix/castore): add RenamedNode helper
This takes a castorev1pb.Node and updates its name, making sure to copy
it.

Change-Id: I57bf2b2b4a74496a1a697bcbce72bf67a12e69fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9594
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:05 +00:00
Florian Klink
78abeaa920 refactor(tvix/nar-bridge): use storepath for store paths
Remove the handwritten parsing and formatting of store paths.

Change-Id: Ia4ba486b4363c33b98937bcbf6f5f7bcda289b82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9588
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-09 22:05:02 +00:00
Florian Klink
28cd4b1a2f feat(tvix/nar-bridge): add ToNixNarInfo()
Convenience function, moves all code converting from a PathInfo struct
to to go-nix's NarInfo.

Change-Id: Idf0dcc38675674563f2dfd3286a4a55fa2a24a82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9593
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:02 +00:00
Florian Klink
92481825b3 test(tvix/store/protos): add go tests for Validate()
We already have validation tests for Rust, let's add the missing ones
for golang too.

Change-Id: Iaf3a3e1ee72d5647da3f2aa977d6e0d0379b2ce5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9595
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-09 22:01:59 +00:00
Florian Klink
67ccbf8c94 fix(tvix/castore-protos-go): fix src
This was unfortunately using ./castore/protos all the time, so not
testing castore at all.

Update src to make sure it does get built.

Change-Id: I563910c259ffca6aee7f964532bd39c4bcf7aed8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9596
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-09 22:01:58 +00:00
Florian Klink
87396d29fd chore(tvix/nar-bridge): bump go-nix
Change-Id: I72d139c47ede56d63d896bb5d5ed385e22a32264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9592
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-09 19:08:28 +00:00
edef
5cc7171b5f docs(tvix/nix-compat): mention Directory::close on Node::directory
Change-Id: I19e6a3b63eb1b1609b96544b2529d71234bcb13b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9613
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-09 18:39:57 +00:00
edef
ca1ede17cb docs(tvix/nix-compat): document the name ordering requirement
Change-Id: Ia2c5f0b357a23d4bfac1fd52cf01845c78ebfa22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9612
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-09 18:39:57 +00:00
edef
49b106b0cb fix(tvix/nix-compat): don't transiently allocate names
The point of clearing and reusing the same Vec is to avoid transiently
allocating for every directory entry. This was lost in cl/8974 when we
switched from String to Vec<u8>.

Change-Id: I65647e5c4e54e88f1fe45e9a752cb5154d98fb33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9607
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-09 18:22:54 +00:00
Florian Klink
dba62157e5 test(tvix/castore): add uds pingpong test for directorysvc
This was missing so far.

Change-Id: Ic8c7781aa0e101342e8a7d94f206937b4163ebfa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9570
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 22:55:10 +00:00
Florian Klink
a464088d2e test(tvix/store): use tokio-retry for exp backoff
Rather than using this loop, use exponential backoff while waiting for
the socket path to be created.

Change-Id: I3056b1525784cd712b1d81f84876c9ca0be10dc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9569
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 22:50:42 +00:00
Florian Klink
b6bf3a87f1 test(tvix/castore): use tokio-retry for exp backoff
Rather than using this loop, use exponential backoff while waiting for
the socket path to be created.

Change-Id: I18706a64ce06f8916a07892dfbcd409ac5b3bff1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9568
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 22:50:42 +00:00
Florian Klink
c0376995c9 refactor(tvix/store): simplify test_valid_unix_path_ping_pong
We don't need to spawn two tokio runtimes anymore, and can do the URL
parsing at once, too.

Change-Id: I7885a894bb1250cd087d4e1893e3e73b631331da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9567
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 22:47:38 +00:00
Florian Klink
b196cbbc67 refactor(tvix/castore): simplify test_valid_unix_path_ping_pong
We don't need to spawn two tokio runtimes anymore, and can do the URL
parsing at once, too.

Change-Id: I38ab96978cb7f8c31ded2726262e0b1366655094
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9566
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 22:47:34 +00:00
Florian Klink
269ab866f1 refactor(tvix/eval/tests): migrate to tempfile
tempdir pulls in remove_dir_all 0.5.3 with
https://rustsec.org/advisories/RUSTSEC-2023-0018.html, and we use
tempfile everywhere else too, so let's just migrate to that.

Change-Id: I735ade7b65e12fc26e3d43ca95fcfa07fcc64642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9565
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-08 22:47:33 +00:00
Florian Klink
4b27cdbc57 docs(tvix/store/bin): drop outdated TODO
We already use store_path::build_nar_based_store_path as a helper
function down there.

Change-Id: I46842bb0a2f4cd5de7dcfc3c4b6aae399618667a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9571
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 22:41:32 +00:00
Florian Klink
c63c55f052 chore(tvix): bump tokio[-util], tonic, prost
This bumps all these dependencies to their more recent versions.

We have to enable the `macros` tonic feature, and looks like we should
also enable `rt`, not only `rt-multi-thread`.

Change-Id: Icc3600848fca0bacce24f4889d088e75711594ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8984
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 21:53:20 +00:00
Florian Klink
c847cc32d9 refactor(tvix/castore): move tests to grpc client, rm tonic-mock
Similar to gen_directorysvc_grpc_client, introduce a
gen_blobsvc_grpc_client function that provides a gRPC client connected
to a blobservice.

The test is update to use that client to test against, rather than the
server trait, removing the last usage of tonic_mock, so it's removed
as well.

Fixes b/243.

Change-Id: If746e8600588da247eb53a63b70fe72f139e9e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9564
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 21:53:20 +00:00
Florian Klink
1629f3064f docs(tvix/castore): remove TODO
This probably was about passing around directory_putter at some point,
which we do, so whatever this meant, it's not actionable anymore.

Change-Id: I1b4e0cdd2119bf2b2a9cf06d186a3b476b0ff367
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9573
Reviewed-by: Linus Heckemann <git@sphalerite.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-08 16:00:44 +00:00
Florian Klink
b172c804b0 refactor(tvix/castore): use DuplexStream instead of unix socket
We can use DuplexStream to create to bidirectional pairs, which avoids
manually waiting for unix sockets to pop up and connect, and creating
temporary directoires to create the unix sockets in.

Turns out, we also don't actually need to spawn the server in a separate
runtime, it works just fine these days. This might be due to all the
sync barriers in between being gone.

Change-Id: I6b79823bc6209cbcb343b7a498c64a2ba6e0aee7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9562
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 11:25:23 +00:00
Florian Klink
09328c79d5 feat(tvix/store/listener): implement connect_info()
For Unix and TCP streams, we can actually give more info than just ().

Change-Id: I80ca261b9e7a880389ec2029ea5893180314f0b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9561
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 07:44:13 +00:00
Florian Klink
3b620e6d0c refactor(tvix/castore): move gRPC directorysvc tests off tonic_mock
The tests were using the gRPC Service/Server interface, not the gRPC
clients.

Use our new gen_directorysvc_grpc_client() function to spin up a server
in a separate thread, and test via that client, rather than *just* the
server implementation.

This is more correct, because we're now really exercising the gRPC
stack, and some of the stream implementations are also only implemented
on the client.

Change-Id: I99434a2399856a44c9e6988d0b720f7a429d0ad1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9560
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 07:44:13 +00:00
Florian Klink
a77914db73 refactor(tvix/castore/directorysvc): factor out gRPC client gen
Move this code into a helper function, which we'll use in other places
in a bit.

Change-Id: Icae6f6dd2d4b2fa86fd2b836ddd7a4ca0e0354e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9559
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 07:30:34 +00:00
Florian Klink
31f28b6105 chore(tvix/store): drop tonic-mock from dependencies
It's not used in here.

Change-Id: If1dbf360ec37b649c27d3a9bf11f7bf0ca947631
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9558
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-08 07:17:54 +00:00
Florian Klink
03fec0b393 refactor(tvix,views/tvix): move shell into separate file
So far, we provided a custom `default.nix` in the root of the tvix josh
workspace, which re-defined the shell attribute from `tvix/default.nix`.
Some of the recent fixes, e.g. the MacOS-specific additions to the list
of dependencies however didn't get ported over to this file, and in
general, it's quite annoying to have two different places for these
things.

Initially I explored the idea of moving this default.nix file to a
default-depot.nix file in the josh worktree only, and then "polyfill"
some of the dependencies, or set up readTree in the josh workspace too,
but it turned out to pull in too many dependencies to be worth
the effort (nix.sparseTree, tools.depotfmt, crate2nix overlay,
third_party.gitignoreSource).

I now took a different approach - moving the definition of the `shell`
attribute from `tvix/default.nix` to its own `shell.nix` file, which is
imported from `tvix/default.nix` in regular depot usecases.

Josh workspace consumers only see the `shell.nix`, which can be used
in a self-contained fashion, the other `default.nix` is gone entirely,
and we update the workspace file to also not show `tvix/default.nix` at
the root either, so running `nix-shell` and then `cargo build` should
still work.

Change-Id: I6cb54d45d150c597612530ba44bc578f9d7f9120
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9556
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-07 05:47:31 +00:00
Connor Brewster
9019d8568e fix(tvix/store): Remove virtiofs from default features
vhost-user-backend doesn't support macOS yet, so the virtiofs features
will not work on macOS. This removes it as a default feature which makes
`cargo build` work out of the box on macOS.

The `virtiofs` feature is enabled for Linux when building via Nix, but
if being built by cargo directly, the feature must be enabled via a
cargo flag.

Change-Id: I2aaca9582f8e3dbcf9ee5f1b9831d614909f3799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9555
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-10-07 02:34:00 +00:00
Florian Klink
9c0d66197b feat(tvix/nar-bridge): do pathInfo.Validate() as additional check
This should make it quite quick to spot writing code breaking some of
the assumptions we have on PathInfo messages ourselves.

Change-Id: I480caaec41f8ea5246c3c3081460c7ad12e78569
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9554
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-07 02:06:55 +00:00
Florian Klink
7e9da4ddef fix(tvix/nar-bridge): properly check error during HTTP shutdown
We didn't bind err := here.

Change-Id: I779d7ba5660ba4b126a4a0db7fe044bf94425855
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9547
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 19:15:00 +00:00
Florian Klink
96c0b3f069 refactor(tvix/store/proto): use NamedNode trait
This saves us writing the name parsing code three times. We can also
delay parsing until we did other (cheaper) checks.

Change-Id: I1abe3f20dba4215b38839cf7466297e028d64656
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9548
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 18:58:55 +00:00
Florian Klink
a2c81de4cb docs(tvix/nar-bridge): fix comment
We run narInfo.Check to ensure this parses to a StorePath, not
nixpath.Check.

Change-Id: Id91183128df74a60d98fa2a31174cd879194c34d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9550
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 18:51:53 +00:00
Florian Klink
61e5a63fe1 chore(tvix/nar-bridge): bump go-nix dependency
nixpath.FromString -> storepath.FromAbsolutePath.

See https://github.com/nix-community/go-nix/pull/113 for details.

Closes: https://b.tvl.fyi/issues/314
Change-Id: I25277fb6006cbbb2a323ffb5809a1be500822a97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9551
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 18:47:52 +00:00
Florian Klink
600815c168 feat(tvix/store/protos): add Validate() method to pathinfo.go
This is very similar to the Rust counterpart.

Change-Id: I40d51aaac3fcf7f52e5896587e561bc2377f6269
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9549
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 18:42:58 +00:00
Vincent Ambo
ddbb25e145 feat(tvix/website): add structured data for search engines
This closes b/303.

Change-Id: Icaeccba23a2590c22106bed6e859ed3188c87a02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9540
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-10-05 17:14:13 +00:00
Florian Klink
cb807ad79b refactor(tvix/nar-bridge): do root directory dgst check in uploader
This check makes more sense there, and gives stronger semantics - Done()
only succeeds if the other side successfully received everything, *and*
came up with the same hashes as we did.

Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 15:02:12 +00:00
Florian Klink
c04041a001 feat(tvix/nar-bridge): add nar-bridge-pathinfo command
This adds an additional nar-bridge-pathinfo command.

It exposes a PathInfoService for a HTTP Binary Cache, ingesting data
into a BlobService/DirectoryService as it goes through the NAR file.

It does this whenever it receives a Get request for a specific output
path, and waits returning with the PathInfo response until it ingested
the data.
It does not do any sort of caching - this means it re-downloads NAR
files again whenever the PathInfo is requested again, so you most likely
do not want to use this currently.

It's one building component as soon as we have store composition (which
we currently don't, so don't use this).

It can be used as an alternative mechanism to ingest data (Blobs and
Directories) of a given store path from a binary cache into tvix-store.

```
❯ nix-build -A third_party.nixpkgs.hello
/nix/store/mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1

❯ nix hash to-sri --type sha1 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6
sha1-Rs/INeK+7IGbG/u7fHoVNm96Yqs=

❯ out=$(mg build //tvix/nar-bridge)
$out/bin/nar-bridge-pathinfo --log-level debug &
INFO[0000] Starting nar-bridge-pathinfosvc at [::]:8001

❯ mg run //tvix:store -- daemon &
[mg] building target //tvix:store
[mg] running target //tvix:store
  2023-10-03T16:21:57.433739Z  INFO tvix_store: tvix-store listening on [::]:8000
    at src/bin/tvix-store.rs:229

❯ evans --host localhost --port 8001 -r repl
[…]
tvix.store.v1.PathInfoService@localhost:8001> call Get
✔ by_output_hash
by_output_hash (TYPE_BYTES) => Rs/INeK+7IGbG/u7fHoVNm96Yqs=
{
  "narinfo": {
    "narSha256": "sXrPtjqhSoc2u0YfM1HVZThknkSYuRuHdtKCB6wkDFo=",
    "narSize": "226552",
    "referenceNames": [
      "aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8",
      "mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1"
    ],
    "signatures": [
      {
        "data": "7guDbfaF2Q29HY0c5axhtuacfxN6uxuEqeUfncDiSvMSAWvfHVMppB89ILqV8FE58pEQ04tSbMnRhR3FGPV0AA==",
        "name": "cache.nixos.org-1"
      }
    ]
  },
  "node": {
    "directory": {
      "digest": "xvo6BYbYaDw76IibLu5sr+VZoj9iM0ET2RUuYSYLwKE=",
      "name": "bWRpN2x2cm4ybXg3cmZ6djNmZHEzdjV5dzhzd2lrczYtaGVsbG8tMi4xMi4x",
      "size": 141
    }
  },
  "references": [
    "ptgFMIhdl2nJxMDdlDkITyXuBFc=",
    "Rs/INeK+7IGbG/u7fHoVNm96Yqs="
  ]
}
```

Change-Id: I50167d0ac081c91adf5cf2733bbc4dc0993bd46e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9539
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Brian Olsen <me@griff.name>
2023-10-05 15:02:12 +00:00
Florian Klink
d45d6de561 test(tvix/store/pathinfo): add more tests for references
This should cover all error cases produced.

Change-Id: If31816d9b087551d86d7913df55df8f9f44bb554
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9546
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-05 10:25:16 +00:00
Florian Klink
1f03a520a9 feat(tvix/store): add validation for references
This validates the size of reference digests in the PathInfo message,
as well as inside the narinfo submessage. If narinfo is set, they need
to parse to StorePath, and have the same digest there as in the PathInfo
message.

`proto::tests::pathinfo::validate_references` needed to be updated,
because we actually did not populate the proper references before.

Change-Id: I9545b2487aab9fe0d229c26aceba5ddc5e6daafd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9545
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-05 09:52:19 +00:00
Florian Klink
7706a8f224 refactor(tvix/store): mv *Err*::Invalid{Node,}DigestLen
There's other digests in the PathInfo structure, that also might have
wrong digest lengths. Rename this to give some room for them, and update
the error message a bit as we go.

Change-Id: I06562664721156e658f2ed14ba1de907377d284b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9543
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-05 09:52:13 +00:00
Florian Klink
c67ab911eb refactor(tvix/castore): move magic number to B3_LEN const
… and export it.

Change-Id: I47d2dc2f5a8174da65c614b43801d648506e2d73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9544
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 09:50:31 +00:00
Florian Klink
8239f32b62 refactor(tvix/nar-bridge): mv nar-bridge{,-http}
Rename the nar-bridge CLI to nar-bridge-http, because it's the one
spinning up an http server.

Change-Id: I0fb75c50e4299272a128dd5ecaa4be8f06fa3dbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9538
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
c18ff1a270 refactor(tvix/nar-bridge): deduplicate NAR HEAD and GET
Use a genNarHandler() function accepting a boolean to construct the
HTTP handler.

Change-Id: I17c054826d91a9dbed8b1f53945a51f27fa60ace
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9537
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
45511004df docs(tvix/nar-bridge): fix typo in comment
Change-Id: I74f6892e089ed306ef93e78a39a25cf3738bb09f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9536
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
28d1b9c01d refactor(tvix/nar-bridge): move pkg/server to pkg/http
This is only dealing with the HTTP interface.

Change-Id: I011b624fd9f11ea96231b92fea1166c118a219f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
0353108e99 refactor(tvix/nar-bridge): move DirectoriesUploader to pkg/importer
This is useful outside a HTTP server scenario.

Change-Id: I0042a6e773906a15a254d850520e6f841035bf20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
b78b8d8338 refactor(tvix/nar-bridge): move blob cb function to pkg/importer
This is useful outside a HTTP server scenario.

Change-Id: If35f1ab245855378fd01f16ad7b5774d0cf590ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9532
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
259269482c docs(tvix/nar-bridge): add more comments for DirectoriesUploader.
Change-Id: I6684186f3120ea42036239baa82e0ca37d3e0e6c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9531
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
d3c44ab03d refactor(tvix/nar-bridge): rename writer package to exporter
Change-Id: I7cf6a3ed0dca4c450fadd9228dc0154ad6254f51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9530
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
49b427d773 refactor(tvix/nar-bridge): combine writers/readers
We can drop most of Hasher if we use a MultiWriter writing to the hash
function and a minimal CountingWriter.

This should make things a bit more understandable.

Change-Id: I37ee72d9a5c73f253aecc1ad761cb723389b89fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9529
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
f92b0ef933 refactor(tvix/nar-bridge): let callbaks return calculated digests
This aligns behaviour more with how it should be - it's the
responsibility of the callback functions to return digests of the things
they consume(d). It allows further cleaning up the hasher struct.

Change-Id: I9cbfc87e6abd4ff17fadf39eb6563ec3cb7fcc6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9528
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
b1ff1267be refactor(tvix/nar-bridge): drop reader package
Make the import function usable on any reader.

Change-Id: I84d2004cb73cdd7a11fe8efb0f2efb6335d5e6b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9527
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:11:27 +00:00
Florian Klink
6e9a5dcd59 refactor(tvix/nar-bridge): rename item to stackItem
Change-Id: I76c8db3b04d9f97e51e91f7d36a4dd8c6500d1a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9526
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:11:23 +00:00
Florian Klink
174485038c refactor(tvix/nar-bridge): inline assemblePathInfo
This is only called once.

Change-Id: I342443b8d04050929733fc84d5f36cd64060afe3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9525
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 06:11:23 +00:00
edef
155f53b264 fix(tvix/nix-compat): reject dotfiles
Nix has historically rejected these. The current behaviour was
accidentally introduced in Nix 2.4, and is considered a bug.

Link: https://github.com/NixOS/nix/pull/9095
Change-Id: I38ffa911f0a413086479bd972de09671dbe85121
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9507
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
2023-10-04 17:18:11 +00:00
edef
9a7c078a69 fix(tvix/castore): explicitly name lifetimes in process_entry
Otherwise this produces absolutely inscrutable errors:

    note: hidden type `[async fn body@castore/src/import.rs:63:1: 63:94]` captures lifetime '_#24r

Change-Id: If5d9626c9edf400de5bcec038bcaa5a3117561f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9506
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-04 08:31:40 +00:00
edef
e014c3ef70 fix(tvix/nix-compat): bits are not bytes
Change-Id: Ib5744ea240895fe84454ad271ee5640b29cab838
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9505
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-03 08:09:28 +00:00
Brian Olsen
cfb810d81a fix(tvix/store): Fix FUSE support on MacOS
This partially fixes b/312 and gets FUSE to work again on MacOS.

It is mostly small type changes and an update to fuse-backend-rs because
upstream currently doesn't work with MacFuse. It also sets the default
FUSE thread count on MacOS to 1 because otherwise the mount command will
hang when shutting down as only one thread gets ENODEV and all the others
just keep blocking.

Change-Id: Ifb3c4268caf296c487049c1dc4618acb32497f44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9490
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-02 15:46:47 +00:00
Brian Olsen
5c2cad0ac4 fix(tvix/store): FuseDaemon unmount is blocking async runtime
The unmount method in FuseDaemon calls join on a bunch of threads and that is
a blocking call but it is called from an async context in the tvix-store
binary.

This change wraps the call to unmount in a spawn_blocking.

Change-Id: If89183b4a3f890874e75f5faf90cd24cb18da1e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9489
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-02 12:50:16 +00:00
Connor Brewster
053e41f4e5 docs(tvix/boot): Update command to reflect new tvix/boot directory
This used to be in tvix/test but was moved to tvix/boot, but not all the
command were updated so they no longer work as documented.

This updates the commands to use the new tvix/boot directory.

Change-Id: I8658d928a152140df182c159bf98ea9d9ecd1302
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9482
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-28 04:24:08 +00:00
Florian Klink
89a8306ded fix(tvix/store): rename Read method in Readme
This call was renamed to `Read` a while ago, but I forgot to update
the README.

Change-Id: Ibcb97fe2520a5e9bb8444ec1adade6890dda4a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9476
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-26 15:24:32 +00:00
Adam Joseph
fac32d0a5f feat(tvix/eval): test case for b/281
This commit adds a test case for b/281.

Change-Id: I8dfbfc0ff636184d7882530d8aefb329a3af9e5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9288
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-09-26 14:13:05 +00:00
Florian Klink
d8f5c5b109 fix(tvix/store): allow building with Nix on Darwin
Curently both fuse and virtiofs features fail to build on Darwin.
Disable them for now until fixed.

Change-Id: Ic99fa5e3ebec8a7072ba884101a5c18652ed409c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9473
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-09-26 10:07:41 +00:00
Florian Klink
6b02800e48 chore(tvix/[ca]store): allow building without tonic-reflection
- rename the feature to `tonic-reflection` in both tvix-store and tvix-
   castore.
 - set it to disabled in tvix-castore by default
 - enable it in tvix-store, and pull in tvix-castore with the feature
   enabled.

Change-Id: Ie22833d85569502cae55812f6eeb17a9c15b9e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9472
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 10:07:40 +00:00
Florian Klink
329067f629 fix(tvix/cli): don't enable default features for tvix-store
These enable fuse and virtiofs features, which fail to build on MacOS.

tvix-cli needs neither of these to be built, so we can disable them.

Fixes b/240 (tvix-cli build on Darwin)

Change-Id: I991d947f31d0185aedd1c8a341f714f4eedd03c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9474
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-26 10:03:38 +00:00
Florian Klink
7062fdc132 chore(tvix/store): only depend on vhost-user-fs if virtiofs enabled
Change-Id: Iaf88d43df27b87548794a8dddc9a1e08279b8ffb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9471
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-26 10:03:37 +00:00
Florian Klink
10717605a2 chore(third_party/crate2nix): bump crate2nix to current HEAD
This will bring conditional features support.

Also invoke crate2nixgenerate with the --all-features argument, so all
dependencies, including the ones for optional features are included in
the Cargo.nix file.

Change-Id: I3bbcb200c9b481f660db89efba650ea4f7418a63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9470
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-26 10:03:36 +00:00
Florian Klink
124c1c06de chore(tvix/store): move fuse-backend-rs back to upstream
It's in no release yet, but this is the current rev in master.

Change-Id: I9e4bc2fde26e85c4182bd66b3e566c2d957072ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9469
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-09-26 09:51:02 +00:00
Connor Brewster
f8746fcab0 feat(tvix/store): Support listening on UNIX domain sockets
This adds support for listening on UNIX domain sockets via the
tokio-listener crate. The crate will automatically determine whether to
start a TCP or UNIX domain socket server based on the listen address.

Unfortunately, it's not compatible with tonic right out of the box so I
added some wrapper types to implement the necessary traits to make
things work. We should investigate upstreaming a `tonic` option to the
tokio-listener crate which implements the relevant `tonic` traits.

Example:
```
$ tvix-store daemon -l /run/tvix-store.sock
INFO tvix_store: tvix-store listening on /run/tvix-store.sock

$ tvix-store mount -l /mnt/tvix --blob-service-addr grpc+unix:///run/tvix-store.sock --directory-service-addr grpc+unix:///run/tvix-store.sock --path-info-service-addr grpc+unix:///run/tvix-store.sock

$ ls /mnt/tvix
```

Change-Id: I91c4a4b0c5a177b3b90e6c01a4e5d263130e6bdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9429
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-25 17:41:37 +00:00
Florian Klink
1b3d6975ed chore(tvix/tests): rename to //tvix/boot
This is mostly boot tooling, the integration test is just one instance
making use of it.

Expose initrd, kernel and runVM as a separate target to CI, and move the
tests to a subdirectory.

Change-Id: I1d22cd68bf5af095bc11dd9d7117b62956c7f7f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9465
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-25 12:46:10 +00:00
Florian Klink
242949ecfb fix(tvix/tests/tvix-init): run as pid1, support init=
We were previously not running as PID1, but were executing things via
uinitcmd (by u-root init).
This had some problems - a real system pid1 really wants to be pid1.

u-root init itself seems to be too complicated to configure as a
chainloader, so instead, we just provide init/pid1 ourselves and do the
little bit of mount setup / stage 1 that's necessary.

`tvix.exec` is renamed to `tvix.run`, to further distinguish it from the
`exec` bash function.

Our init will now also power off the machine if the shell, or the custom
tvix.run command finished. In the case of switch_root (`init=` being
passed), we terminate before, so it's up to the real system anyways.

Change-Id: If00c0c11ca9393968e76ae0d131594b5c135fa48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9454
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-25 10:49:33 +00:00
Adam Joseph
05f42519b5 fix(tvix/eval): fix b/281 by adding Value::Catchable
This commit makes catchable errors a variant of Value.

The main downside of this approach is that we lose the ability to
use Rust's `?` syntax for propagating catchable errors.

Change-Id: Ibe89438d8a70dcec29e016df692b5bf88a5cad13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9289
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-09-24 21:54:10 +00:00
Adam Joseph
926459ce69 refactor(tvix/eval): factor CatchableErrorKind out of ErrorKind
This commit creates a separate enum for "catchable" errors (the kind
that `builtins.tryEval` can detect).

Change-Id: Ie81d1112526d852255d9842f67045f88eab192af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9287
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-09-24 21:54:09 +00:00
Florian Klink
6015604bd8 docs(tvix) sync readme and website component listing, extend
These existed with slightly similar descriptions in both places, and
castore and nar-bridge and castore were missing entirely.

Change-Id: I4794c18665e25ee1f812975b526ff27ce197d0af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9453
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-09-24 21:45:13 +00:00
Florian Klink
abef229791 chore(tvix): add /*.sled to .gitignore
The README suggests putting state here for some examples, so let's make
sure it doesn't cause a dirty worktree.

Change-Id: I3a191f5b2681ef6937a03d3a9686ae0d7aebd76e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9450
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-24 17:05:48 +00:00
Florian Klink
ba3edc5638 feat(tvix/tests): add basic plumbing for integration tests
This adds a basic skeleton to test various tvix components in
combination.

As a first start, we test the virtiofs integration, by seeding the tvix-
store with some data, booting a VM, and listing the contents from there.

Change-Id: Ice6516d0eb508d19ea8fabefee860a0950046171
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9449
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-24 17:05:48 +00:00
Florian Klink
93071ca450 feat(tvix/nar-bridge): provide a default for store-addr
Defaulting to where `tvix-store daemon` listens to is a sane choice.

Change-Id: Ic52fa856c5708e0e1a8d51618f8c9ad1894fd28f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9452
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-24 17:05:48 +00:00
Florian Klink
ae9f4c7de8 refactor(tvix/nar-bridge): rename binary to nar-bridge
nar_bridge is an odd name for the binary.

Change-Id: I761ec5f986bde2f7e50e5a0c0b6182164a6cdc7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9451
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-09-24 17:05:48 +00:00
Connor Brewster
993c505cdb feat(tvix/store/fs): Add support for virtiofs backend
This adds a virtiofs daemon implementation which hooks into the existing
tvix-store filesystem implementation that is used for FUSE.

This allows adding the filesystem to a microvm without having to set up
FUSE inside the guest.

Change-Id: If80c36c9657f2289853e8d9a364bf4f1f7b7559c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9344
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-24 17:05:21 +00:00
Florian Klink
e5f2281856 feat(tvix/cli/derivation): reject derivations with empty names
As shown in the previous CLs, we can very well have store paths starting
with periods, but we can't have derivations with an empty name:

```
nix-build -E 'derivation { name = ""; builder = "/bin/sh"; system = "x86_64-linux"; }'
error: store path 'nr7i5pf18hw2zg487vkdyrbasdqylfcj-' has an empty name
```

I'm currently using ErrorKind::Abort here, because we don't have a
Derivation- related error in tvix-eval (and probably don't want to).

Change-Id: I0e9743cee98dbfa69e9caa2a58352176270f15bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9448
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-09-23 12:54:26 +00:00
Florian Klink
dfb3d30d45 fix(tvix/nix-compat): drop first character period check
Suggested in https://cl.tvl.fyi/c/depot/+/9108/5, but this disallows
adding a .gitignore file to the store.

Remove the check, and add a testcase.

Change-Id: Ieb78c417934756b2dbeb493040fe79726d1b9079
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9447
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-09-23 12:54:18 +00:00
Florian Klink
4f13b52047 feat(tvix/cli): add back a derivation test
Setting up the evaluator is a bit annoying currently, might get easier
with b/262, but it's better than no tests on that granularity at all.

Change-Id: Ie8c61466768f37f4efbc19ad497d37f87ddc2044
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9446
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-23 12:54:18 +00:00
Florian Klink
09a901de5b feat(tvix/nix-compat): fix and improve error message
Similar to cl/9350. Most of the times, this is still a regular string,
so let's print it like that if we can, and resort to base64 encoding if
we can't.

We were also wrongly always outputting the second character in the
string.

Change-Id: Id0e2a9d9f1ad3d2d7b554893ecd89a7e6383e9c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9445
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-23 12:54:18 +00:00
Florian Klink
732dc68727 refactor(tvix/nar-bridge): update to new separated protos
Change-Id: I5eaadc837a4d3a7f635574437127a22de88f556b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9407
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-09-22 17:41:32 +00:00
Florian Klink
e13c86d916 chore(tvix): re-enable {ca,}store-protos-go
This couldn't be enabled before until go get was able to fetch the
modules again.

Change-Id: I82db7d22e497661ea2a0c8a45c0c865fcc0a1f18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9406
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 17:36:05 +00:00
Florian Klink
80a02f96ce chore(tvix/castore): remove dependency on anyhow
We don't use this here, except in a test, where we don't really use
it either.

Change-Id: Ia6c45fccf663fe328942b1e1a2cd1c3a1b7f9ae5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9376
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-22 12:51:21 +00:00
Florian Klink
32f41458c0 refactor(tvix): move castore into tvix-castore crate
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.

Notable changes:
 - `fixtures` and `utils` had to be moved out of the `test` cfg, so they
   can be imported from tvix-store.
 - Some ad-hoc fixtures in the test were moved to proper fixtures in the
   same step.
 - The protos are now created by a (more static) recipe in the protos/
   directory.

The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)

Bug: https://b.tvl.fyi/issues/301

Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:51:21 +00:00
Florian Klink
d8ef0cfb4a docs(tvix/castore): rename traverse_to -> descend_to
With the move of this code out into castore it has become apparent this
is a general descent inside the castore. Concerns like making sure the
whole Directory closure has been fetched/is fetched initially is nothing
this layer needs to worry about. We can handle this during substitution
of a new PathInfo, once there's store composition.

Closes b/270.

Change-Id: I661ed08e54bc81478e032cfb9abeb23e5b337fbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9373
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:25 +00:00
Florian Klink
7c9a52e3ee chore(tvix): remove some unused crate dependencies
Change-Id: I31b0e7be3a2ebc268c6f4045ba282e8fb1c72a64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9375
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:24 +00:00
Florian Klink
8f56e71cba feat(tvix/shell): add cargo-machete
This helps spotting unused crate dependencies.

It's preferred over cargo-udeps for the reasons described in
https://blog.benj.me/2022/04/27/cargo-machete/

Change-Id: Ie801004485858741f2fa5ae6f33bd0ddfc292cd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9374
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:24 +00:00
Florian Klink
3d064f714c docs(tvix/store/pathinfosvc): fix copypasta error
This is not a simple get and put returning digest.

You currently can get by output hash, and a put gives you back the
(possibly modified) PathInfo message.

Change-Id: I43174d87a764417b8620c4d5bdac058cc25b0373
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9371
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-22 09:14:33 +00:00
Florian Klink
731996fbfe docs(tvix/eval): fix some broken docstr references
There's some more left, but they've been renamed/refactored out of
sight.

Change-Id: I41579dedc74342b4c5f8cb39d2995b5b0c90b0f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9372
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-09-22 09:14:33 +00:00
Connor Brewster
37a348b4fa refactor(tvix/store): Asyncify PathInfoService and DirectoryService
We've decided to asyncify all of the services to reduce some of the
pains going back and for between sync<->async. The end goal will be for
all the tvix-store internals to be async and then expose a sync
interface for things like tvix eval io.

Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-21 17:58:23 +00:00
Connor Brewster
7e737fde34 refactor(tvix/store/fs): Separate FUSE and filesystem code
In prepration for adding virtiofs support, I thought it would make sense
to split out the filesystem implementation from FUSE itself.

The `fs` module holds the tvix-store filesystem implemetation and the
`fuse` module holds the code to spawn a FUSE daemon backed by multiple
threads.

Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-20 14:27:53 +00:00
Connor Brewster
6b7c936bc5 refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rs
This switches the FUSE implementation from fuser to fuse-backend-rs.
fuse-backend-rs is designed to work with both FUSE and virtiofs.
Virtiofs support will make it possible to plug the tvix-store into a
microvm and have `/nix/store` access without having to setup FUSE inside
the guest.

Additionally fuse-backend-rs has nice support for running multiple FUSE
threads and has some async support.

The goal of this commit is to mechanically switch over to
fuse-backend-rs with minimal changes. I did have to add some locks here
and there because fuse-backend-rs uses `&self` on all methods whereas
fuser uses `&mut self`. `&self` is required for concurrent access to the
FUSE server, so this makes sense.

We can consider switching to concurrent maps and use some other
techniques to reduce lock contention and critical section size.

Issue: https://b.tvl.fyi/issues/305

Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-20 14:21:32 +00:00
Florian Klink
6c586bc2a7 feat(tvix/nar-bridge): graceful shutdown
This gives existing clients 30s to finish their requests after receiving
an interrupt.

Change-Id: Ia9b0e662fd1ffbbb6c2d03f3dd6548b13cf3d241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9365
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:34:12 +00:00
Florian Klink
02aed32bf2 refactor(tvix/nar-bridge): rename contentReader to blobReader
More aligned with how it's called in other places

Change-Id: I759ac7ca3b5b69c1101d2d51a569d76c183a6330
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9362
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
a9c47ffc50 refactor(tvix/nar-bridge): don't buffer blob in memory
Create a pipe, pass the read end, and have a goroutine write to the
write end.

Change-Id: I301c273355705e60113b018e7e84b76972200e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9361
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:02:28 +00:00
Florian Klink
07af692ecb refactor(tvix/nar-bridge): simplify CLI interface
Only keep the `serve` subcommand, and make it appear at the root.
Introduce a --log-level argument, and be a bit less noisy in normal
operation.

Change-Id: I86b8abde1869a5c0c947508bcc29f845222aac09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9360
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
dd7cc6ed68 fix(tvix/nar-bridge): don't log error on simple 404s
`nix copy` checks if NARs and NARInfo files are present, before
uploading them. That's not an error, but normal behaviour, so no need to
log with level info for these cases.

We only want to log if the error is not a 404, and log with Warn level.

Change-Id: I762de3b862d070a0f18bc62e324e94ca5c7c3693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9359
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
749ab67216 feat(tvix/nar-bridge): buffer request body a bit more
Let's make sure we don't end up blocking a client too much when
inserting very small blobs.

Change-Id: I640dda92efae538c70d32a40e6e85a23e9749e20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9358
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:02:28 +00:00
Florian Klink
0c031461c3 fix(tvix/nar-bridge): chunk blobs
Instead of creating one big BlobChunk containing all data, and creating
way too large proto messages, chunk blobs up to a reasonable (1MiB)
chunk size, and send them to the server like that.

Change-Id: Ia45a53956a6d7c0599cc59ac516ba37e9fb1b30e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9357
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
136b12ddd7 chore(tvix/nar-bridge): bump tvix/store/protos
Include the changes from cl/9351

Change-Id: Ie60c9dddcafaeee190439fa19fa7704917600fdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9363
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
ca25acf11b chore(tvix/store/protos): drop unused fields for now
This wasn't removed yet, and no code is using/populating it so far.

It's confusing, let's update it to the state of things now, and re-
introduce it once we get there.

Change-Id: I68f5ba17a8eee604d8ccd82749da7c8be094cb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9351
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-18 10:33:30 +00:00
Florian Klink
bf2fe88a5c fix(tvix/store/fuse): fix executable bit
We were blindly returning 0o444 for all regular files, but regular files
with executable bit need to be 0o555.

This wasn't spotted because stat'ing executable files was not part of
the test suite, it's now added.

Change-Id: I04c69784053e7e43d838c01bb288f2df48f40b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9345
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 10:33:30 +00:00
Florian Klink
da6cbb4a45 refactor(tvix/store/blobsvc): make BlobStore async
We previously kept the trait of a BlobService sync.

This however had some annoying consequences:

 - It became more and more complicated to track when we're in a context
   with an async runtime in the context or not, producing bugs like
   https://b.tvl.fyi/issues/304
 - The sync trait shielded away async clients from async worloads,
   requiring manual block_on code inside the gRPC client code, and
   spawn_blocking calls in consumers of the trait, even if they were
   async (like the gRPC server)
 - We had to write our own custom glue code (SyncReadIntoAsyncRead)
   to convert a sync io::Read into a tokio::io::AsyncRead, which already
   existed in tokio internally, but upstream ia hesitant to expose.

This now makes the BlobService trait async (via the async_trait macro,
like we already do in various gRPC parts), and replaces the sync readers
and writers with their async counterparts.

Tests interacting with a BlobService now need to have an async runtime
available, the easiest way for this is to mark the test functions
with the tokio::test macro, allowing us to directly .await in the test
function.

In places where we don't have an async runtime available from context
(like tvix-cli), we can pass one down explicitly.

Now that we don't provide a sync interface anymore, the (sync) FUSE
library now holds a pointer to a tokio runtime handle, and needs to at
least have 2 threads available when talking to a blob service (which is
why some of the tests now use the multi_thread flavor).

The FUSE tests got a bit more verbose, as we couldn't use the
setup_and_mount function accepting a callback anymore. We can hopefully
move some of the test fixture setup to rstest in the future to make this
less repetitive.

Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-18 10:33:30 +00:00
Connor Brewster
4ac6423b26 refactor(tvix/nar-bridge): Clean up directory popping loop
This change got lost in the rebases in cl/9348. There's unnecessary
`break`/`continues` that can be replaced by moving the conditional into
the for loop condition.

Change-Id: I559e21087630b05e483f768ab59f8067961a2eae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9352
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-17 13:55:22 +00:00
Connor Brewster
84aa07a736 fix(tvix/nar-bridge): Fix directory stack tracking
Previously, nar-bridge, had a couple of bugs with tracking the current
directory when traversing a NAR file.

The included test case looks like:
```
/ (dir)
/test (dir)
/test/tested (file)
/tested (file)
```

Previously, we would do a string prefix match between the current node
and the top of the directory stack to determine if the node is in the
directory. In this case `/test` is a substring of `/tested`; however,
`/tested` is not in the `/test` directory. The fix is to append a `/` to
the directory name when doing the prefix match, so `/test/` is not a
prefix of `/tested`.

Additionally, when popping the stack, we need to continuously pop the
stack until the new node is in the directory at the top of the stack
(stopping before we pop the root directory)

Example:
```
/ (dir)
/a (dir)
/a/b (dir)
/a/b/c (file)
/z (file)
```

Previously, `z` would end up in directory `/a` because we only the pop
the stack once.

The included test case requires both of these issues to be fixed for it
to pass, so I think it is sufficient.

Change-Id: I22f601babf04d39d85535ba7ad585d3970757211
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9348
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-17 13:34:00 +00:00
Florian Klink
5841047c38 feat(tvix/nar-bridge): increase timeouts
In some cases, Nix is not able to stream the NAR file fast enough. Bump the
timeouts for now. We might want to get a better understanding in what's
happening here long-term, and/or make the timeouts configurable.

Change-Id: Ieaa9c8f04bc73c6ce0679a058d07eaf87126634e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9340
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-17 13:24:53 +00:00
Florian Klink
0ecd10bf30 feat(tvix/nar-bridge): init
This provides a Nix HTTP Binary Cache interface in front of a tvix-store
that's reachable via gRPC.

TODOs:

 - remove import command, move serve up to toplevel. We have nix-copy-
   closure and tvix-store commands.
 - loop into CI. We should be able to fetch the protos as a third-party
   dependency.
 - Check if we can test nar-bridge slightly easier in an integration
   test.
 - Ensure we support connecting to unix sockets and grpc+http at least,
   using the same syntax as tvix-store.
 - Don't buffer the entire blob when rendering NAR

Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Co-Authored-By: Vo Minh Thu <noteed@gmail.com>

Change-Id: I6064474e49dfe78cea67676957462d9f28658d4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9339
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-17 13:24:53 +00:00
Florian Klink
683d3e0d2d chore(tvix): upgrade workspace.resolver to "2"
This shuts off the following warning:

```
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
```

Stuff still seems to work with it, so no need to stick with "1".

See https://doc.rust-lang.org/edition-guide/rust-2021/default-cargo-resolver.html
for more details.

Change-Id: I6056d95cd5cb793f37ef843ed43009a27ad36367
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9342
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-17 13:14:03 +00:00
Florian Klink
51b8571c48 feat(tvix/store): improve error message formatting
WrongSorting(Vec<u8>) actually encodes the name of a node, so if we can,
we want this to display it as a string (and fall back to the base64
encoding if we can't).

Before:

> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: [108, 111, 99, 97, 108, 101] is not sorted

After:

> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: locale is not sorted

Change-Id: I68420c53a89cb1aa96e4bdce414366cebcb7915f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9350
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-17 13:14:03 +00:00
Florian Klink
54bb63dea3 chore(tvix/store): allow non_snake_case
Some of the tonic-generated types create new warnings.

Change-Id: Ib538bfd7d90d857d80a129fbbdf3a38c89cc04ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9349
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-17 13:14:03 +00:00
Florian Klink
3816eaa5e9 feat(tvix/store): add simple integration test
This imports the docs folder into the tvix store, and ensures it comes
up with the same store path as Nix.

While we validate hashes in general through tvix-cli output path checks
already, it doesn't include the tvix-store CLI itself.

See https://cl.tvl.fyi/c/depot/+/9329/comment/339f0720_524f0104/
for context.

Change-Id: I239ce5b6a07cb962b242142ab716693359b8674c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9338
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 21:00:54 +00:00
Florian Klink
32f2f8b618 feat(tvix/store/import): print store path to stdout
This allows comparing the output in an integration test.

Change-Id: I8bb2254e18e90005a4f1b30fd47ef69642e3732e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9337
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 21:00:54 +00:00
Florian Klink
b52bb6e791 fix(tvix/store): log to stderr, not stdout
This leaves some space in stdout to provide actual meaningful output.

Also, rename print_node to log_node because that's what it does, it's
using the logger to log out more detailed info.

Change-Id: Ic64a6330dbfcdc63eb4198067a5c5e47b841b9a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9336
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-09-15 19:38:23 +00:00
Adam Joseph
9256621009 fix(tvix/eval): update identifier quoting to match cppnix 2.17
In cppnix 2.17, commit b72bc4a972fe568744d98b89d63adcd504cb586c, the
libexpr pretty-printing routine was fixed so that it would no longer
pretty-print attrsets with keywords in their attrnames incorrectly.

This commit implements the corresponding fix for tvix, fixes our
tests to work with cppnix>=2.17 oracles, and expands our test cases
to cover all the keywords.

Change-Id: I4b51389cd3a9c44babc8ab2a84b383b7b0b116ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9283
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-09-15 00:00:59 +00:00
Florian Klink
c77c7ae9a5 fix(tvix/store/blobservice/seeker): fix debug assert
We were asserting absolute_offset > self.pos, but that's not true for
both being zero.

Ramp up the tracing bits a bit, so we actually can see this in the debug
logs.

Change-Id: I21693bcafab227549b19cd6f1215d2f8dee77ecc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9292
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-12 12:15:08 +00:00
Florian Klink
5c32fb8ee2 docs(tvix/website): add tvix-store
Change-Id: I9f7021ce08d0259be06932b71a83ee0361e5096d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9296
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-09-12 08:19:17 +00:00
Brian Olsen
424ac8b577 fix(tvix): Get tvix build and shell to work on Darwin
On Darwin rust crates sometimes needs iconv manually added to
compile successfully. There is currently also a bug in strip that
requires that you set dontStrip on buildRustCrate for it to work.

See: https://github.com/NixOS/nixpkgs/issues/218712
Change-Id: I13555c7bbee1d34f08fc51a668d2067dbbe550ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9291
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-11 10:11:36 +00:00
Florian Klink
8c53ab5310 refactor(tvix/website): remove unused structuredData
Change-Id: I2445b977c5bed3e2d4b90e8ad81119d316b29b50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9281
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-09-10 13:13:33 +00:00
Florian Klink
afe4dfb61e refactor(tvix): move logo to //tvix:logo
Also expose both formats, then use it from
users/tazjin/presentations/tvix-eval-2023.

Change-Id: Id906e8aff5510a7a4f33336326472e86db18ea32
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9280
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-10 13:13:14 +00:00
Adam Joseph
6fa80ecc3d fix(tvix/eval): make compare_strict_eval_tests use Strict
Previously, compare_strict_eval_tests() was using Strictness::Lazy.
This appears to be a minor oversight from
0ab6494286.

This commit corrects that, by changing Strictness::Lazy to
Strictness::Strict.

Change-Id: Ia2389a5d30481cd322ed55230731340b795e5d87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9282
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-09-09 16:11:57 +00:00
Florian Klink
737846bf17 feat(tvix/store): add :store-protos-go
This target builds and runs the go tests in that directory, allowing us
to spot regressions as the one fixed in cl/9285.

Change-Id: Ia16c0622f29db343eae7c0386e715b292703bd4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9286
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-09-09 12:54:23 +00:00
Brian McGee
1549f020ee fix(tvix/protos): support []byte node names
The underlying protos were changed to return `[]byte` instead of `string`.

Change-Id: I5d3e5d8de0ed7200325f7ab0d62e3c10d8eb1b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9285
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-09 12:13:02 +00:00
Vincent Ambo
d546b9a15d docs(tvix/eval): document remaining opcodes
Change-Id: Iad8a5f78930872719b6481ddf1bbad8532bfa888
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7981
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-08 21:38:42 +00:00
Vincent Ambo
35e8dc97cf docs(tvix/eval): document attribute set related opcodes
Change-Id: Ib98a9fe8c9aa3f0e61c27d10285c0926cda7a969
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7979
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-08 21:38:42 +00:00
Vincent Ambo
3cf37c7cb6 docs(tvix/eval): add documentation strings for some OpCode variants
Change-Id: I42e610740b3687e1fd897d36694cce425751a8bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7975
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-09-08 21:38:42 +00:00
Florian Klink
ec22a80f26 docs(tvix/store/fuse): add more comment to list_root arg
Forgot to squash this into cl/9217.

Change-Id: Id5a87cbe729592f3b53e8c329b6890519f5da319
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9272
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-08 16:13:21 +00:00
Vincent Ambo
110328232d chore(3p/sources): bump channels & overlays (2023-08-21)
Included changes:

* tvix/eval: enable some lang tests on nix_latest

  Nix 2.16 contains some breaking language changes which Tvix does
  not yet implement, but the existing tests for them are now passed by
  Nix 2.16 (but not yet by Tvix).

* tvix/eval: disable a lang test on nix_latest

  In Nix 2.17, the identifier formatting test fails because some
  behaviour changed. We have not investigated further yet.

* 3p/overlays: use version of ihp-hsx that works with GHC 9.4

  Originally from the separate cl/9185.

* top-level: introduce a mechanism to exclude build targets from CI in
  the top level. This fixes b/296.

* users/grfn: disable builds of xanthous (and dependents) until the
  CLs fixing its build are submitted

* 3p/overlays: build nixos-option against Nix 2.15, the only version
  with which it builds

* 3p/overlays: bump tdlib to 1.8.16

Change-Id: Ia377f39dbdb08ac45ff830a615e64babc091e5ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9125
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-09-06 13:38:15 +00:00
Florian Klink
7923cc19f6 refactor(tvix/store): use tokio::task::JoinHandle
This makes the inside code a bit less verbose.

I wasn't able to describe the type of the async move closure itself,
which would allow us to remove the JoinHandle<_> type annotation
entirely.

Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-05 21:13:50 +00:00
Florian Klink
f499d2e031 feat(tvix/store): fix ctrl-c handling on mount command
This enables the tokio `signal` feature, and registers a ctrl_c signal
handler, which will use the unmount handle to unmount in case a ctrl-c
signal is received.

This avoids having disconnected mountpoints when Ctrl-C'ing a
`tvix-store mount` invocation.

In case the filesystem is unmounted externally (via `umount /path/to/
mountpoint`), the future is waiting for the signal is never resolved and
the task is stopped.

Change-Id: I149f705a6cb50188177f2a6c6a5fcd77218e2a3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9218
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-05 21:13:49 +00:00
Florian Klink
f9b5fc49b1 feat(tvix/store/fuse): allow listing
This provides an additional configuration flag to the tvix-store mount
subcommand, and logic in the fuse module to request listing for the
root of the mountpoint.

Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: adisbladis <adisbladis@gmail.com>
Tested-by: BuildkiteCI
2023-09-05 21:13:49 +00:00
Florian Klink
da9d706e0a feat(tvix/store/pathinfosvc): provide listing
This provides an additional method in the PathInfoService trait, as
well as an RPC method on the gRPC layer to list all PathInfo objects in
a PathInfoService.

Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-05 20:46:42 +00:00
Florian Klink
e41b5ae3f0 refactor(tvix/store): infer more types
We don't need to explicitly describe the type of the task itself,
describing the return type of the async closure is sufficient.

Also, use io::Result<_> instead of Result<_, io::Error>.

Change-Id: I9ab3f990eb49929b0aea335b2bb07da392ab631f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9267
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-09-05 20:41:41 +00:00
Florian Klink
7bd3c42c74 chore(tvix/store): drop walkdir workaround for symlinks at root
https://github.com/BurntSushi/walkdir/pull/170 got merged, meaning we
don't need to keep our own logic in here anymore.

Our test cases already cover this.

Change-Id: Ied3043ee651c8aafa10271c1e1ca5d460fb6c0b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9269
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-05 15:19:43 +00:00
Florian Klink
3c340b28bd refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate
This trait is eval-specific, there's no point in dealing with these
things in tvix-store.

This implements the EvalIO interface for a Tvix store.
The proper place for this glue code (for now) is tvix-cli, which knows
about both tvix-store and tvix-eval.

There's one annoyance with this move: The `tvix-store import` subcommand
previously also used the TvixStoreIO implementation (because it
conveniently did what we wanted).
Some of this code had to be duplicated, mostly logic to calculate the
NAR-based output path and create the PathInfo object.

Some, but potentially more of this can be extracted into helper
functions in a shared crate, and then be used from both TvixStoreIO in
tvix-cli as well as the tvix-store CLI entrypoint.

Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-09-04 16:02:12 +00:00
Florian Klink
583cdec3d8 docs(tvix/store): address cargo doc warnings
Fix some broken link references.

Change-Id: I69c9b2b62af35bb777e4df1a01ba3181a368be47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9214
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-03 10:01:40 +00:00
Florian Klink
9bbda4421e docs(tvix/store): update rpc_pathinfo.proto comments
Don't repeat the name of the method in the description, don't repeat
things already described in request message comments.

Change-Id: I180e4792577419050947eea8fea7043861aba463
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9213
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-03 10:01:40 +00:00
Florian Klink
116a137be0 refactor(tvix/store): implement rename for node::Node
This returns a node with a new name.

Change-Id: Iebcab537f8dd63d826b9841d4d0181fcb941afdd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-02 20:08:52 +00:00
Florian Klink
1e9d262ad5 docs(tvix/store): update comment
Since the refactor to use URIs for all three services, this actually
does talk to a daemon by default.

Change-Id: Ied296772b77eef514bfcae0a9dfc50f848a1c2f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9210
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-02 20:03:34 +00:00
Linus Heckemann
a3dbb60213 fix(tvix/eval): off-by-one in replaceStrings
replaceStrings would previously fail to replace the last character
in a string.

Change-Id: I43a7c960945350b2e7a5b731b7fdb617723eb38f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9151
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-24 10:07:48 +00:00
Florian Klink
778bd6894b refactor(tvix): assemble cragoDeps outputHashes from Cargo.nix
We already have these hashes accessible in the Cargo.nix file created
by cargo2nix, so there's no need to also manually maintain them here.

It removes one potential footgun I ran into while updating wu-manber to
a different rev, without updating it here.

Change-Id: I93932ac8ba55f83746ee38571b7740af2d49bbdf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9090
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-08-20 22:42:39 +00:00
Florian Klink
bba7bbf820 docs(tvix/nix-compat/nixhash): update comments
Change-Id: I46660da84065fd6938f581e14d67e231dca3c3ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9112
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 22:40:53 +00:00
Florian Klink
0193f07642 refactor(tvix/nix-compat/nixhash): validate digest lengths
There was a NixHash::new() before, which didn't perform any validation
of the digest length. We had some length validation when parsing nix
hashes or SRI hashes, but some places didn't perform validation and/or
constructed the struct directly.

Replace NixHash::new() with a
`impl TryFrom<(HashAlgo, Vec<u8>)> for NixHash`,  which does do this
validation, and update constructing code to use that, rather than
populating structs directly. In some rare cases where we're sure the
digest length is correct we still populate the struct manually.

Fixes b/291.

Change-Id: I7a323c5b18d94de0ec15e391b3e7586df42f4229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9109
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-08-20 22:19:22 +00:00
Florian Klink
4017039595 refactor(tvix/nix-compat): cargo clippy
Change-Id: I1c1608a6e75e451940fe1c61dc5ace5f0e7d7752
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9111
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 22:19:22 +00:00
Florian Klink
9bb3f74062 docs(tvix/docs): add document about drvPath differences
This question popped up every once in a while. While already explained
quite well at
https://inbox.tvl.su/depot/20230316120039.j4fkp3puzrtbjcpi@tp/T/#t,
it's not easily accessible.

Lift it from there into tvix/docs for better visibility.

Change-Id: I5f2d4aff31ab4adc421e06a7d36c871f45e09100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9080
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 22:08:02 +00:00
Florian Klink
366e11b48a refactor(tvix/eval): don't use format! in write! args
Change-Id: I0b2fdb418c2e36280d5c551a81634e1742193903
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9105
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-08-20 21:52:21 +00:00
Florian Klink
c67b18bf45 refactor(tvix/eval/vm): don't put HashMap in a Box
HashMap already is on the heap.

Change-Id: I53763e17469359e85862f297b5c2e7c0d8c3a980
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9104
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-08-20 21:52:21 +00:00
Florian Klink
aead2001d7 refactor(tvix/eval): impl Default for SourceCode
… instead of new(). Suggested by clippy.

Change-Id: Iac7be733392afefc2b4ff2e38386eee95f3bce94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9103
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-08-20 21:52:21 +00:00
Florian Klink
cf1a884ec5 refactor(tvix/eval/observer): cargo clippy
Change-Id: I11d7d55f31ddd27952f201a0abbed7445013f434
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9102
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-08-20 21:51:04 +00:00
Florian Klink
1a99b10b7e refactor(tvix/eval): cargo clippy
This passes a unit value to the function.

Change-Id: I4df3ad8fb0f35c0f110cee3349971ae28ce2878c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9101
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 21:51:04 +00:00
Florian Klink
ded577a73b refactor(tvix/eval/io): cargo clippy &Path
Change-Id: Ifec2a4fc93c482e41ff5993183643b5126a7728b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9100
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 21:51:04 +00:00
Florian Klink
def773d4d5 refactor(tvix/eval): cargo clippy (len() is usize)
Change-Id: I9d931ffcc03c6df7c0392dbc1c9a4ae0e3804213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9099
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 21:51:04 +00:00
Florian Klink
1f02dc7eba refactor(tvix/eval): cargo clippy &GenCo
Change-Id: I6b1b902ccbc12bf2acdb0fdf406d6ef336ff0b2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9098
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-08-20 21:51:04 +00:00
Florian Klink
690884426b refactor(tvix/eval): derive default for value::AttrsRep enum
Change-Id: Iaeb9ed7024c2ce85373f8aec0d223f52e1a3a539
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9097
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 21:51:04 +00:00
Florian Klink
3ffbcc6c8a refactor(tvix/store): cargo clippy
Change-Id: I3a80560d036e7ed08036b5e9f0974080d1a30ded
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9096
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-08-20 21:51:04 +00:00
Florian Klink
bc852fa56f refactor(tvix/nix_cli): cargo clippy
Change-Id: Ib48063660432bea9de2fc9aff5914ef76db3ad64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9095
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-08-20 21:51:04 +00:00
Florian Klink
146175b83d refactor(tvix/cli): cargo clippy
Change-Id: I723bc9b29733f5ee4e3094b92696fbdeb4b15a33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9094
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-08-20 21:43:02 +00:00
Florian Klink
600d4eec77 refactor(tvix/serde): cargo clippy
Change-Id: I3c86225040a04c9cb5702b226b4a5008ae219c42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9093
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 21:43:01 +00:00
Florian Klink
9cb38c5ba5 refactor(tvix/cli): stop parsing NixHash twice
We already have the parsed output_hash from above, no need to construct
it again.

Change-Id: Ie6d924ab446137c25c29fbeaf671aa7e5418262d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9110
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 19:44:19 +00:00
Ryan Lahfa
2d687e068a fix(tvix/nix-compat): disallow empty derivation names
Yes:

```
$ nix-build -E 'derivation { name = ""; builder = "/bin/sh"; system = "x86_64-linux"; }'
error: store path 'nr7i5pf18hw2zg487vkdyrbasdqylfcj-' has an empty name
```

Change-Id: I552f9ed1c1fe3bfceca18ca9b8e13d4b06dc6ff7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9108
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-08-20 19:34:35 +00:00
Ryan Lahfa
d504b440c2 feat(tvix/nix-compat): don't swallow hash validation errors
Previously, Output deserialization would silence validation errors and
provide `None` for `hash_with_mode` as soon as a validation error would
happen inside of the `NixHashWithMode` deserialization, e.g. invalid
hash length would not provide an validation error but a silent `None`
value.

This is problematic, we workaround a serde limitation here by writing
our own Deserializer.

As you can see, we write some boilerplate code unfortunately, as, for
example:

- `#[serde(fail_if_unparsed_as="Option::is_none")]` is not a thing,
  otherwise, we could have been able to just bubble up errors in case of
  "not fully parsed" (and not missing) values.

- `From<&serde_json::Value> for serde:🇩🇪:Unexpected` is not a thing,
  otherwise, we could just map invalid type errors and reuse the
  existing types instead of doing extremely bizarre things with
  `serde:🇩🇪:Unexpected::Other`, note: this is a not problem for
  expected, we know what we expect, we don't know what we received in
  practice.

I decided to write a `NixHashWithMode::from_map` which will eat a map
deserialized via `serde_json`, so our serde magic is not totally "data
model" agnostic.

I wanted to go for data model agnosticity and enable maximal
performance, e.g. building the structure as the map values are streamed
in the Visitor, this is needlessly painful because `Output` and
`NixHashWithMode` are in different files and this really makes sense
only if we write the full implementation in one file, indeed, otherwise,
you end up duplicating the work or having strange coupling.

So, for now, we will allocate a full map of the fields inside the
`Output`, i.e. if any "unknown field" is in that map, we will
deserialize it for no reason.

Doing it properly, as I repeat it in the code and to flokli at C3Camp
2023, requires to patch serde upstream IMHO.

Change-Id: I46fe6ccb8c390c48d6934fd3e3f02a0dfe59557b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9107
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-08-20 19:34:20 +00:00
Florian Klink
bb48d8c61b refactor(tvix): drop rust-analyzer from rust-docs build inputs
We don't need rust-analyzer to run cargo doc.

Change-Id: I5e2fd559e4045cadeab24b438c28d6df7f1d5d5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9092
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-08-19 20:16:28 +00:00
Florian Klink
6abf20f56b docs(tvix): update comment
cl/8306 fixed building all docs, but we forgot to update the comment.

Change-Id: I17829612f13e7357bd0efe8223cc28ed0f6cdea2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9091
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-08-19 20:12:45 +00:00
Florian Klink
946270142e refactor(tvix/nix-compat/derivation): cargo clippy
This `.into_iter()` call is equivalent to `.iter()` and will not consume
the `BTreeMap`.

Change-Id: Ie26637ebecb0bea5b09c447cc45ed207f8b50913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9088
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-19 11:47:48 +00:00
Florian Klink
311752412b docs(tvix/nix-compat/store_path): fix broken docstr reference
Change-Id: Ie95ac265f8707e138c6a7b529760650d211f259d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9087
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-08-19 11:41:26 +00:00
Florian Klink
999b2d6995 refactor(tvix/nix-compat/nixhash): use shorthand struct init
Change-Id: I1e52b752408a70fdb27e3e5c6f3bef5417b3e922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9085
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-08-19 11:37:46 +00:00
Florian Klink
b5ed3a90f2 docs(tvix/nix-compat): remove disambiguity
Don't import thiserror::Error directly, so rustdoc knows what
`crate::store_path::Error` we're talking about.

Change-Id: I755c9377521a6833e9a77cb1a41b48edafb31fe0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9086
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-08-19 11:27:45 +00:00
Florian Klink
39efe50311 refactor(tvix/nix-compat/derivation/escape): move to Vec<u8>
We only need bstr::ByteSlice to be able to use replace, it doesn't need
to return a BString.

Change-Id: I811948436fb89652e880970c2c05356183f3e439
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9084
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-08-19 10:54:32 +00:00
sterni
08a5849e2b chore(tvix/cli): delete unused NixCompatIO code
Change-Id: Icb91b102208fea512e04383ce9d65b0681af18ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9079
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-08-14 11:50:00 +00:00
Vincent Ambo
8478448146 fix(tvix/eval): fix a comment position in value::json
Change-Id: Ic58653254b36694f1991a18db9ceea0d532c2e31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9068
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-08-13 15:58:38 +00:00
sterni
8fb4c949dc test(tvix/eval): check truncation direction of builtins.div
builtins.div ought to truncate towards zero so that

    -(builtins.div a b) == builtins.div (-a) b
    -(builtins.div a b) == builtins.div a (-b)

Change-Id: I8b7c08cd7f4fa8a1363c786d42c8d484f6cd133d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9006
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-08-11 22:22:30 +00:00
Vincent Ambo
fcda068235 feat(nix/sparseTree): add optional name argument
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.

I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.

Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-08 20:11:56 +00:00
Florian Klink
f9fbc9be85 refactor(tvix/nix-compat/derivation/tests): fix typo
Change-Id: I764232378f79f16c781870d467469db91a2f14d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9001
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-31 21:59:42 +00:00
Florian Klink
a10d1e7bdf refactor(tvix/nix-compat): use local write::*;
Change-Id: I43cc1d5b3419255f815490b2e10f05db19e3bffc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9000
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 21:59:42 +00:00
Florian Klink
edde496040 refactor(tvix/nix-compat/test): rename data identifier
Make it more obvious if these are bytes pointing to JSON or ATerm, so we
don't get confused.

Change-Id: I2402c687b7ba9c05aac20ed63b0df54e4e96a9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8998
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 21:59:42 +00:00
Florian Klink
8258dbecdf refactor(tvix/nix-compat/derivation): drop some useless allocations
Change-Id: I85e44e24e9ec3fe4284f02623a1ac7ca9935e554
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8999
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 21:59:42 +00:00
Florian Klink
779fad29a1 refactor(tvix/nix-compat/derivation): use writer.write_all
This is more concise than a io::copy of a Cursor to bytes, and we have
everything to be written in memory.

Change-Id: I81f34666aa61aef4e16b33423ce4a69c3781efc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8997
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-31 21:50:48 +00:00
Florian Klink
737a6ca01e refactor(tvix/cli/refscan): use wu-manber crate with &[u8] support
PR'ed at https://github.com/tvlfyi/wu-manber/pull/1, and now merged.

Change-Id: I8c71e359196396a1d42a3ea2ab7ac15b137b2db0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8992
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 21:50:47 +00:00
Florian Klink
aa1982c085 refactor(tvix/derivation): move comma writing business up
write_input_derivations shouldn't need to write a comma to separate it
from the previous output from write_outputs.
This is better placed in the function calling all of these helper
functions.

Change-Id: I9ccc440e4665b52369ef39e75151b9a29469ce48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8995
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-31 21:50:44 +00:00
Florian Klink
3f856d8977 refactor(tvix/nix-compat/derivation): generic write_array_elements
We're happy with any &[S], as long as <S: AsRef<[u8]>.

This allows passing both strings and &[u8].

Change-Id: If2a80d9b1ee33ba328c9cdab4fa83ca7b98a71e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8994
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-31 21:50:44 +00:00
Florian Klink
34b7620764 refactor(tvix/nix-compat/derivation): simplify
Let the escape function only take care of string escaping, not quoting.

Let write_array_elements always quote and escape strings it consumes.
Move the business of writing additional wrapping characters around it to
the caller.

Change-Id: Ib8dea69c409561b49862c531ba5a3fe6c2f061f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8993
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-31 21:41:39 +00:00
Florian Klink
79531c3dab refactor(tvix/nix-compat): support non-unicode Derivations
Derivations can have non-unicode strings in their env values, so the
ATerm representations are not necessarily String anymore, but Vec<u8>.

Change-Id: Ic23839471eb7f68d9c3c30667c878830946b6607
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8990
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-07-31 21:41:38 +00:00
Florian Klink
9521df708f feat(nix-compat/store_path): implement PartialOrd and Ord
This allows sorting Store Paths. We delegate the sorting business to the
PartialOrd, Ord impls for our digest fields only, as two StorePaths with
the same digest, but different names can't exist.

Change-Id: I5f81631e5f5063893b316c63a240c5266b7e5bad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8988
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 20:29:25 +00:00
Florian Klink
0dea3086f5 docs(tvix/cli/refscan): fix comment
The comment still mentions Aho-Corasick, which isn't correct.

Change-Id: I846a2ca9ea7075c2456ca6ef04a132ff6161227a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8991
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-31 09:01:34 +00:00
Florian Klink
92e976b3a9 chore(tvix/nix-compat): drop useless clone
HashAlgo implements Copy, no need to clone here.

Change-Id: Ief11d2cfbd4fd0cd44224c7ddd575f518edbbd55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8989
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-31 09:01:34 +00:00
Florian Klink
432222f098 feat(tvix/store/proto): use Bytes instead of Vec<u8>
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our
bytes field cheaper to clone.

It's a bit annoying to configure due to
https://github.com/hyperium/tonic/issues/908, but the workaround does
get the job done.

Change-Id: I25714600b041bb5432d3adf5859b151e72b12778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-22 09:03:02 +00:00
Florian Klink
7971d7d9ff feat(tvix/nix-compat/store_path): store position in InvalidName
Add the position in the string where the name is problematic.

Change-Id: If6fd8be6100b718f8d68568eafc77ebb3cfb82d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8979
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-07-21 19:06:30 +00:00
Florian Klink
a6580748aa feat(tvix/store/digests): use bytes::Bytes instead of Vec<u8>
This will save us some copies, because a clone will simply create an
additional pointer to the same data.

Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-21 19:01:50 +00:00
Florian Klink
72e82ffcb1 refactor(tvix/store): use bytes for node names and symlink targets
Some paths might use names that are not valid UTF-8. We should be able
to represent them.

We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.

While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?

Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-21 19:01:49 +00:00
Florian Klink
638f3e874d feat(tvix/store/fuse): implement open explicitly
This "reverts" commit 9f600de226 (the
initial revert of f5e291cf83).

Now with BlobService returning a BlobReader that implements io::Seek, we
can actually just call blob_reader.seek(io::SeekFrom::Start(offset as
u64)).

This means, we currently will fail to seek backwards inside a file.

Change-Id: I9c19448df6831a3537252f99210374f2126ecfc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8886
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 18:52:47 +00:00
Florian Klink
7613e2e769 feat(tvix/store/blobservice): implement seek
For memory and sled, it's trivial, as we already have a Cursor<Vec<u8>>.
For gRPC, we simply reject going backwards, and skip n bytes for now.

Once the gRPC protocol gets support for offsets and verified streaming,
this can be improved.

Change-Id: I734066a514aed287ea3db64bfb1680911ac1eeb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8885
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-21 18:52:46 +00:00
Florian Klink
42dc18353d feat(tvix/nix-compat): have StorePath accept bytes
The primary constructor for this is now from_bytes, from_string is
simply calling .as_bytes() on the string, passing it along.

The InvalidName error now contains a Vec<u8>, to encode the invalid name
(which might not be a string anymore).

from_absolute_path now accepts a &[u8] (even though we might want to
make this a OSString of some sort).

StorePath::validate_name has been degraded to a pub(crate) function.
It's still used in src/derivation, even though it probably shouldn't at
all - that cleanup is left for cl/8412 though.

Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 18:04:38 +00:00
Florian Klink
5364fcb127 feat(tvix/nix-compat): fold NameError into Error
This being a nested error makes things more complicated than necessary.

Also, this caused BuildStorePathError to only hold NameError,
so refactor these utility functions to either return Error, or
BuildStorePathError.

Change-Id: I046fb403780cc5135df8b8833a291fc2a90fd913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8972
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-21 18:04:38 +00:00
sterni
8adc9c56f2 docs(tvix): document when pointer equality is preserved in C++ Nix
This explicitly documents behavior of C++ Nix that goes against the
intuition you'd gather from this document: that e.g. a simple select
from an attribute set causes a value to no longer be pointer equal to
its former self.

The point of documenting this is that we can show in a to be written
section on the use of pointer equality in nixpkgs that pointer equality
is only needed in a limited sense for evaluating it (C++ Nix's exterior
pointer equality). Tvix's pointer equality is far more powerful since
value identity preserving operations also preserve pointer equality,
generally speaking (this is because we implement interior pointer
equality in my made up terminology). This should eventually also be
documented.

Change-Id: I6ce7ef2d67b012f5ebc92f9e81bba33fb9dce7d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8856
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-11 16:17:42 +00:00
sterni
4ba624efae fix(tvix/eval): use byte, not codepoint index for slicing in escape
This fixes a subtle issue which would occasionally lead to a crash (e.g.
when evaluating (pkgs.systemd.outPath with --trace-runtime): With each
character in the string that has a multi byte representation in UTF-8,
the actual byte position and what tvix thought it was would get out of
sync. This could either lead to

* Tvix swallowing characters or jumbling characters if multi byte
  characters would cause the tracked index to become out of sync with
  the byte position before the first character to be escaped, or

* Tvix crashing if (in the same situation) the out of sync index would
  be within a UTF-8 byte sequence.

Luckily, std's `char_indices()` iterator implements exactly what
`nix_escape_char()`'s original author had in mind with
`.chars().enumerate()`. Using `i + 1` for continuing is safe, since all
characters that need (in fact, can) to be escaped in Nix are represented
as a single byte in UTF-8.

Change-Id: I1c836f70cde3d72db1c644e9112852f0d824715e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8952
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-07-11 16:11:40 +00:00
sterni
88b0545908 chore(3p/sources): Bump channels & overlays
* //tvix/store: update code generated from proto files

Change-Id: Ie6aa53e71f07a0a0e637673ba4005ed983db315d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8929
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-08 10:46:31 +00:00
DavHau
e751372f2f docs(tvix/store): various improvements
Improve change some little things I noticed while reading through it.

Change-Id: I033209eece395e5aad4e10825e8dd6c0cfe68191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8725
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-07 09:48:32 +00:00
Florian Klink
c767460440 fix(tvix/store/fuse): fix log level for a debug message
Change-Id: Ib801e46636901553d71455a739aed34e5828ca0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8888
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-02 11:40:29 +00:00
Florian Klink
2a601d26a8 docs(tvix/store/fuse/inodes): clippy lint
Change-Id: I7b1bcb9c0cca76f05271f25912d26b14152fe0c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8887
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-02 11:18:52 +00:00
sterni
2aaabb709f docs(tvix): update C++ Nix code links for pointer equality
Change-Id: Icfd79b36c09607b4183e7378cd3c17f6238297b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8853
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-07-01 21:45:53 +00:00
sterni
864931b297 docs(tvix): start restructuring pointer equality document
I want to expand on the C++ Nix behavior, since it seems relevant to
note that a lot of operations in C++ Nix (like select) don't preserve
pointer equality (see
<https://github.com/NixOS/nix/issues/3371#issuecomment-1596167957>).
It is especially so, as Tvix establishes pointer equality in a different
way and thus shows differing behavior. Therefore I want to additionally
document Tvix's current behavior and make it more explicit to what
extent nixpkgs needs pointer equality.

Change-Id: I9b4ba75dacb749c9fcbba4b9646c6b48bb57bbad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8852
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-07-01 21:45:53 +00:00
Florian Klink
5796563f1f chore(tvix/eval/vm): drop unused import
Change-Id: Ia04778391c198fde21da217bf697aa40157898b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8846
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-30 12:42:59 +00:00
Florian Klink
6b599b3763 feat(tvix/store/fuse): add test reading large file
Change-Id: Ic705d05909c59c764d68a730169e9cc2b2538d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8847
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 12:35:56 +00:00
Florian Klink
7fbf874f1c feat(tvix/store/blobsvc): add more blobservice tests
Change-Id: I3e27dfb4ce3e52974d7614814abb7b5ae4a37f8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8782
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 12:35:56 +00:00
Florian Klink
3edc580e80 docs(tvix/store/blobservice): update docstring
Change-Id: I6ae693a66530e1ecca57723a97d56b309fa0651a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8881
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 12:30:54 +00:00
Florian Klink
9f600de226 fix(tvix/store/fuse): revert "implement open explicitly"
This reverts commit f5e291cf83.

The offsets are relative to the start of the file, and as long as we
don't have BlobReaders implement seek, this will be very annoying to
deal with.

Change-Id: I05968f7c5c0ec0000597da90f451d6bb650c3e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8882
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-30 12:30:53 +00:00
Florian Klink
6604ce4e51 fix(tvix/store/blobservice): write into hasher from b, not buf
buf contains everything written so far, whereas b is the slice passed in
the current write() call. If we copy from &buf, we end up with the wrong
hash, because we keep writing the wrong data to the hash function.

Change-Id: I768d4645934a6a7d75b9c8eeba35f8f3be5edd26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8880
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 12:30:53 +00:00
Florian Klink
f5e291cf83 feat(tvix/store/fuse): implement open explicitly
This moves from stateless I/O to actually dealing with file handles,
allowing the filesystem to keep reusing existing blobreaders, instead of
opening a new reader on every read() call.

Change-Id: I3fc35c071e4aee1021c8bbd58749d082b0abd188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8834
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-30 11:59:48 +00:00
Evgeny Zemtsov
4528052eb7 fix(tvix/serde): remove unnecessary dependency on genawaiter
Change-Id: I52f13c6b508793603bc726353e9bf7d5f12fddf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8850
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-22 20:30:09 +00:00
Evgeny Zemtsov
c8fcdca4eb feat(tvix/eval): allow extending builtins outside of tvix_eval
The change allows applications that use tvix_serde for parsing
nix-based configuration to extend the language with domain-specific
set of features.

Change-Id: Ia86612308a167c456ecf03e93fe0fbae55b876a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8848
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-22 20:19:06 +00:00
Evgeny Zemtsov
f32abbdc1a feat(tvix/serde): make from_str_with_config public
Change-Id: I61962297101d35ed02673edf1bca0584b7be37cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8845
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-22 12:53:25 +00:00
sterni
66047063e0 fix(tvix/eval): use realpaths for import cache
I've noticed this behavior when writing the admittedly cursed test case
included in this CL. Alternatively we could use some sort of machinery
using `builtins.trace`, but I don't think we capture stderr anywhere.

I've elected to put this into the eval cache itself while C++ Nix does
it in builtins.import already, namely via `realisePath`. We don't have
an equivalent for this yet, since we don't support any kind of IfD, but
we could revise that later. In any case, it seems good to encapsulate
`ImportCache` in this way, as it'll also allow using file hashes as
identifiers, for example.

C++ Nix also does our equivalent of canon_path in `builtins.import`
which we still don't, but I suspect it hardly makes a difference.

Change-Id: I05004737ca2458a4c67359d9e7d9a2f2154a0a0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8839
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-21 07:48:52 +00:00
Florian Klink
b25d98a84e feat(tvix/store/fuse): initial implementation
This is a first implementation of a FUSE filesystem, mounting tvix-store
to a given location.

This is mostly meant as one additional lens into a store, and could be
used for builds. It's not meant to be used as a general-purpose thing.

It still has some rough edges:

 - It doesn't implement open/close, so it doesn't use file handles.
   Which means, we need to open blobs for partial reads over and over
   again.
 - It doesn't implement seek, as BlobReader doesn't implement seek yet.
 - It doesn't track "lifetimes" of inodes by listening on forget,
   meaning it might hold more data in memory than necessary.
 - As we don't have store composition (and a caching layer) yet,
   operations might be slow.

Change-Id: Ib1812ed761dfaf6aeb548443ae939c87530b7be8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8667
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-20 10:14:00 +00:00
sterni
4516cd09c5 fix(tvix/eval): only finalise formal arguments if defaulting
When dealing with a formal argument in a function argument pattern that
has a default expression, there are two different things that can happen
at runtime: Either we select its value from the passed attribute
successfully or we need to use the default expression. Both of these may
be thunks and both of these may need finalisers. However, in the former
case this is taken care of elsewhere, the value will always be finalised
already if necessary. In the latter case we may need to finalise the
thunk resulting from the default expression. However, the thunk
corresponding to the expression may never end up in the local's stack
slot. Since finalisation goes by stack slot (and not constants), we need
to prevent a case where we don't fall back to the default expression,
but finalise anyways.

Previously, we worked around this by making `OpFinalise` ignore
non-thunks. Since finalisation of already evaluated thunks still
crashed, the faulty compilation of function pattern arguments could
still cause a crash.

As a new approach, we reinstate the old behavior of `OpFinalise` to
crash whenever encountering something that is either not a thunk or
doesn't need finalisation. This can also help catching (similar)
miscompilations in the future. To then prevent the crash, we need to
track whether we have fallen back or not at runtime. This is done using
an additional phantom on the stack that holds a new `FinaliseRequest`
value. When it comes to finalisation we check this value and
conditionally execute `OpFinalise` based on its value.

Resolves b/261 and b/265 (partially).

Change-Id: Ic04fb80ec671a2ba11fa645090769c335fb7f58b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8705
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-20 10:07:44 +00:00
Florian Klink
6656b865b6 docs(tvix/store): correct some cargo doc warnings
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-19 22:54:46 +00:00
Florian Klink
ad687ccaf4 chore(tvix/store/pathinfosvc): clippy
Change-Id: Ied4bed08e989791f832922da8776d2104035e28a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8812
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-19 14:29:05 +00:00
Florian Klink
b10f008b03 chore(tvix/store/directorysvc): clippy
Change-Id: Idf45aaa0f6211ac35a9a41d0f3f60dfbe1009398
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8811
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-19 14:29:05 +00:00
Florian Klink
71093a513a chore(tvix/store/blobsvc): clippy
Change-Id: Ie384bdd27e1e9282ceda83edc74ffaad387f352b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8810
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-06-19 14:29:04 +00:00
Vincent Ambo
a3b212b82b test(tvix/serde): add a test for from_str_with_config
Change-Id: I451851f4e01cb6dd0005e55bb852823b618f2ee6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8809
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-16 12:11:06 +00:00
Vincent Ambo
3e915af8bb feat(tvix/serde): add a function to with custom configuration
This adds a `from_str_with_config` function which takes a
user-supplied closure that sets additional settings on the
`tvix_eval::Evaluation`.

Note that users can not set `strict = false`, but other settings are
not restricted.

This solves b/262.

Change-Id: Ice184400b843cfbcaa5b6fe251ced12b6815e085
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8808
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-06-16 12:05:04 +00:00
sterni
77b0dddc3d chore(tvix/eval): fix markdown labeled link syntax
Change-Id: I639dc0801090eaba56b61858e28204b5a0e631b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8784
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-15 19:40:08 +00:00
sterni
3b8c9ec9c8 test(tvix/eval): update nix_tests suite to C++ Nix master
Adds new tests for foldl', intersectAttrs as well as fills in missing
.exp files.

New test cases we don't pass:

- fromTOML timestamp capabilities
- path antiquotation
- replaceStrings is lazier on C++ Nix master

The C++ Nix revision used is 7066d21a0ddb421967980094222c4bc1f5a0f45a.

Change-Id: Ic619c96e2d41e6c5ea6fa93f9402b12e564af3c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8778
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-06-15 19:28:16 +00:00
sterni
9278a0cd16 docs(tvix/eval): update test suite documentation
Change-Id: Ie9153c00b95ede4837a8eeab341e68bc90e97921
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8777
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-15 19:28:16 +00:00
sterni
0f71d8f813 test(tvix/eval): genericClosure (pointer) comparison support
genericClosure has very limited support for pointer equality: It relies
on comparison (not equality!) in C++ Nix, so as soon as C++ Nix supports
comparing lists (langVersion >= 6) we can rely on pointer equality for
key.

Since Tvix uses equality, not comparison for the insert, our behavior is
currently different, as documented by the notyetpassing tests.

Change-Id: Ifcd741ed4fc3ccc3825f7038875d56a9918b786a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8720
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-15 11:01:06 +00:00
sterni
0005737f11 fix(tvix/eval): make tvix display values like nix-instantiate(1)
In order for the test suite we have currently to be comparable to C++
Nix, we need to display values in the same way. This was largely the
case except in some weird cases.

* <CODE> for thunks and <CYCLE> for repeated thunks (?) are already in
  use. <CODE> formatting is tested by the oracle test suite already.

* Instead of lambda, we need to use <LAMBDA>

* <<primop>> and <<primop-app>> (a formatting C++ Nix uses nowhere)
  now are <PRIMOP> and <PRIMOP-APP>.

We'll probably want to have a fancier display of values (in a separate
trait) down the line. This could be used for interactive usage, e.g. the
REPL or a potential debugger.

There is a peculiarity with C++ Nix 2.3 formatting primops: import is
considered a <<PRIMOP-APP>>, since it is internally implemented by means
of scopedImport. This implementation detail no longer leaks in C++ Nix
2.13 nor in Tvix.

<CYCLE> display is untested at the moment, since we exhibit a
discrepancy to C++ Nix 2.3. Our current detection is more similar to C++
Nix 2.13—luckily it is also the more consistent of the two. See also
b/245.

Change-Id: I1d534434b02e470bf5475b3758920ea81e3420dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8760
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-06-15 11:01:06 +00:00
Florian Klink
f087e07ca1 fix(tvix/store/blobsvc): drop stray println! debug statement
Change-Id: Ica073820ea8240d42ce8b979309f881af18a8cde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8781
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-06-14 23:16:09 +00:00
Florian Klink
c5851fb815 refactor(tvix/store/blobsvc): make from_addr not async
A previous iteration of this code did actually connect (in the gRPC
client), which was why we had this function async.

However, as the connection there is now lazy too, we can drop the
asyncness in this function.

Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 23:16:09 +00:00