TvixStoreIO triggers builds whenever IO into a not-yet-built store path
is requested, if it knows how to build that path.
Change-Id: If30e9db6be2f2a30cbc9d0576f357f3ecfa0d35a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10645
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We propagate a `TvixStoreIO` as the `state` of our derivation-specific
builtins in the glue crate.
The evaluators `io_handle` itself is using a Rc<dyn EvalIO>.
An earlier version of TvixStoreIO was also introducing generics over the
different internal services themselves, but we opted for instead
hardcoding this to Arc<dyn …> for the sake of less macro voodoo.
Change-Id: I535c476f06b840858fa3070c4a237ece47f7a15b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10636
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Have a Evaluation::new() function that's used to set up the Evaluation
struct initially - which is also used by both new_pure and new_impure
internally.
It's generic over the exact type of IO, making it easier to instantiate
Evaluation with non-tvix-eval EvalIO implementations, that might not be
in a Box.
Change-Id: Ibf728da24aca59639c5b6df58d00ae98c99a63f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10640
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I lost a lot of hope and had to read the source code of `quote!`, `cargo expand` was invaluable
in this adventure. We should keep it IMHO.
Change-Id: Icfb4c80d413602f2bdc6deab0d595183825d88ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10635
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Don't restrict to a Box<dyn EvalIO>.
There's still one or two places where we do restrict, this will be
solved by b/262.
Change-Id: Ic8d927d6ea81fa12d90b1e4352f35ffaafbd1adf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10639
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
`throw (throw "a")` should work and propagate the internal throw.
Before this commit, it didn't work.
Change-Id: Id5d46f74e484dba99e912ad9fa211f3bf1617bac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10600
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
`elem` did not catch the list being a catchable.
This surfaced during Nixpkgs evaluation.
Change-Id: Icf19b94e914e35a435c4412d769ee63ba59ab7b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10599
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is an additional test suite on the top of the Nix ones
for context strings matters.
It already smoked out multiple mistakes and potential bugs and non-deterministic result from the evaluator.
It uses a similar technology as the one in the tvix-eval albeit we instantiate a fully fledged evaluator
with in-memory store.
We copy the files instead of symlinking them because crates are built in
isolation, so symlinks cannot work.
Change-Id: I63ae225ce4f83c6e2c8ccd60d779c2f8eb9d08fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10619
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Previously, we were assembling very naively an attribute set composed of context we saw.
But it was forgetting that `"${drv}${drv.drvPath}"` would contain 2 contexts with the same key, but
with different values, one with `outputs = [ "out" ];` and `allOutputs = true;`.
Following this reasoning and comparing with what Nix does, we ought to merge underlying values systematically.
Hence, I bring `itertools` to perform a group by on the key and merge everything on the fly, it's not
beautiful but it's the best I could find, notice that I don't use
`group_by` but I talk about group by, that is, because `group_by` is a
`group_by_consecutive`, see
https://github.com/rust-itertools/itertools/issues/374.
Initially, I tried to do it without a `into_grouping_map_by`, it was akin to assemble the final `NixAttrs` directly,
it was less readable and harder to pull out because we don't have a lot of in-place mutable functions on
our data structures.
Change-Id: I9933c9bd88ffe04de50dda14f21879b60d8b8cd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10620
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Yes, `hasContext e` should work where `e` is a contextful strings, otherwise, it is really useless.
Change-Id: I5eb071fc257217d6e8a63fe519132ebd98186696
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10617
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
`test-generator` has not been updated in the past 2 years.
`rstest` has not been updated in the past 5 months.
This is an improvement in the maintenance state… I guess?
We get also new features, it changes the name of the tests with numbers too.
Change-Id: I5376104c7704f525dba7524da78daa09867cc669
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10623
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We want to handle bottoms in a consistent fashion. Previously this was
handled by repetitive is_catchable checks, which were not consistently
present.
Change-Id: I9614c479cc6297d1f64efba22b620a26e2a96802
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10485
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Rather than passing strings around, use a StorePathRef.
This makes things a bit more typesafe, and more aligned with what we
want to do in b/264.
Change-Id: Ib7080addf27e7f1a9c8da1d8aaa66744468e3b5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10633
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We need to vendor in the package expression, as it's not possible to
override cargoHash.
Change-Id: Ib123647bb9b96d41f4630daa431d020f1cb8d4fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10624
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This starts a BuildService as a separate process, currently defaulting
to the DummyBuildService.
Change-Id: Ic206f00831641d3ffebaa44883b7dc053700b9ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10631
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This allows constructing a BuildService from a URI, similar to how it's
done in tvix-[ca]store.
Change-Id: Ib962b329535c6c7e378ab7ac7f4dd254366497b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10630
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Also provide a dummy implementation that just fails on any build that's
requested.
Change-Id: I0df743a730c5331ec9ce6e97a966abe18ce067f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10627
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Determining the inputs might trigger additional builds/substitutions,
so answering these lookups via a lambda in a lazy fashion gets
complicated.
You end up assembling the list of input nodes upfront, and the lambda
will just be a dumb lookup into that preassembled list.
Rather than doing that, simply have derivation_to_build_request leave
the work of determining the inputs to the caller.
Change-Id: I75880132916c76b930807c989090da298b6891bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10626
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This are leftovers from the "reference scanning" approach (which we
didn't end up using).
We still want a concept of known paths, so we can trace IO into
storepaths back to the build recipe that'll produce it, so let's keep
the rest of this struct around.
Change-Id: I73d38e21e5b97950b8fc2a42176cae5f80d371c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10632
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
A bunch of operations in Tvix are not aware of catchable values
and does not propagate them.
In the meantime, as we wait for a better solution, we just offer this
commit for moving the needle.
Change-Id: Ic3f0e1550126b0847b597dfc1402c35e0eeef469
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10473
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
`args` was not propagating context, here's a regression test for it.
Change-Id: I8b6a3148508d40df0077128f0bafe68c098a03bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10610
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds support to handle the __structuredAttrs argument, which can be
passed to builtins.derivationStrict.
If __structuredAttrs is passed, and set to true, most of the arguments
passed to builtins.derivationStrict are not simply coerced to a string
and passed down to "environments", but instead kept in a more structured
fashion.
Inside ATerm, which is what's relevant as far as path calculation is
concerned, a virtual `__json` environment variable is present,
containing these structured values.
Inside Builds, these structured values are not made available as an
environment variable, but a JSON file (and source-able bash script).
This will need to be respected once we start emitting BuildRequests,
and for that we can probably just parse the `__json` key in
Derivation.environment again - or keep this additionally in
non-serialized form around during Evaluation.
No matter what, this is left for a followup CL.
The existing handle_derivation_parameters and populate_outputs helper
function were removed, as __structuredAttrs causes quite a change
in behaviour, and so handling both in the same place makes it more
readable.
There's some open questions w.r.t. string contexts for structured attrs
itself. A TODO is left for this, but at least path calculation for
individual structured attrs derivations are correct now.
Part of b/366.
Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Allow other crates (like tvix-glue) to look at a Value in JSON, which is
used by the structured attrs feature.
Change-Id: Iba02ace6e11a74c3f9b19dcbef4b008b76dec046
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10602
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
My OCD could not be stopped.
Change-Id: I2bf504fe0865a5084ad02aee18e6180a8a3e19d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10609
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The Derivation input_derivations field contains a list of input
derivations and (a subset of their) output names.
This means, multiple nodes can be returned, so return a Vec.
Also, update the name to better reflect the nodes are the nodes of the
selected outputs, not a node representing the .drv file itself.
Additionally, use a proto::node::Node (the naked enum), rather than
proto::Node, which wraps this in an optional struct field until
realizing the BuildRequest.
Change-Id: Iec5620b5d7ac0462f2c76acac4abcaeea2de0aad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10608
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Provide a store_path_to_node_sync function which uses the runtime handle
to block on the async function internally, but make store_path_to_node
itself async, so it can call async functions internally.
We'll use that later when triggering builds and waiting on their
results.
Change-Id: Idae9da7aa5b0878e0d3a2eba34ea2623e1ba84b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10607
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We don't need Arcs in most of the cases, we're fine with some container.
Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
To render NARs, we're fine with a simple AsRef to a BlobService and
DirectoryService. We just need to have the function pass back the
references, so we can reuse it after the recursion.
Change-Id: I8a1b899134ddda26cf14aa829a08383986101850
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
The error message is misleading. The errors we return can happen both
during serialization or deserialization, though the messages suggested
the latter only.
Change-Id: I2dafe17ec78ee75cab5937a3a81540fda3175eac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10603
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds support to retrieve a list of chunks for a given blob to the
BlobService interface.
While theoretically all chunk-awareness could be kept private inside
each BlobService reader, we'd not be able to resolve individual chunks
from different Blobservices - and due to this, not able to substitute
chunks we already have in a more local store.
This function allows asking a BlobService for the list of chunks,
leaving any actual fetching up to the caller (be it through individual
calls to open_read), or asking another store for it.
Change-Id: I1d33c591195ed494be3aec71a8c804743cbe0dca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10586
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Make it clear this is only used inside the scope.
Change-Id: Ie94f88d7f0fb58cd4bf9c2f1176000b272e6f2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10585
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We need to be a bit careful and pass the BlobService around (similar to
how we already do with the directory_putter), but that allows getting
rid of a bunch of annoying trait bounds.
We also stop spawning additional tasks where we can just use block_on.
Change-Id: If36de0ee947d2c779d20a384308241d2262d4764
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10580
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
The docstrings were not updated once we made the BlobService trait async.
There's no more need to turn things into a sync reader.
Also, rearrange the stream manipulation a bit, and remove the need to
create a new VecDeque for each element in the stream. bytes::Bytes
implements the Buf trait.
Fixes b/289.
Change-Id: Id2bbedca5876b462e630c144b74cc289c3916c4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10582
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This test serves as a minimal reproducer for output path calculation.
Derivations with the same name and output hash, but different build
recipe should end up with the same outPath.
However derivations with different name should end up with a different
outPath.
Change-Id: I555be59dd87ea675a0816188ed973f96c311e4e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
It was a `//` not a `///`.
Change-Id: Iee3e8c116d73b5dd8a41c027153714415a66695f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't need to spawn in all these places, we can just block_on
directly, this is all IO bound.
This also means, we don't need to clone any of the service handles
(except preserving clone-ability of the BlobService).
Change-Id: I7d90f4d6a263a98491caa071ada538a5197a5472
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10540
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
