Determining the inputs might trigger additional builds/substitutions,
so answering these lookups via a lambda in a lazy fashion gets
complicated.
You end up assembling the list of input nodes upfront, and the lambda
will just be a dumb lookup into that preassembled list.
Rather than doing that, simply have derivation_to_build_request leave
the work of determining the inputs to the caller.
Change-Id: I75880132916c76b930807c989090da298b6891bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10626
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This are leftovers from the "reference scanning" approach (which we
didn't end up using).
We still want a concept of known paths, so we can trace IO into
storepaths back to the build recipe that'll produce it, so let's keep
the rest of this struct around.
Change-Id: I73d38e21e5b97950b8fc2a42176cae5f80d371c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10632
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
A bunch of operations in Tvix are not aware of catchable values
and does not propagate them.
In the meantime, as we wait for a better solution, we just offer this
commit for moving the needle.
Change-Id: Ic3f0e1550126b0847b597dfc1402c35e0eeef469
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10473
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
`args` was not propagating context, here's a regression test for it.
Change-Id: I8b6a3148508d40df0077128f0bafe68c098a03bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10610
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds support to handle the __structuredAttrs argument, which can be
passed to builtins.derivationStrict.
If __structuredAttrs is passed, and set to true, most of the arguments
passed to builtins.derivationStrict are not simply coerced to a string
and passed down to "environments", but instead kept in a more structured
fashion.
Inside ATerm, which is what's relevant as far as path calculation is
concerned, a virtual `__json` environment variable is present,
containing these structured values.
Inside Builds, these structured values are not made available as an
environment variable, but a JSON file (and source-able bash script).
This will need to be respected once we start emitting BuildRequests,
and for that we can probably just parse the `__json` key in
Derivation.environment again - or keep this additionally in
non-serialized form around during Evaluation.
No matter what, this is left for a followup CL.
The existing handle_derivation_parameters and populate_outputs helper
function were removed, as __structuredAttrs causes quite a change
in behaviour, and so handling both in the same place makes it more
readable.
There's some open questions w.r.t. string contexts for structured attrs
itself. A TODO is left for this, but at least path calculation for
individual structured attrs derivations are correct now.
Part of b/366.
Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Allow other crates (like tvix-glue) to look at a Value in JSON, which is
used by the structured attrs feature.
Change-Id: Iba02ace6e11a74c3f9b19dcbef4b008b76dec046
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10602
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
My OCD could not be stopped.
Change-Id: I2bf504fe0865a5084ad02aee18e6180a8a3e19d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10609
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The Derivation input_derivations field contains a list of input
derivations and (a subset of their) output names.
This means, multiple nodes can be returned, so return a Vec.
Also, update the name to better reflect the nodes are the nodes of the
selected outputs, not a node representing the .drv file itself.
Additionally, use a proto::node::Node (the naked enum), rather than
proto::Node, which wraps this in an optional struct field until
realizing the BuildRequest.
Change-Id: Iec5620b5d7ac0462f2c76acac4abcaeea2de0aad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10608
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Provide a store_path_to_node_sync function which uses the runtime handle
to block on the async function internally, but make store_path_to_node
itself async, so it can call async functions internally.
We'll use that later when triggering builds and waiting on their
results.
Change-Id: Idae9da7aa5b0878e0d3a2eba34ea2623e1ba84b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10607
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We don't need Arcs in most of the cases, we're fine with some container.
Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
To render NARs, we're fine with a simple AsRef to a BlobService and
DirectoryService. We just need to have the function pass back the
references, so we can reuse it after the recursion.
Change-Id: I8a1b899134ddda26cf14aa829a08383986101850
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
The error message is misleading. The errors we return can happen both
during serialization or deserialization, though the messages suggested
the latter only.
Change-Id: I2dafe17ec78ee75cab5937a3a81540fda3175eac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10603
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds support to retrieve a list of chunks for a given blob to the
BlobService interface.
While theoretically all chunk-awareness could be kept private inside
each BlobService reader, we'd not be able to resolve individual chunks
from different Blobservices - and due to this, not able to substitute
chunks we already have in a more local store.
This function allows asking a BlobService for the list of chunks,
leaving any actual fetching up to the caller (be it through individual
calls to open_read), or asking another store for it.
Change-Id: I1d33c591195ed494be3aec71a8c804743cbe0dca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10586
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Make it clear this is only used inside the scope.
Change-Id: Ie94f88d7f0fb58cd4bf9c2f1176000b272e6f2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10585
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We need to be a bit careful and pass the BlobService around (similar to
how we already do with the directory_putter), but that allows getting
rid of a bunch of annoying trait bounds.
We also stop spawning additional tasks where we can just use block_on.
Change-Id: If36de0ee947d2c779d20a384308241d2262d4764
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10580
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
The docstrings were not updated once we made the BlobService trait async.
There's no more need to turn things into a sync reader.
Also, rearrange the stream manipulation a bit, and remove the need to
create a new VecDeque for each element in the stream. bytes::Bytes
implements the Buf trait.
Fixes b/289.
Change-Id: Id2bbedca5876b462e630c144b74cc289c3916c4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10582
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This test serves as a minimal reproducer for output path calculation.
Derivations with the same name and output hash, but different build
recipe should end up with the same outPath.
However derivations with different name should end up with a different
outPath.
Change-Id: I555be59dd87ea675a0816188ed973f96c311e4e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
It was a `//` not a `///`.
Change-Id: Iee3e8c116d73b5dd8a41c027153714415a66695f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't need to spawn in all these places, we can just block_on
directly, this is all IO bound.
This also means, we don't need to clone any of the service handles
(except preserving clone-ability of the BlobService).
Change-Id: I7d90f4d6a263a98491caa071ada538a5197a5472
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10540
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This was wrongly named, it returns a specific node at a subpath.
Also, this code can be simplified a lot - we don't need to spawn
additional tasks, and can get rid of some clones too.
This is also where we need a certain build - so add some TODO to block /
fetch here.
Change-Id: Id26d7bd80f7a2095121e642b3f7716de78d6b6a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10539
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Extend our validation function to also check for the None case.
Change-Id: Ib75f880646d7fb3d66588f1988e61ec18be816a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10534
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Make this an `AsRef<dyn DirectoryService>`.
This helps dropping some Clone requirements.
Unfortunately, we can't thread this through to TvixStoreIO just yet.
Change-Id: I3f07eb28d6c793d3313fe21506ada84d5a8aa3ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
While doing the obvious thing, ensuring clippy doesn't complain on our
benchmarks and examples, this will also flag build failures on them,
and contrary to cl/10301 not pollute the /nix/store with stuff we can't
execute anyways.
Change-Id: I7ee1c2babbc67dac9794274d05bb9d6e1251ed01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10544
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
cl/10475 updated the API of tvix_eval::Evaluation, but didn't update
this benchmark to use that new API.
Also, fixes the docstring to no longer specify that there is a "given
snippet".
Change-Id: Ibb8285731849dbeec814e2585bbaa36f22368afe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10542
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
With context strings, we are now able to reproduce up to firefox.
Until a more problematic example can be found, we should at least
lean on this particular example.
Change-Id: Ibeaf799d26e1f160b6c93f8ccd978702a2a7e3e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10460
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Raw printing will transform the result into a string
and print it.
In case of a contextful string, this will fail by default, as expected.
Change-Id: I5e564329e7b001adc57a77a9153b4425cb332bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10457
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Nix does something like:
```cpp
NixStringContext context;
const auto str = state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.split");
```
And then do nothing with that context, therefore, we follow them and
make `split` aware of the context but still do nothing with it.
Change-Id: I4fee1936600ce86d99d00893ca3f64013213935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10428
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
We calculate the input context by performing the union of context
over all input of the derivation.
Then, we just pass the rest of it to the remaining machinery.
Finally, we re-emit an `outPath` and a `drvPath` containing the expected
contexts.
Change-Id: I74905fb258b5bee8b08d1208c9eb87f51b92a890
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10436
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This removes the reference tracking and uses instead the context for
references and returns some.
Change-Id: Ic359ca6b903b63f1a9c679c566004c617b792442
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10435
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
In the past reference tracking system, `tvix-io` glue was appending
plain paths in the known path state.
Now, we make up for this by just making contextful coercion of file
imports.
Change-Id: Ieb9b04dd83302c77909252d5f7733857ac3cf8fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10443
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
String with contexts are always coerced to a string with the same
context.
Change-Id: I224814febd9cad196bb28876793e76bed564dc72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10440
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: sterni <sternenseemann@systemli.org>
Otherwise, you just fail because they are not... contextless strings!
Change-Id: I0b8f63a18cd89c3841b613d41c12ec4ee336f953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10442
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
`match` silently ignore the input context and do not propagate
it and successful matches.
The why is unclear but nixpkgs does rely implicitly on this behavior
because dynamic attribute selection cannot be done with contextful
strings.
Change-Id: I5167fa9b2c2db8ecab0c2fb3e9895c9cfce6eeb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10441
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
And it also preserve the original context if it exists.
Change-Id: I904f7c13b7f003a267aace6301723780fccaafb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10434
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
We just perform union of contexts of every pieces.
Change-Id: Ief925c1818cd8bbec0503e9c625b0630feebfdda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10432
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
