Add multiple additional helpers such as:
- `path_to_name`: derive the basename of a given path
- `derive_nar_ca_path_info`: derive the `PathInfo` for a content
addressed NAR
which isolates further the tree walking feature and the ingestion feature.
Additionally, we don't `expect` anymore and propagate properly ingestion errors up.
Change-Id: I60edb5b633911c58ade7e19f5002e6f75f90e262
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10574
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
In one function that does the heavy lifting: `ingest_entries`, and three additional helpers:
- `walk_path_for_ingestion` which perform the tree walking in a very naive way and can be replaced by the user
- `leveled_entries_to_stream` which transforms a list of a list of
entries ordered by their depth in the tree to a stream of entries in
the bottom to top order (Merkle-compatible order I will say in the
future).
- `ingest_path` which calls the previous functions.
Change-Id: I724b972d3c5bffc033f03363255eae448f017cef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10573
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
To make use of the filtering feature, we need to revert the internal walker to a real DFS.
We will therefore just invert the whole tree by storing all of its
contents in a level-keyed vector.
This is horribly expensive in memory, this is a compromise between CPU
and memory, here is the fundamental reason for why:
When you encounter a directory, it's either a leaf or not, i.e. it
contains subdirectories or not.
To know this fact, you can:
- wait until you notice subdirectories under it, i.e. you need to store
any intermediate nodes you see in the meantime -> memory penalty.
- getdents or readdir on it to determine *NOW* its subdirectories -> CPU
penalty and I/O penalty.
This is an implementation of the first proposal, we pay memory.
In practice, we are paying O(#nb of nodes) in memory.
There's a smarter albeit much more complicated algorithm that pays only
O(\sum_i #siblings(p_i)) nodes where (p_1, ..., p_n) is the path to a leaf.
which means for:
A
/ \
B C
/ / \
D E F
We would never store D, E, F but only E, F at a given time.
But we would still store B, C no matter what.
Change-Id: I456ed1c3f0db493e018ba1182665d84bebe29c11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10567
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
In cl/10468, we accepted SRI hashes of invalid padding while checking
their trailing bits.
In this commit, we accept SRI hashes of invalid padding and invalid length, as Nix does.
Real world example: `pkgs.javaPackages.openjfx11.deps`
<849e4dc5ff/pkgs/development/compilers/openjdk/openjfx/11.nix (L71)>
in nixpkgs.
Change-Id: I834437e7b94dab9fbb030163f7a2741f52bbf03a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10668
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Impure builtins need to be explicitly added. This was missed from
cl/10640.
Change-Id: I1f8a84aa02f1cf8a26247960d6eb15895e09c610
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10667
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
- Adjust to ecl 23.9.9 release
- Regenerate go protos after protoc-gen-go update
- Drop dhall fork which hasn't kept up with 1.42.*
- Address new clippy warnings:
- Variant naming of Error::ValidationError
- Simplify .try_into().unwrap()
- Drop unnecessary identity function
- Test module must be last in file
- Drop unused `pub use`
- Update agenix to 0.15.0. Current master has a installCheckPhase that
doesn't work with C++ Nix 2.3.*:
a23aa271be (commitcomment-137185861)
Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Having something running the depot crate2nix and formatting it with
depotfmt is useful outside of tvix too.
Change-Id: Iecc8f207da38cc6995747c5ea48d3911433fd416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10658
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
So that we can just `map_err` easily in functions returning `std::io::Error` but calling functions
returning `castore::import::Error`.
Change-Id: Id181b95e8431c69e95f3a8cd569ca10306656e1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10572
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
TvixStoreIO triggers builds whenever IO into a not-yet-built store path
is requested, if it knows how to build that path.
Change-Id: If30e9db6be2f2a30cbc9d0576f357f3ecfa0d35a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10645
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We propagate a `TvixStoreIO` as the `state` of our derivation-specific
builtins in the glue crate.
The evaluators `io_handle` itself is using a Rc<dyn EvalIO>.
An earlier version of TvixStoreIO was also introducing generics over the
different internal services themselves, but we opted for instead
hardcoding this to Arc<dyn …> for the sake of less macro voodoo.
Change-Id: I535c476f06b840858fa3070c4a237ece47f7a15b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10636
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Have a Evaluation::new() function that's used to set up the Evaluation
struct initially - which is also used by both new_pure and new_impure
internally.
It's generic over the exact type of IO, making it easier to instantiate
Evaluation with non-tvix-eval EvalIO implementations, that might not be
in a Box.
Change-Id: Ibf728da24aca59639c5b6df58d00ae98c99a63f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10640
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I lost a lot of hope and had to read the source code of `quote!`, `cargo expand` was invaluable
in this adventure. We should keep it IMHO.
Change-Id: Icfb4c80d413602f2bdc6deab0d595183825d88ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10635
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Don't restrict to a Box<dyn EvalIO>.
There's still one or two places where we do restrict, this will be
solved by b/262.
Change-Id: Ic8d927d6ea81fa12d90b1e4352f35ffaafbd1adf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10639
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
`throw (throw "a")` should work and propagate the internal throw.
Before this commit, it didn't work.
Change-Id: Id5d46f74e484dba99e912ad9fa211f3bf1617bac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10600
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
`elem` did not catch the list being a catchable.
This surfaced during Nixpkgs evaluation.
Change-Id: Icf19b94e914e35a435c4412d769ee63ba59ab7b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10599
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is an additional test suite on the top of the Nix ones
for context strings matters.
It already smoked out multiple mistakes and potential bugs and non-deterministic result from the evaluator.
It uses a similar technology as the one in the tvix-eval albeit we instantiate a fully fledged evaluator
with in-memory store.
We copy the files instead of symlinking them because crates are built in
isolation, so symlinks cannot work.
Change-Id: I63ae225ce4f83c6e2c8ccd60d779c2f8eb9d08fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10619
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Previously, we were assembling very naively an attribute set composed of context we saw.
But it was forgetting that `"${drv}${drv.drvPath}"` would contain 2 contexts with the same key, but
with different values, one with `outputs = [ "out" ];` and `allOutputs = true;`.
Following this reasoning and comparing with what Nix does, we ought to merge underlying values systematically.
Hence, I bring `itertools` to perform a group by on the key and merge everything on the fly, it's not
beautiful but it's the best I could find, notice that I don't use
`group_by` but I talk about group by, that is, because `group_by` is a
`group_by_consecutive`, see
https://github.com/rust-itertools/itertools/issues/374.
Initially, I tried to do it without a `into_grouping_map_by`, it was akin to assemble the final `NixAttrs` directly,
it was less readable and harder to pull out because we don't have a lot of in-place mutable functions on
our data structures.
Change-Id: I9933c9bd88ffe04de50dda14f21879b60d8b8cd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10620
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Yes, `hasContext e` should work where `e` is a contextful strings, otherwise, it is really useless.
Change-Id: I5eb071fc257217d6e8a63fe519132ebd98186696
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10617
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
`test-generator` has not been updated in the past 2 years.
`rstest` has not been updated in the past 5 months.
This is an improvement in the maintenance state… I guess?
We get also new features, it changes the name of the tests with numbers too.
Change-Id: I5376104c7704f525dba7524da78daa09867cc669
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10623
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We want to handle bottoms in a consistent fashion. Previously this was
handled by repetitive is_catchable checks, which were not consistently
present.
Change-Id: I9614c479cc6297d1f64efba22b620a26e2a96802
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10485
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Rather than passing strings around, use a StorePathRef.
This makes things a bit more typesafe, and more aligned with what we
want to do in b/264.
Change-Id: Ib7080addf27e7f1a9c8da1d8aaa66744468e3b5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10633
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We need to vendor in the package expression, as it's not possible to
override cargoHash.
Change-Id: Ib123647bb9b96d41f4630daa431d020f1cb8d4fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10624
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This starts a BuildService as a separate process, currently defaulting
to the DummyBuildService.
Change-Id: Ic206f00831641d3ffebaa44883b7dc053700b9ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10631
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This allows constructing a BuildService from a URI, similar to how it's
done in tvix-[ca]store.
Change-Id: Ib962b329535c6c7e378ab7ac7f4dd254366497b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10630
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Also provide a dummy implementation that just fails on any build that's
requested.
Change-Id: I0df743a730c5331ec9ce6e97a966abe18ce067f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10627
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Determining the inputs might trigger additional builds/substitutions,
so answering these lookups via a lambda in a lazy fashion gets
complicated.
You end up assembling the list of input nodes upfront, and the lambda
will just be a dumb lookup into that preassembled list.
Rather than doing that, simply have derivation_to_build_request leave
the work of determining the inputs to the caller.
Change-Id: I75880132916c76b930807c989090da298b6891bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10626
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This are leftovers from the "reference scanning" approach (which we
didn't end up using).
We still want a concept of known paths, so we can trace IO into
storepaths back to the build recipe that'll produce it, so let's keep
the rest of this struct around.
Change-Id: I73d38e21e5b97950b8fc2a42176cae5f80d371c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10632
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
A bunch of operations in Tvix are not aware of catchable values
and does not propagate them.
In the meantime, as we wait for a better solution, we just offer this
commit for moving the needle.
Change-Id: Ic3f0e1550126b0847b597dfc1402c35e0eeef469
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10473
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
`args` was not propagating context, here's a regression test for it.
Change-Id: I8b6a3148508d40df0077128f0bafe68c098a03bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10610
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds support to handle the __structuredAttrs argument, which can be
passed to builtins.derivationStrict.
If __structuredAttrs is passed, and set to true, most of the arguments
passed to builtins.derivationStrict are not simply coerced to a string
and passed down to "environments", but instead kept in a more structured
fashion.
Inside ATerm, which is what's relevant as far as path calculation is
concerned, a virtual `__json` environment variable is present,
containing these structured values.
Inside Builds, these structured values are not made available as an
environment variable, but a JSON file (and source-able bash script).
This will need to be respected once we start emitting BuildRequests,
and for that we can probably just parse the `__json` key in
Derivation.environment again - or keep this additionally in
non-serialized form around during Evaluation.
No matter what, this is left for a followup CL.
The existing handle_derivation_parameters and populate_outputs helper
function were removed, as __structuredAttrs causes quite a change
in behaviour, and so handling both in the same place makes it more
readable.
There's some open questions w.r.t. string contexts for structured attrs
itself. A TODO is left for this, but at least path calculation for
individual structured attrs derivations are correct now.
Part of b/366.
Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Allow other crates (like tvix-glue) to look at a Value in JSON, which is
used by the structured attrs feature.
Change-Id: Iba02ace6e11a74c3f9b19dcbef4b008b76dec046
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10602
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
My OCD could not be stopped.
Change-Id: I2bf504fe0865a5084ad02aee18e6180a8a3e19d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10609
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The Derivation input_derivations field contains a list of input
derivations and (a subset of their) output names.
This means, multiple nodes can be returned, so return a Vec.
Also, update the name to better reflect the nodes are the nodes of the
selected outputs, not a node representing the .drv file itself.
Additionally, use a proto::node::Node (the naked enum), rather than
proto::Node, which wraps this in an optional struct field until
realizing the BuildRequest.
Change-Id: Iec5620b5d7ac0462f2c76acac4abcaeea2de0aad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10608
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Provide a store_path_to_node_sync function which uses the runtime handle
to block on the async function internally, but make store_path_to_node
itself async, so it can call async functions internally.
We'll use that later when triggering builds and waiting on their
results.
Change-Id: Idae9da7aa5b0878e0d3a2eba34ea2623e1ba84b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10607
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We don't need Arcs in most of the cases, we're fine with some container.
Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
To render NARs, we're fine with a simple AsRef to a BlobService and
DirectoryService. We just need to have the function pass back the
references, so we can reuse it after the recursion.
Change-Id: I8a1b899134ddda26cf14aa829a08383986101850
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
The error message is misleading. The errors we return can happen both
during serialization or deserialization, though the messages suggested
the latter only.
Change-Id: I2dafe17ec78ee75cab5937a3a81540fda3175eac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10603
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds support to retrieve a list of chunks for a given blob to the
BlobService interface.
While theoretically all chunk-awareness could be kept private inside
each BlobService reader, we'd not be able to resolve individual chunks
from different Blobservices - and due to this, not able to substitute
chunks we already have in a more local store.
This function allows asking a BlobService for the list of chunks,
leaving any actual fetching up to the caller (be it through individual
calls to open_read), or asking another store for it.
Change-Id: I1d33c591195ed494be3aec71a8c804743cbe0dca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10586
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Make it clear this is only used inside the scope.
Change-Id: Ie94f88d7f0fb58cd4bf9c2f1176000b272e6f2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10585
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We need to be a bit careful and pass the BlobService around (similar to
how we already do with the directory_putter), but that allows getting
rid of a bunch of annoying trait bounds.
We also stop spawning additional tasks where we can just use block_on.
Change-Id: If36de0ee947d2c779d20a384308241d2262d4764
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10580
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
The docstrings were not updated once we made the BlobService trait async.
There's no more need to turn things into a sync reader.
Also, rearrange the stream manipulation a bit, and remove the need to
create a new VecDeque for each element in the stream. bytes::Bytes
implements the Buf trait.
Fixes b/289.
Change-Id: Id2bbedca5876b462e630c144b74cc289c3916c4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10582
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This test serves as a minimal reproducer for output path calculation.
Derivations with the same name and output hash, but different build
recipe should end up with the same outPath.
However derivations with different name should end up with a different
outPath.
Change-Id: I555be59dd87ea675a0816188ed973f96c311e4e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
It was a `//` not a `///`.
Change-Id: Iee3e8c116d73b5dd8a41c027153714415a66695f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't need to spawn in all these places, we can just block_on
directly, this is all IO bound.
This also means, we don't need to clone any of the service handles
(except preserving clone-ability of the BlobService).
Change-Id: I7d90f4d6a263a98491caa071ada538a5197a5472
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10540
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This was wrongly named, it returns a specific node at a subpath.
Also, this code can be simplified a lot - we don't need to spawn
additional tasks, and can get rid of some clones too.
This is also where we need a certain build - so add some TODO to block /
fetch here.
Change-Id: Id26d7bd80f7a2095121e642b3f7716de78d6b6a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10539
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Extend our validation function to also check for the None case.
Change-Id: Ib75f880646d7fb3d66588f1988e61ec18be816a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10534
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Make this an `AsRef<dyn DirectoryService>`.
This helps dropping some Clone requirements.
Unfortunately, we can't thread this through to TvixStoreIO just yet.
Change-Id: I3f07eb28d6c793d3313fe21506ada84d5a8aa3ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
While doing the obvious thing, ensuring clippy doesn't complain on our
benchmarks and examples, this will also flag build failures on them,
and contrary to cl/10301 not pollute the /nix/store with stuff we can't
execute anyways.
Change-Id: I7ee1c2babbc67dac9794274d05bb9d6e1251ed01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10544
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
cl/10475 updated the API of tvix_eval::Evaluation, but didn't update
this benchmark to use that new API.
Also, fixes the docstring to no longer specify that there is a "given
snippet".
Change-Id: Ibb8285731849dbeec814e2585bbaa36f22368afe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10542
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
With context strings, we are now able to reproduce up to firefox.
Until a more problematic example can be found, we should at least
lean on this particular example.
Change-Id: Ibeaf799d26e1f160b6c93f8ccd978702a2a7e3e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10460
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Raw printing will transform the result into a string
and print it.
In case of a contextful string, this will fail by default, as expected.
Change-Id: I5e564329e7b001adc57a77a9153b4425cb332bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10457
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Nix does something like:
```cpp
NixStringContext context;
const auto str = state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.split");
```
And then do nothing with that context, therefore, we follow them and
make `split` aware of the context but still do nothing with it.
Change-Id: I4fee1936600ce86d99d00893ca3f64013213935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10428
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
We calculate the input context by performing the union of context
over all input of the derivation.
Then, we just pass the rest of it to the remaining machinery.
Finally, we re-emit an `outPath` and a `drvPath` containing the expected
contexts.
Change-Id: I74905fb258b5bee8b08d1208c9eb87f51b92a890
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10436
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This removes the reference tracking and uses instead the context for
references and returns some.
Change-Id: Ic359ca6b903b63f1a9c679c566004c617b792442
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10435
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
In the past reference tracking system, `tvix-io` glue was appending
plain paths in the known path state.
Now, we make up for this by just making contextful coercion of file
imports.
Change-Id: Ieb9b04dd83302c77909252d5f7733857ac3cf8fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10443
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
String with contexts are always coerced to a string with the same
context.
Change-Id: I224814febd9cad196bb28876793e76bed564dc72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10440
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: sterni <sternenseemann@systemli.org>
Otherwise, you just fail because they are not... contextless strings!
Change-Id: I0b8f63a18cd89c3841b613d41c12ec4ee336f953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10442
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
`match` silently ignore the input context and do not propagate
it and successful matches.
The why is unclear but nixpkgs does rely implicitly on this behavior
because dynamic attribute selection cannot be done with contextful
strings.
Change-Id: I5167fa9b2c2db8ecab0c2fb3e9895c9cfce6eeb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10441
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
And it also preserve the original context if it exists.
Change-Id: I904f7c13b7f003a267aace6301723780fccaafb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10434
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
We just perform union of contexts of every pieces.
Change-Id: Ief925c1818cd8bbec0503e9c625b0630feebfdda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10432
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
`substring` has a very funny behavior when it comes to empty strings,
it propagates the context too, this is used in nixpkgs to attach context
to strings without using any builtin: `lib.addContextFrom`.
Change-Id: Id655356799b3485f7519b3d1914c630f9d8416c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10448
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
We make a case for adding a `reject_context` `CoercionKind` here.
It does happen during concatenation actually for path concats.
Change-Id: I0c196aad917550b9bcd0896cd2127a94f8181ffb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10444
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
I am still undecided whether we need a CoercionKind to control
the coerced context, here's a simple attempt.
Change-Id: Ibe59d09ef26c519a6acfdfe392014446646dd6d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10426
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
This extends derivation_to_build_request to handle passAsFile the same
way Nix does, and adds a unit test for it.
I opted to making this function fallible (if passAsFile contains a
non-existent env var), rather than pushing all of this into the
Derivation validate function.
Change-Id: I75b635f1f6c0c78d72b9a8fc7824f77e97b69951
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10522
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
While this can be influenced with `OTEL_SERVICE_NAME` or
`OTEL_RESOURCE_ATTRIBUTES`, "unknown_service" is an annoying name.
Let's pick tvix.store as a default.
Change-Id: I9b5c45703d9546775bb9266b29ab54daf7c54a9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10524
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Prevent code in this function from accessing the Vec, realize it only
when returning the BuildRequest struct.
Change-Id: I385c1adca552c9cb261cd70c5ca3ec45f0d7a9c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10521
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This already succeeds, as the output path calculation path does not need
to be aware of any builder-specific custom handling for passAsFile.
Change-Id: I9cc9e4e4351cdeaa3ec33ba58ee1569e7a368150
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10520
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Nix builds also have some other inputs, outside of inputs_dir, used in
derivations setting `passAsFile` and `__structuredAttrs`.
This extends our `BuildRequest` structure to accomodate these usecases.
Change-Id: I8de6516eb467fa48a4961d88933ebd548f131049
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10519
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
And include a test to ensure we show the warning.
Change-Id: Ib6a436dbba2592b398b54e44f15a48d1aa345099
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10470
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
In the past, we had `emit_warning` be no-op and we used `push_warnings` exclusively
but as we have consumers of this function, we need it to work somewhat.
Change-Id: I78a5ece199a473dec9ef5ea1fae60b36e35137b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10477
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Nix is quite tolerant when it comes to parsing SRI hashes and their
padding (and only for SRI hashes, it funnily is strict about that in the
non-SRI-hash case).
Nix essentially accepts any number of padding characters, no matter if
it's too much or too little. So we do the only sane thing - simply strip
all padding characters, and parse it with BASE64_NOPAD and the length
the algo uses.
Change-Id: I6a721aa289b06cc36741589792b9dd4c4f930b86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10468
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is behind the otlp feature flag (currently enabled by default).
By default, this will try to push traces to a OTLP collector running at
https://localhost:4317.
You can make one available by running:
```
docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-e COLLECTOR_OTLP_ENABLED=true \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
-p 14250:14250 \
-p 14268:14268 \
-p 14269:14269 \
-p 9411:9411 --rm \
jaegertracing/all-in-one:1.42
```
Started like that, jaeger brings a web interface at
http://localhost:16686/search
As documented in
https://docs.rs/opentelemetry-otlp/latest/opentelemetry_otlp/, you can
point this to another location by setting `OTEL_EXPORTER_OTLP_ENDPOINT`.
Change-Id: Id1dca367d70027b2ea98bb70bcf99a68363ec2be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8194
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: aaqaishtyaq <aaqaishtyaq@gmail.com>
Only convert to and reuse an Arc<…> where needed.
Change-Id: I2c1bc69cca5a4a3ebd3bdb33d6e28e1f5fb86cb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10514
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
