No situation should be allowed in which a path is inserted into
known_paths with different types twice, which we previously enforced
only for some path types.
Change-Id: I8cb47d4b29c0aab3c58694f8b590e131deba7043
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7843
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Instead of having a representation of suspended native thunks that
involves constructing a fake code chunk, make these thunks a
first-class part of the internal thunk representation.
The previous code was not that simple to understand, and actually
contained a critical bug which could lead to Tvix crashes. This
version fixes the particular instance of that bug, but instead
uncovers another (b/238) which can still lead to Tvix crashes.
Fixes: b/237.
Change-Id: I771d03864084d63953bdbb518fec94487481f839
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7750
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is unnecessary, Rc already provides all the boxing we need.
Change-Id: I08cf0939c48da43f04c847526c7e5dae5336d528
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7749
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
This is a somewhat terrifying hack that enables us to support
`builtins.builtins`, by running a "fake compilation" inside of a
suspended native thunk that can resolve the weak pointer to the
globals.
With this implementation, the thunk at `builtins.builtins` actually
resolves to the "real" `builtins` (verified with a new test).
This is kind of ugly, and it's something users shouldn't use, but
bubbling a warning out of this is difficult at the moment due to a
little bit of trickery with how the spans in suspended native thunks
work (they don't) (see b/237, b/238)
Change-Id: I67d0e93246dd5b279c960aeda00402031aa12af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Replacement strings are some weird internal feature of Nix that is
required for calculating derivation hashes. We need to track these
like other paths, as they need to be re-used on builds with
dependencies on values from previous builds.
Change-Id: Ie955b3fb5ae3685cfadfbe4d06ea6b5e219590c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7828
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
When adding things to a C++ Nix store, ensure that the path is tracked
in the tracker.
Since the mechanism for propagating the tracker instance isn't
finalised yet, I've opted to take an Rc<RefCell> parameter for it. How
exactly that ends up there is going to become clear in the next
commits, but for now it's just instantiated in main with
Default::default.
Change-Id: I90f0b44f2d4f292dedc98ff1aa39041d279b61fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7833
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This gets very complex very quickly otherwise, as all the construction
paths for a reference scanner and all the access patterns for the
KnownPaths structure are not yet fully understood.
Change-Id: Ibadf1f18b476695f3c286fc6896ae557760edf63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7827
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Instead of going through Vec/BTreeMap for generating our internal
types, use the proptest strategies from imbl.
The one thing I couldn't figure out in the previous implementation is
where the ranges/sizes of generated collections came from. The
strategies in proptest use different types (Range, with an unknown
default value, and SizeRange with 0..100). I've opted to specify
0..100 directly, but we can probably make it configurable.
Change-Id: I749bc4c703fe424099240cab822b1642e5216361
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7791
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This module implements types used to track the set of known paths in
the context of an evaluation.
These are used to determine the build references of a derivation.
Change-Id: I81e15ae33632784e699128916485751613b231a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7816
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
These need to be sorted anyways, so let's use the correct data structure for it.
Change-Id: I009c9989d7647dc1df716170f3680c981db6e4b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7846
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Different to the go-nix implementation, we don't have a custom "writing
mode" where output paths are omitted from output, but we simply run the
serialization with these fields unpopulated (during construction).
This means, there's no need for a custom writing mode that strips
outputs, so we can drop the comments.
Change-Id: Ic0aafd4e34e0294603490cfce2b1aef4085ff34b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7845
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This variant is required for external builtins (which in our case
includes `derivation`) to thread through reasonable error messages.
This has some potential for improvement, but it's an improvement over
the status quo of panicking in the external builtins when no
appropriate error is available.
Change-Id: I7e4bdb0a156c7717092dde30aa4785192182dc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7841
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
External implementors of builtins must be able to force values, which
necessitates publishing a bunch more items from the crate.
Change-Id: I8f6b8ae88156aae417dbe630a698d123d0c1c8d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7830
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This was put together by digging through the C++ Nix git history as
described in the second paragraph. Note that this is only based on
changes given when the language version was actually increased.
There is no guarantee that there have been no other changes inbetween
that would have warranted an increase as well.
Change-Id: I4ddee0d4ecafa1b3e5e1a867e9700d6c32e936ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7836
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
When constructing derivations inside builtins.derivationStrict, we'd
have to very frequently check whether certain outputs have already
been inserted into the derivation inputs.
Using a set type is much easier, especially as this has to be ordered
and the source data that is being inserted also comes from a set,
which might let us pass this more efficiently in the future.
Note that the validate function no longer checks the order of the
entries, as that is now guaranteed by the type.
Change-Id: I2fbb984facba3e668075f6f8df8992092368c63d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7826
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This CL addresses clippy warning len_without_is_empty
which expects `.is_empty()` method to be present when
implementing `.len()` method for an item.
Change-Id: I8878db630b9ef5853649a906b764a33299bb5dc8
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7806
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This explains my current thinking on string contexts. Thanks to
everyone who gave input so far.
Change-Id: I773219402a79a9d4753b4e7cfbf3a4a751a993a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7807
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Implements `Serialize` for `tvix_eval::Value`. Special care is taken
with serialisation of attribute sets, and forcing of thunks.
The tests should cover both cases well.
Change-Id: I9bb135bacf6f87bc6bd0bd88cef0a42308e6c335
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Using yet more machinery from the pretty comprehensive aho_corasick
crate, this makes it possible to pass anything implementing `io::Read`
to the `ReferenceScanner` to accumulate matches.
Change-Id: I5b0e28eb44ea4df24010f40831e29f2cbb8c1f80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7810
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This module implements a ReferenceScanner struct which uses the
aho_corasick crate to scan string inputs for known, non-overlapping
candidates (store paths, in our case).
I experimented with several different APIs, and landed on this version
with an initial accumulator in the scanner. The scanner is
instantiated from the candidates and "fed" all the strings, then
consumed by the caller to retrieve the result.
Right now only things that look vaguely like bytestrings can be fed to
the scanner, there is no streaming support in the API yet.
Change-Id: I7782f0f0df5fc64bccd813aa14712f5525b0168c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7808
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This CL address clippy warnings related to use of 'format!' macro
to return unmodified 'String'.
Change-Id: I88726e59d8f39f6a455a8c1f48075b52d167e489
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7804
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This just shuffles the Display implementations around so that
ErrorKind itself is displayable, which is useful in some situations
where errors under construction need to be type-converted.
Change-Id: I7b633d03d0dc34f345c4f20676e0023ecb1db0c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7802
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Some of the fields in a Derivation struct stay empty, and manually
creating BTreeMap or vec for it is annoying.
Derive Default instead, so we can use the defaults instead of writing
more by hand.
Change-Id: I5d41b4b55c8187cb101eb4266451a470008e0067
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7788
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This placeholder should not live in the main crate anymore as we will
be injecting the real one from outside of eval, but there are still
language tests that depend on a (simple, mockable) version of it.
Change-Id: I68ea169db15cbdbeed320930d3069e21e376c90d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7783
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Instead of concatenating STORE_DIR manually, use
StorePath::to_absolute_path() that does it for us.
Change-Id: Ia288851a05b4e339679db268f3dd7924e7b65586
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7782
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This project was not previously covered by CI (fixed in this commit),
so we didn't catch breakage due to a renamed module.
This was noticed while rebasing a CL that has a dependency on this
crate in its Nix build.
Change-Id: Ic48570b9313e5f73e14daab50cf7ea70918c94d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7778
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This embeds the build config directly at the point where `Cargo.nix`
is imported, making it transparent to library consumers.
Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
As discussed in #tvl, this is a more common term for it.
Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This exercises the output path calculation functions like a constructing
client (an implementation of builtins.derivation) would do.
It first assembles the bar derivation, does the output path calculation
on it, then continues with the foo derivation.
The code ensures the resulting Derivations match our fixtures.
Change-Id: If93f89c6622fac1c1941085083931b6f657c04bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7775
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This provides a function returning a string starting with the store
path prefix, the counterpart of `from_absolute_path`.
Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This will return the fixed output of a derivation (and its hash), or
None if the Derivation is not fixed-output.
It will simplify the logic in the output path calculation a bit.
Change-Id: I1066cc18ee4fc419421a8c5995c93ba91b35588f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7760
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This moves all the hash compression logic into a common helper function.
Also update the docstring, which said "path" here, which could have been
confused with output paths.
Change-Id: Iedfb59aeb24f7638afac669dcd18d57b6cfaece2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7759
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is marginally more efficient and has simpler bytecode.
Change-Id: Iad37c9aeef24583e8f696911bcd83d43639f2e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7769
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a mechanism to the compiler to compile an expression without
emitting any code. This allows for detected dead code to still be
compiled to detect errors & warnings inside of it.
Change-Id: Ie78479173570e9c819d8f32ae683ce34234a4c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7767
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
