The underlying protos were changed to return `[]byte` instead of `string`.
Change-Id: I5d3e5d8de0ed7200325f7ab0d62e3c10d8eb1b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9285
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Included changes:
* tvix/eval: enable some lang tests on nix_latest
Nix 2.16 contains some breaking language changes which Tvix does
not yet implement, but the existing tests for them are now passed by
Nix 2.16 (but not yet by Tvix).
* tvix/eval: disable a lang test on nix_latest
In Nix 2.17, the identifier formatting test fails because some
behaviour changed. We have not investigated further yet.
* 3p/overlays: use version of ihp-hsx that works with GHC 9.4
Originally from the separate cl/9185.
* top-level: introduce a mechanism to exclude build targets from CI in
the top level. This fixes b/296.
* users/grfn: disable builds of xanthous (and dependents) until the
CLs fixing its build are submitted
* 3p/overlays: build nixos-option against Nix 2.15, the only version
with which it builds
* 3p/overlays: bump tdlib to 1.8.16
Change-Id: Ia377f39dbdb08ac45ff830a615e64babc091e5ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9125
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This makes the inside code a bit less verbose.
I wasn't able to describe the type of the async move closure itself,
which would allow us to remove the JoinHandle<_> type annotation
entirely.
Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This enables the tokio `signal` feature, and registers a ctrl_c signal
handler, which will use the unmount handle to unmount in case a ctrl-c
signal is received.
This avoids having disconnected mountpoints when Ctrl-C'ing a
`tvix-store mount` invocation.
In case the filesystem is unmounted externally (via `umount /path/to/
mountpoint`), the future is waiting for the signal is never resolved and
the task is stopped.
Change-Id: I149f705a6cb50188177f2a6c6a5fcd77218e2a3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9218
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This provides an additional configuration flag to the tvix-store mount
subcommand, and logic in the fuse module to request listing for the
root of the mountpoint.
Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: adisbladis <adisbladis@gmail.com>
Tested-by: BuildkiteCI
This provides an additional method in the PathInfoService trait, as
well as an RPC method on the gRPC layer to list all PathInfo objects in
a PathInfoService.
Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We don't need to explicitly describe the type of the task itself,
describing the return type of the async closure is sufficient.
Also, use io::Result<_> instead of Result<_, io::Error>.
Change-Id: I9ab3f990eb49929b0aea335b2bb07da392ab631f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9267
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
https://github.com/BurntSushi/walkdir/pull/170 got merged, meaning we
don't need to keep our own logic in here anymore.
Our test cases already cover this.
Change-Id: Ied3043ee651c8aafa10271c1e1ca5d460fb6c0b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9269
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This trait is eval-specific, there's no point in dealing with these
things in tvix-store.
This implements the EvalIO interface for a Tvix store.
The proper place for this glue code (for now) is tvix-cli, which knows
about both tvix-store and tvix-eval.
There's one annoyance with this move: The `tvix-store import` subcommand
previously also used the TvixStoreIO implementation (because it
conveniently did what we wanted).
Some of this code had to be duplicated, mostly logic to calculate the
NAR-based output path and create the PathInfo object.
Some, but potentially more of this can be extracted into helper
functions in a shared crate, and then be used from both TvixStoreIO in
tvix-cli as well as the tvix-store CLI entrypoint.
Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Don't repeat the name of the method in the description, don't repeat
things already described in request message comments.
Change-Id: I180e4792577419050947eea8fea7043861aba463
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9213
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This returns a node with a new name.
Change-Id: Iebcab537f8dd63d826b9841d4d0181fcb941afdd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Since the refactor to use URIs for all three services, this actually
does talk to a daemon by default.
Change-Id: Ied296772b77eef514bfcae0a9dfc50f848a1c2f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9210
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
replaceStrings would previously fail to replace the last character
in a string.
Change-Id: I43a7c960945350b2e7a5b731b7fdb617723eb38f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9151
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
We already have these hashes accessible in the Cargo.nix file created
by cargo2nix, so there's no need to also manually maintain them here.
It removes one potential footgun I ran into while updating wu-manber to
a different rev, without updating it here.
Change-Id: I93932ac8ba55f83746ee38571b7740af2d49bbdf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9090
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
There was a NixHash::new() before, which didn't perform any validation
of the digest length. We had some length validation when parsing nix
hashes or SRI hashes, but some places didn't perform validation and/or
constructed the struct directly.
Replace NixHash::new() with a
`impl TryFrom<(HashAlgo, Vec<u8>)> for NixHash`, which does do this
validation, and update constructing code to use that, rather than
populating structs directly. In some rare cases where we're sure the
digest length is correct we still populate the struct manually.
Fixes b/291.
Change-Id: I7a323c5b18d94de0ec15e391b3e7586df42f4229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9109
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This question popped up every once in a while. While already explained
quite well at
https://inbox.tvl.su/depot/20230316120039.j4fkp3puzrtbjcpi@tp/T/#t,
it's not easily accessible.
Lift it from there into tvix/docs for better visibility.
Change-Id: I5f2d4aff31ab4adc421e06a7d36c871f45e09100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9080
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This passes a unit value to the function.
Change-Id: I4df3ad8fb0f35c0f110cee3349971ae28ce2878c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9101
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We already have the parsed output_hash from above, no need to construct
it again.
Change-Id: Ie6d924ab446137c25c29fbeaf671aa7e5418262d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9110
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Previously, Output deserialization would silence validation errors and
provide `None` for `hash_with_mode` as soon as a validation error would
happen inside of the `NixHashWithMode` deserialization, e.g. invalid
hash length would not provide an validation error but a silent `None`
value.
This is problematic, we workaround a serde limitation here by writing
our own Deserializer.
As you can see, we write some boilerplate code unfortunately, as, for
example:
- `#[serde(fail_if_unparsed_as="Option::is_none")]` is not a thing,
otherwise, we could have been able to just bubble up errors in case of
"not fully parsed" (and not missing) values.
- `From<&serde_json::Value> for serde:🇩🇪:Unexpected` is not a thing,
otherwise, we could just map invalid type errors and reuse the
existing types instead of doing extremely bizarre things with
`serde:🇩🇪:Unexpected::Other`, note: this is a not problem for
expected, we know what we expect, we don't know what we received in
practice.
I decided to write a `NixHashWithMode::from_map` which will eat a map
deserialized via `serde_json`, so our serde magic is not totally "data
model" agnostic.
I wanted to go for data model agnosticity and enable maximal
performance, e.g. building the structure as the map values are streamed
in the Visitor, this is needlessly painful because `Output` and
`NixHashWithMode` are in different files and this really makes sense
only if we write the full implementation in one file, indeed, otherwise,
you end up duplicating the work or having strange coupling.
So, for now, we will allocate a full map of the fields inside the
`Output`, i.e. if any "unknown field" is in that map, we will
deserialize it for no reason.
Doing it properly, as I repeat it in the code and to flokli at C3Camp
2023, requires to patch serde upstream IMHO.
Change-Id: I46fe6ccb8c390c48d6934fd3e3f02a0dfe59557b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9107
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't need rust-analyzer to run cargo doc.
Change-Id: I5e2fd559e4045cadeab24b438c28d6df7f1d5d5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9092
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
cl/8306 fixed building all docs, but we forgot to update the comment.
Change-Id: I17829612f13e7357bd0efe8223cc28ed0f6cdea2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9091
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This `.into_iter()` call is equivalent to `.iter()` and will not consume
the `BTreeMap`.
Change-Id: Ie26637ebecb0bea5b09c447cc45ed207f8b50913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9088
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We only need bstr::ByteSlice to be able to use replace, it doesn't need
to return a BString.
Change-Id: I811948436fb89652e880970c2c05356183f3e439
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9084
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
builtins.div ought to truncate towards zero so that
-(builtins.div a b) == builtins.div (-a) b
-(builtins.div a b) == builtins.div a (-b)
Change-Id: I8b7c08cd7f4fa8a1363c786d42c8d484f6cd133d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9006
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.
I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.
Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Make it more obvious if these are bytes pointing to JSON or ATerm, so we
don't get confused.
Change-Id: I2402c687b7ba9c05aac20ed63b0df54e4e96a9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8998
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This is more concise than a io::copy of a Cursor to bytes, and we have
everything to be written in memory.
Change-Id: I81f34666aa61aef4e16b33423ce4a69c3781efc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8997
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
write_input_derivations shouldn't need to write a comma to separate it
from the previous output from write_outputs.
This is better placed in the function calling all of these helper
functions.
Change-Id: I9ccc440e4665b52369ef39e75151b9a29469ce48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8995
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We're happy with any &[S], as long as <S: AsRef<[u8]>.
This allows passing both strings and &[u8].
Change-Id: If2a80d9b1ee33ba328c9cdab4fa83ca7b98a71e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8994
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Let the escape function only take care of string escaping, not quoting.
Let write_array_elements always quote and escape strings it consumes.
Move the business of writing additional wrapping characters around it to
the caller.
Change-Id: Ib8dea69c409561b49862c531ba5a3fe6c2f061f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8993
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Derivations can have non-unicode strings in their env values, so the
ATerm representations are not necessarily String anymore, but Vec<u8>.
Change-Id: Ic23839471eb7f68d9c3c30667c878830946b6607
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8990
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This allows sorting Store Paths. We delegate the sorting business to the
PartialOrd, Ord impls for our digest fields only, as two StorePaths with
the same digest, but different names can't exist.
Change-Id: I5f81631e5f5063893b316c63a240c5266b7e5bad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8988
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Add the position in the string where the name is problematic.
Change-Id: If6fd8be6100b718f8d68568eafc77ebb3cfb82d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8979
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This will save us some copies, because a clone will simply create an
additional pointer to the same data.
Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Some paths might use names that are not valid UTF-8. We should be able
to represent them.
We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.
While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?
Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This "reverts" commit 9f600de226 (the
initial revert of f5e291cf83).
Now with BlobService returning a BlobReader that implements io::Seek, we
can actually just call blob_reader.seek(io::SeekFrom::Start(offset as
u64)).
This means, we currently will fail to seek backwards inside a file.
Change-Id: I9c19448df6831a3537252f99210374f2126ecfc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8886
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
For memory and sled, it's trivial, as we already have a Cursor<Vec<u8>>.
For gRPC, we simply reject going backwards, and skip n bytes for now.
Once the gRPC protocol gets support for offsets and verified streaming,
this can be improved.
Change-Id: I734066a514aed287ea3db64bfb1680911ac1eeb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8885
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The primary constructor for this is now from_bytes, from_string is
simply calling .as_bytes() on the string, passing it along.
The InvalidName error now contains a Vec<u8>, to encode the invalid name
(which might not be a string anymore).
from_absolute_path now accepts a &[u8] (even though we might want to
make this a OSString of some sort).
StorePath::validate_name has been degraded to a pub(crate) function.
It's still used in src/derivation, even though it probably shouldn't at
all - that cleanup is left for cl/8412 though.
Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This being a nested error makes things more complicated than necessary.
Also, this caused BuildStorePathError to only hold NameError,
so refactor these utility functions to either return Error, or
BuildStorePathError.
Change-Id: I046fb403780cc5135df8b8833a291fc2a90fd913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8972
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This explicitly documents behavior of C++ Nix that goes against the
intuition you'd gather from this document: that e.g. a simple select
from an attribute set causes a value to no longer be pointer equal to
its former self.
The point of documenting this is that we can show in a to be written
section on the use of pointer equality in nixpkgs that pointer equality
is only needed in a limited sense for evaluating it (C++ Nix's exterior
pointer equality). Tvix's pointer equality is far more powerful since
value identity preserving operations also preserve pointer equality,
generally speaking (this is because we implement interior pointer
equality in my made up terminology). This should eventually also be
documented.
Change-Id: I6ce7ef2d67b012f5ebc92f9e81bba33fb9dce7d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8856
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
This fixes a subtle issue which would occasionally lead to a crash (e.g.
when evaluating (pkgs.systemd.outPath with --trace-runtime): With each
character in the string that has a multi byte representation in UTF-8,
the actual byte position and what tvix thought it was would get out of
sync. This could either lead to
* Tvix swallowing characters or jumbling characters if multi byte
characters would cause the tracked index to become out of sync with
the byte position before the first character to be escaped, or
* Tvix crashing if (in the same situation) the out of sync index would
be within a UTF-8 byte sequence.
Luckily, std's `char_indices()` iterator implements exactly what
`nix_escape_char()`'s original author had in mind with
`.chars().enumerate()`. Using `i + 1` for continuing is safe, since all
characters that need (in fact, can) to be escaped in Nix are represented
as a single byte in UTF-8.
Change-Id: I1c836f70cde3d72db1c644e9112852f0d824715e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8952
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Improve change some little things I noticed while reading through it.
Change-Id: I033209eece395e5aad4e10825e8dd6c0cfe68191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8725
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
I want to expand on the C++ Nix behavior, since it seems relevant to
note that a lot of operations in C++ Nix (like select) don't preserve
pointer equality (see
<https://github.com/NixOS/nix/issues/3371#issuecomment-1596167957>).
It is especially so, as Tvix establishes pointer equality in a different
way and thus shows differing behavior. Therefore I want to additionally
document Tvix's current behavior and make it more explicit to what
extent nixpkgs needs pointer equality.
Change-Id: I9b4ba75dacb749c9fcbba4b9646c6b48bb57bbad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8852
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This reverts commit f5e291cf83.
The offsets are relative to the start of the file, and as long as we
don't have BlobReaders implement seek, this will be very annoying to
deal with.
Change-Id: I05968f7c5c0ec0000597da90f451d6bb650c3e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8882
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
buf contains everything written so far, whereas b is the slice passed in
the current write() call. If we copy from &buf, we end up with the wrong
hash, because we keep writing the wrong data to the hash function.
Change-Id: I768d4645934a6a7d75b9c8eeba35f8f3be5edd26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8880
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This moves from stateless I/O to actually dealing with file handles,
allowing the filesystem to keep reusing existing blobreaders, instead of
opening a new reader on every read() call.
Change-Id: I3fc35c071e4aee1021c8bbd58749d082b0abd188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8834
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
The change allows applications that use tvix_serde for parsing
nix-based configuration to extend the language with domain-specific
set of features.
Change-Id: Ia86612308a167c456ecf03e93fe0fbae55b876a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8848
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I've noticed this behavior when writing the admittedly cursed test case
included in this CL. Alternatively we could use some sort of machinery
using `builtins.trace`, but I don't think we capture stderr anywhere.
I've elected to put this into the eval cache itself while C++ Nix does
it in builtins.import already, namely via `realisePath`. We don't have
an equivalent for this yet, since we don't support any kind of IfD, but
we could revise that later. In any case, it seems good to encapsulate
`ImportCache` in this way, as it'll also allow using file hashes as
identifiers, for example.
C++ Nix also does our equivalent of canon_path in `builtins.import`
which we still don't, but I suspect it hardly makes a difference.
Change-Id: I05004737ca2458a4c67359d9e7d9a2f2154a0a0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8839
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is a first implementation of a FUSE filesystem, mounting tvix-store
to a given location.
This is mostly meant as one additional lens into a store, and could be
used for builds. It's not meant to be used as a general-purpose thing.
It still has some rough edges:
- It doesn't implement open/close, so it doesn't use file handles.
Which means, we need to open blobs for partial reads over and over
again.
- It doesn't implement seek, as BlobReader doesn't implement seek yet.
- It doesn't track "lifetimes" of inodes by listening on forget,
meaning it might hold more data in memory than necessary.
- As we don't have store composition (and a caching layer) yet,
operations might be slow.
Change-Id: Ib1812ed761dfaf6aeb548443ae939c87530b7be8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8667
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
When dealing with a formal argument in a function argument pattern that
has a default expression, there are two different things that can happen
at runtime: Either we select its value from the passed attribute
successfully or we need to use the default expression. Both of these may
be thunks and both of these may need finalisers. However, in the former
case this is taken care of elsewhere, the value will always be finalised
already if necessary. In the latter case we may need to finalise the
thunk resulting from the default expression. However, the thunk
corresponding to the expression may never end up in the local's stack
slot. Since finalisation goes by stack slot (and not constants), we need
to prevent a case where we don't fall back to the default expression,
but finalise anyways.
Previously, we worked around this by making `OpFinalise` ignore
non-thunks. Since finalisation of already evaluated thunks still
crashed, the faulty compilation of function pattern arguments could
still cause a crash.
As a new approach, we reinstate the old behavior of `OpFinalise` to
crash whenever encountering something that is either not a thunk or
doesn't need finalisation. This can also help catching (similar)
miscompilations in the future. To then prevent the crash, we need to
track whether we have fallen back or not at runtime. This is done using
an additional phantom on the stack that holds a new `FinaliseRequest`
value. When it comes to finalisation we check this value and
conditionally execute `OpFinalise` based on its value.
Resolves b/261 and b/265 (partially).
Change-Id: Ic04fb80ec671a2ba11fa645090769c335fb7f58b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8705
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
This adds a `from_str_with_config` function which takes a
user-supplied closure that sets additional settings on the
`tvix_eval::Evaluation`.
Note that users can not set `strict = false`, but other settings are
not restricted.
This solves b/262.
Change-Id: Ice184400b843cfbcaa5b6fe251ced12b6815e085
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8808
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Adds new tests for foldl', intersectAttrs as well as fills in missing
.exp files.
New test cases we don't pass:
- fromTOML timestamp capabilities
- path antiquotation
- replaceStrings is lazier on C++ Nix master
The C++ Nix revision used is 7066d21a0ddb421967980094222c4bc1f5a0f45a.
Change-Id: Ic619c96e2d41e6c5ea6fa93f9402b12e564af3c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8778
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
genericClosure has very limited support for pointer equality: It relies
on comparison (not equality!) in C++ Nix, so as soon as C++ Nix supports
comparing lists (langVersion >= 6) we can rely on pointer equality for
key.
Since Tvix uses equality, not comparison for the insert, our behavior is
currently different, as documented by the notyetpassing tests.
Change-Id: Ifcd741ed4fc3ccc3825f7038875d56a9918b786a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8720
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
In order for the test suite we have currently to be comparable to C++
Nix, we need to display values in the same way. This was largely the
case except in some weird cases.
* <CODE> for thunks and <CYCLE> for repeated thunks (?) are already in
use. <CODE> formatting is tested by the oracle test suite already.
* Instead of lambda, we need to use <LAMBDA>
* <<primop>> and <<primop-app>> (a formatting C++ Nix uses nowhere)
now are <PRIMOP> and <PRIMOP-APP>.
We'll probably want to have a fancier display of values (in a separate
trait) down the line. This could be used for interactive usage, e.g. the
REPL or a potential debugger.
There is a peculiarity with C++ Nix 2.3 formatting primops: import is
considered a <<PRIMOP-APP>>, since it is internally implemented by means
of scopedImport. This implementation detail no longer leaks in C++ Nix
2.13 nor in Tvix.
<CYCLE> display is untested at the moment, since we exhibit a
discrepancy to C++ Nix 2.3. Our current detection is more similar to C++
Nix 2.13—luckily it is also the more consistent of the two. See also
b/245.
Change-Id: I1d534434b02e470bf5475b3758920ea81e3420dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8760
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
A previous iteration of this code did actually connect (in the gRPC
client), which was why we had this function async.
However, as the connection there is now lazy too, we can drop the
asyncness in this function.
Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This allows constructing blob stores with a URL syntax at runtime,
by passing the --blob-service-addr arg.
We probably still want to have some builder pattern here, to allow
additional schemes to be registered.
Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
These were added by us in r/5276, so they should go into our test suite.
Change-Id: I6dc74fc242f33c22a17e0b4aee546ccae886ac85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
C++ Nix resolves home relative paths at [parse] time. This is not an
option for us, since it prevents being able to separate the compilation
and execution phase later (e.g. precompiled nix expressions). However, a
practical consequence of this is that paths expressions are always
literals (strict) and never thunks.
[parse]: 7066d21a0d/src/libexpr/parser.y (L518-L527)
Change-Id: Ie4b9dc68f62c86d6c7fd5f1c9460c850d97ed1ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7041
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
There's very little reason to instantiate a GRPCPathInfoService in a
context where we are not already in a tokio context.
Change-Id: Ib81d649387717cb98de8a8039f92472f727b10c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8755
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The only place where we did use new was also already where we've been in
a tokio context, so just using from_client is easier.
Change-Id: I39dbc18f6aaa3abc342409be623395647f968530
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8754
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
There's very little reason to instantiate a GRPCBlobService in a context
where we are not already in a tokio context.
Change-Id: Ic6e18809a9f2a76f1c098ed330118d8dcfba5137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8753
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This removes the use of generics, like previously done with Blob and
Directory services.
Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
We never returned Err here anyways, and we can still return an error
during the first (or subsequent) write(s).
Change-Id: I4b4cd3d35f6ea008e9ffe2f7b71bfc9187309e2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8750
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
From 64 bytes to 100 KBytes.
We need to provide a custom wrapper with a different Default instance.
Change-Id: Id7c6c437b8183b355a9e388f98cef1622b363f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This allows us to blob services without closing them before putting them
in a box.
We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers
require Sync.
Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Once we support configuring services at runtime, we don't know what
DirectoryService we're using at compile time.
This also means, we can't explicitly use the is_closed method from
GRPCPutter, without making it part of the DirectoryPutter itself.
Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Putting this in the PathInfoService trait makes much more sense, we can
have direct control over where/how to cache the results in the
implementation.
This now requires each PathInfoService to hold pointers to BlobService
and DirectoryService.
Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
There's only one way to calculate NAR files, by walking through them.
Things like caching such replies should be done closer to where we use
these, composing NARCalculationService doesn't actually give us much.
Instead, expose two functions, `nar::calculate_size_and_sha256` and
`nar::writer_nar`, the latter writing NAR to a writer, the former using
write_nar to only keeping the NAR size and digest.
Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
To construct various stores at runtime, we need to eliminate associated
types from the BlobService trait, and return Box<dyn …> instead of
specific types.
This also means we can't consume self in the close() method, so
everything we write to is put in an Option<>, and during the first close
we take from there.
Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Unfortunately, nixpkgs has at least one case[1] where the out environment
variable is shadowed -- though it doesn't cause a problem, since it's
shadowed with the correct value, odd as this may be!
[1]: c7c2984716/pkgs/development/python-modules/pybind11/default.nix (L19)
Change-Id: Ibf6790d2484dc9cce8e424feeb5886664d498dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8696
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
`tvix-store mount PATH` will mount the tvix-store to the given path.
Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This allows using a StorePath as a key in a hashmap.
Change-Id: Id3eed623da4e1fc44a970a3982c7caa21d2495c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8666
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This brings in fuse (via the `fuser` crate), and adds pkg-config and
libfuse to the dev shell, so `cargo build` can link against it.
Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This testcase tests a missing blob fails the rendering, the comment has
been copied from elsewhere.
Change-Id: I48fa3fa454e12506590fa14a3591d156bafa8b5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8722
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The CLs did bitrot a bit, they're based on an older version of the
protocol, and it's unclear if they'll be a separate Go Binary, or just
another HTTP handler inside tvix-store itself, considering we now have
way more NAR juggling code than before.
Change-Id: I3632035cda8d75a8ff23b3132312f0f086d9e02f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8732
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This has moved to a `daemon` subcommand.
Change-Id: Iae9778d8a59e6bf84555119fabfd62db3917bb62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8731
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
When comparing to C++ Nix, we notice that the thunking of default
expressions in function formals corresponds to their normal thunking,
e.g. literals are not thunked. This means that we can just invoke
compile() without much of a care and trust that it will sort it out
correctly.
If function formals blow up as a result of this, it likely indicates
that the expression is treated incorrectly by compile(), not
compile_param_pattern().
Change-Id: I64acbff2f251423eb72ce43e56a0603379305e1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8704
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
C++ Nix forces and typechecks the passed argument even if it is not
necessary in order to compute the return value of the function. I
discovered this when I thought our formals miscompilation might be that
we are too strict, but doesn't look like it in this case.
Change-Id: Ifb3c92592293052c489d1e3ae8c7c54e4b6b4dc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8701
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Before, the construction of a TwoByteWM would panic when no patterns
were provided, as in `tvix --expr 'builtins.toFile "snens" "soos"'`.
Change-Id: I25ed498c475523aec5baf8683b23059fadabb21c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8697
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This now creates different store client, depending on the cli
subcommand.
The `import` command will connect to the gRPC service, and the `daemon`
command will use the sled implementation.
It might make sense to define some URI syntax to make this configurable
by the user, via the CLI.
Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We don't need to be able to clone these services in here.
Change-Id: Ifb69450f7ebdc8364cbf9cdfb6464f8560440e4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8645
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
nix_oracle.rs now gives us the possibility to check this by stuffing the
expressions in a list. In fact, the incorrect behavior fixed in
- cl/8656
- cl/8655
- cl/8662
was discovered using this test suite.
Change-Id: Id0ab01ee6be0b875791214e0a72a2ac941c46c96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8658
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This will be useful for comparing thunking behavior to C++ Nix. I
considered adding this capability to the tvix_tests/nix_tests
infrastructure, but as it would require changing the test file naming
scheme to do it in a clean way, I've postponed it–it's nice that our
tests are compatible with C++ Nix's test suite.
Change-Id: I60bcdd98ed25140e716f0858f8dc28f21ab957aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8657
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
HasAttrs was weird because with longer attribute paths it would
sometimes not turn out to be a thunk. If it was a thunk, it'd usually
still do some eval strictly which we'll want to avoid.
Verified against C++ Nix using a new test suite introduced in a later
CL.
Change-Id: I6d047ccc68d046bb268462f170a3c4f3c5ddeffe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8656
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Probably no real world code broken by this overzealous evaluation, but
let's be thorough!
Change-Id: Ib405a677182eab7940ace940c68e107573473a54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8655
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Unary operator applications are thunked which can easily be observed by
nix-instantiate --eval -E '[ (!true) (-1) ]'
Unfortunately, there are few simple expressions where this makes a
difference in the end result. Thus it only cropped up when using nixpkgs
for cross compilation: Here we would compile the expression
!(stdenv.cc.isGNU or false)
to assemble python3Minimal's passthru attribute set (at least this seems
to be the most likely explanation from the backtraces I've studied).
This means that an unthunked
<stdenv.cc.isGNU or false>
OpForce
OpInvert
would be performed in order to assemble this attribute set, causing
stdenv.cc to be evaluated too early, causing an infinite recursion.
Resolves b/273.
It seems that having a test suite that doesn't use --strict and relies
on thunks rendered as <CODE> would be beneficial for catching such
issues. I've not been able to find a test case with --strict that
demonstrates the problem fixed in this CL.
Change-Id: I640a5827b963f5b9d0f86fa2142e75e3a6bbee78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8654
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
This will prevent tvix from printing any warnings.
As a followup, we can also thread this parameter through into the
evaluator itself, to prevent warnings from being constructed in first
place.
Change-Id: I15381396f86573484bdd1a73d09034a665638e35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8646
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
mapAttrs, map and genList call Nix functions provided by the caller and
store the result of applying them in a Nix data structure that does not
force all of its contents when forced itself. This means that when such
a builtin application is forced, the Nix function calls performed by the
builtin should not be forced: They may be forced later, but it is also
possible that they will never be forced, e.g. in
builtins.length (builtins.map (builtins.add 2) [ 1 2 3 ])
it is not necessary to compute a single application of builtins.add.
Since request_call_with immediately performs the function call
requested, Tvix would compute function applications unnecessarily before
this change. Because this was not followed by a request_force, the
impact of this was relatively low in Nix code (most functions return a
new thunk after being applied), but it was enough to cause a lot of
bogus builtins.trace applications when evaluating anything from
`lib.modules`. The newly added test includes many cases where Tvix
previously incorrectly applied a builtin, breaking a working expression.
To fix this we add a new helper to construct a Thunk performing a
function application at runtime from a function and argument given as
`Value`s. This mimics the compiler's compile_apply(), but does itself
not require a compiler, since the necessary Lambda can be constructed
independently.
I also looked into other builtins that call a Nix function to verify
that they don't exhibit such a problem:
- Many builtins immediately use the resulting value in a way that makes
it necessary to compute all the function calls they do as soon as
the outer builtin application is forced:
* all
* any
* filter
* groupBy
* partition
- concatMap needs to (shallowly) force the returned list for
concatenation.
- foldl' is strict in the application of `op` (I added a comment that
makes this explicit).
- genericClosure needs to (shallowly) force the resulting list and some
keys of the attribute sets inside.
Resolves b/272.
Change-Id: I1fa53f744bcffc035da84c1f97ed25d146830446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8651
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Instead of instantiating it once in every loop iteration, put it in an
Arc, and clone that before passing it to the spawned task.
Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This feature allows the compiler to detect situations where the
created thunk is useless and can be merged into the parent chunk
instead.
The only case where the compiler does this initially is when
statically optimising a select expression.
For example, previously the expression `builtins.length` compiled into
two thunks:
1. An "inner" thunk which contained an `OpConstant` that had the
optimised `length` builtin in it.
2. An "outer" thunk which contained an `OpConstant` to access the
inner thunk, and the trailing OpForce of the top-level program.
With this change, the inner thunk is skipped completely and the outer
chunk directly contains the `length` builtin access.
This can be applied in several situations, some easier than others,
and we will add them in as we go along.
Change-Id: Ie44521445fce1199f99b5b17712833faea9bc357
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7959
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This method extends the contents of one chunk with that of another,
effectively merging the thunks together.
This will be used for the upcoming "unthunking" functionality.
Change-Id: I6ad74232cd7f3eca198ed921e455205e00d76e6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7958
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
With traverse_to not requiring a &mut anymore, we can drop the &mut self
in all these function signatures.
Change-Id: I22105376b625cb281c39e92d3206df8a6ce97a88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This operation is blocking, so it should be run inside a blocking tokio
task. Tokio panics if it detects a blocking operation inside a non-
blocking task, so cl/8619 would cause it to panic (as the GRPC clients
use spawn_blocking under the hood).
As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614),
we create a new instance of TvixStoreIO inside each loop iteration.
Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This switches tvix-cli over from using `NixCompatIO` to `TvixStoreIO`.
For now, we simply instantiate in-memory services instead of getting
fancy with constructors inside tvix-store, but long-term, we might want
to support some URI syntax, to make this configurable at runtime.
nixpkgs eval tests might be fine (and fast!) with a purely in-memory
backend, but other usages might involve talking to a local tvix-store
over gRPC (using the gRPC client, either unix domain socket or even
further away remote), or running tvix-store in "embedded" mode (using
another client than the gRPC client).
Change-Id: I509afd3dc5ce3f2d52b0fb7067748fab820e26ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8572
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows closing a TvixStoreIO if it contains a SledPathInfoService.
Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We already evaluate this the same as Nix does, so worth adding it into
CI.
Change-Id: I529faccac6e5e16e0bc985ab4c3e0cd07bb23293
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8624
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This introduces a function that can be instantiated with an attribute
path to instantiate, as well as the expected path (normally interpolated
with the nix evaluator).
Check both pkgs.stdenv.drvPath and pkgs.stdenv.outPath to match.
Change-Id: Id667ed35fa159ff83fedb3017ef8d3271aa42695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8606
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We have code.namespace as a field, set to
tvix_store::directoryservice::sled, so there's no need to repeat the
name here.
Change-Id: Ic1aa8a2b24de439c6a189966bd773e9acf49d1e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8623
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This asks a remote tvix-store for the nar size and digest of a given
root node.
Change-Id: If9f916d9bfc5f8dc3166e2c6c1671c0f0124d1c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8611
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This introduces a new struct, B3Digest, which internally holds a
Vec<u8>, but only allows construction with 32 bytes.
It also implements display, which will print the base64 representation.
This should reduce some boilerplate when parsing Vec<u8>.
Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
These two places didn't add the path from the context to the
ErrorKind, but simply relied on the
impl From<std::io::Error> for tvix_eval::ErrorKind, which doesn't add
the path.
Change-Id: Ifc7dbbe305d24242b0705de1dea34e8923e9d2cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8603
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
We didn't return anything useful other than ErrorKind::IO anyways.
We can use io::ErrorKind::Unsupported for DummyIO.
Fixes b/271.
Change-Id: Icb231e9b38168e8b6fa473bfa405d160357b317f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8602
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
One test in here assumed the server was fast, but when very busy, a
computer running these tests might not be fast.
Treat both cases that can occur separately.
Change-Id: Iba168ad39f2458c4fb8873211df33beeaff7c6c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8595
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Also, get rid of the explicit byte comparison here, but unwrap like in
the rest of the codebase. We'll deal with this in a generic manner in
b/267 and b/189.
Change-Id: Ie5f3d27ab35b7e88d67a2796c29cdd7bc7df71f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8589
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This remained around from some previous time where this type was being
cloned around directly, but it's not anymore.
Change-Id: I2c61cf9cecb8e5d7d08644ed20642ee61357bc21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8580
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a wrapper type TvixIO<T: EvalIO>, which can wrap around an
arbitrary EvalIO implementation and perform actions needed for the
Tvix CLI (marking imported paths as known, and handling __corepkgs__).
Change-Id: I5fc1ca199b9f94b21a89103b84575e0f8f58dff9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8579
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This walks from a node further down until it reaches the requested path.
Change-Id: I2f9a15a8601db4d06c95d7b47cd6153264e203e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8568
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This distinguishes it better from the EvalIO::import_path method.
Also update the docstring to explain what it does (and what it doesn't).
Change-Id: I32a8b2869fa67a894df28532b22bf170961a2abf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8578
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This was missed while renaming NixPath to StorePath.
Change-Id: Ibcc929c43b111e4370e8222c1dd86d403548367f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8577
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
We currently only support querying by the output hash digest.
This makes the interface a bit simpler.
Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This allows decomposing a path consisting of a store path AND a suffix
into a StorePath, and a PathBuf containing the rest.
Change-Id: I81290e2fd804cdc9d1e88c71cb22c0fb882d7936
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8567
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Make it cleaner that StorePath only does encode the first path component
after the STORE_DIR prefix. Also, move some of the comments around a
bit, so it makes more sense what's using what.
Change-Id: Ibb57373a13526e30c58ad561ca50e1336b091d94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8566
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
It's okay if these calls mutate some internal state inside an
implementation.
Change-Id: I12bb11bde0310778c3da1275696bf7de058863a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8571
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Make it more explicit that we expect the from_string calls to fail here.
Change-Id: Ib3d46fc0850e364125e3548670ef301eeea2e45c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8565
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This connects to a (remote) tvix-store BlobService over gRPC.
Change-Id: If31f706738a5c3445886c117feca8b61f3203e9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8552
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Whether chunking is involved or not, is an implementation detail of each
Blobstore. Consumers of a whole blob shouldn't need to worry about that.
It currently is not visible in the gRPC interface either. It
shouldn't bleed into everything.
Let the BlobService trait provide `open_read` and `open_write` methods,
which return handles providing io::Read or io::Write, and leave the
details up to the implementation.
This means, our custom BlobReader module can go away, and all the
chunking bits in there, too.
In the future, we might still want to add more chunking-aware syncing,
but as a syncing strategy some stores can expose, not as a fundamental
protocol component.
This currently needs "SyncReadIntoAsyncRead", taken and vendored in from
https://github.com/tokio-rs/tokio/pull/5669.
It provides a AsyncRead for a sync Read, which is necessary to connect
our (sync) BlobReader interface to a GRPC server implementation.
As an alternative, we could also make the BlobReader itself async, and
let consumers of the trait (EvalIO) deal with the async-ness, but this
is less of a change for now.
In terms of vendoring, I initially tried to move our tokio crate to
these commits, but ended up in version incompatibilities, so let's
vendor it in for now.
Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We already returned UnexpectedEof in case the reader stopped returning
bytes too early, but similarly we should also fail if there's still
bytes left to be read in the reader passed.
We normally use the NAR writer to produce new NAR files, so the readers
point to the blobs we actually want to render, and having some data left
in there should be an error.
If for some reason the reader points to more data than just the blob,
the `.take` method can be used to limit it to the (known) size.
Change-Id: I9e8fa0a6dd9c794492abb6dc9e55995e619cb3bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8553
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Before there was code scattered about (e.g. text hashing module and
derivation output computation) constructing store paths from low level
building blocks --- there was some duplication and it was easy to make
nonsense store paths.
Now, we have roughly the same "safe-ish" ways of constructing them as
C++ Nix, and only those are exposed:
- Make text hashed content-addressed store paths
- Make other content-addressed store paths
- Make input-addressed fixed output hashes
Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The logic validating connectivity of Directory nodes should be moved
to SimplePutter, and this use whatever DirectoryPutter the store comes
with.
Change-Id: Id68a86a96cc49ff73920017839788859ea9c5161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8358
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This should allow import_path to communicate to a gRPC remote store,
that actually verifies the Directory nodes are interconnected.
Change-Id: Ic5d28c33518f50dedec15f1732d81579a3afaff1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8357
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This provides a handle to upload multiple proto::Directory as part of
the same closure.
Change-Id: I9213dde257a260c8622239918ea541064b270484
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8356
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
When building store paths we can just construct the thing.
Change-Id: Ife5d461d6a440ecbb22f32a86a6d51d212a2035b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8409
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
They can go under `nixhash`
Change-Id: Ia15835c57130b66d58f5df80ae9595dceee00941
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8408
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
It is moved into `store_path::utils` with the other path builders.
Change-Id: I3257170e442af5d83bcf79e63fa7387dd914597c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8410
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This switches out the previous compressed representation (count of
instructions per span) with a representation where the chunk's span
list stores the index of the first operation that belongs to a span,
and finds the right span by using a binary search when looking them
up.
This improves the lookup complexity from O(n) to O(log n).
This improvement was suggested and (mostly) implemented by GPT-4. I
only fixed up some names and updated the logic for deleting
spans (which it only did not do because I didn't tell it about that).
The code was verified by producing a complex error before/after the
change and ensuring that all spans in the error match exactly.
Co-Authored-By: GPT-4
Change-Id: Ibfa12cc6973af1c9b0ae55bb464d1975209771f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8385
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This doesn't have anything to do with ATerms, we just happen to be using
the aterm representation of a Derivation as contents.
Moving this into store_path/utils.rs makes these things much cleaner -
Have a build_store_path_from_references function, and a
build_store_path_from_fingerprint helper function that makes use of it.
build_store_path_from_references is invoked from the derivation module
which can be used to calculate the derivation path.
In the derivation module, we also invoke
build_store_path_from_fingerprint during the output path calculation.
Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This only added a suffix to the input argument, if build_store_path was
building the path of a Derivation.
As we need to also add the `.drv` suffix to the name we pass into
text_hash_string inside calculate_derivation_path, we can simply add the
suffix there and drop the parameter from build_store_path.
Change-Id: Icd5343dd1458f112b9296b389e81ce2ebdd16a9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8365
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Use it to calculate the text_hash_string, which is then used in the
calculate_derivation_path and path_with_references functions.
Relates to b/263.
Change-Id: I7478825e2a23a11224212fea5e3fd06daa97d5e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8364
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
… and keep the pub exports as is.
Change-Id: I2ad21660577553395f05b5ba71083626429b0dfc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8363
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
… and keep the pub exports as is.
Change-Id: I9f89a738c508c478ddba61303c21ea294f01ee9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8362
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
We do compare for equality. This comment probably was when I tried to
compare the `Result<T, E>`, and as `E` doesn't derive PartialEq it was
annoying.
Change-Id: I18bb19528c76af91c9d24d88d55dd46d0c092d20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8354
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This moves the recursive BFS traversal of Directory closures from the
GRPCDirectoryServiceWrapper out into a a DirectoryTraverser struct
implementing Iterator.
It is then used from various implementors of DirectoryService in the
`get_recursive()` method.
This allows distinguishing between recursive requests and non-recursive
requests in the gRPC client trait implementation.
Change-Id: I50bfd4a0d9eb11832847329b78c587ec7c9dc7b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8351
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This grows the frame stack as the call stack grows, which yields *much*
better user-facing error messages.
I haven't measured the performance impact this has yet, for now I'm
still just trying to add more information to errors and then cut down
again where necessary.
Change-Id: I89f058ef31979edacf4667775d460b60704ce4d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8334
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This provides a GRPCDirectoryService struct implementing
DirectoryService, allowing a client to Directory objects from a (remote)
tvix-store.
Remote in this case is anything outside the current process, be it
another process, or an endpoint on the network.
To keep the sync interface in the `DirectoryService` trait, a handle to
some tokio runtime needs to be passed into the constructor, and the two
methods use `self.tokio_handle.spawn` to start an async function, and
`self.tokio_handle.block_on` to wait for its completion.
The client handle, called `grpc_client` itself is easy to clone, and
treats concurrent requests internally. This means, even though we keep
the `DirectoryService` trait sync, there's nothing preventing it from
being used concurrently, let's say from multiple threads.
There's still two limitations for now:
1) The trait doesn't make use of the `recursive` request, which
currently leads to a N+1 query problem. This can be fixed
by `GRPCDirectoryService` having a reference to another
`DirectoryService` acting as the local side.
I want to wait for general store composition code to pop up before
manually coding this here.
2) It's currently only possible to put() leaf directory nodes, as the
request normally requires uploading a whole closure. We might want
to add another batch function to upload a whole closure, and/or do
this batching in certain cases. This still needs some more thinking.
Change-Id: I7ffec791610b72c0960cf5307cefbb12ec946dc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8336
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
We query the blob service for detailled blob info, not the chunk
service.
Change-Id: I85a6a57b1dae74a950f734be7d4455c5c35ae355
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8348
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This toggles whether tvix will evaluate the top-level value and
deep-force it, or return it potentially still containing thunks.
Change-Id: Ie910941e3b6a0f16c5c0cb896d73947626335f4b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8326
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This makes it possible for callers to control whether they can receive
partially evaluated values from an evaluation or not.
We're actually flipping the default behaviour to non-strict top-level
evaluation, which means that callers have to set `strict = true` on
the Evaluation to get the previous behaviour.
Change-Id: Ic048e9ba09c88866d4c3177d5fa07db11c4eb20e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8325
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
After ingestion of the contents into the store, this will use the
NonCachingNARCalculationService to create a NAR stream or the contents
of the path, and use our Derivation output path calculation machinery to
determine the output path (using recursive hashing strategy).
In a real-world scenario, we obviously want to cache these calculations,
but this should be sufficient to tinker around with it.
Change-Id: I9b2e69384414f0be1bdcb5a99a4bfd46e8db9932
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8317
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Emits the span of the `set` that is being accessed in the `force`
operation of an attribute access.
Looking at traces, it's a lot more useful to get information about
*what* is being forced, as in cases like `foo.bar` it can be
misleading to have an error highlight `bar`, when the error occured
while forcing `foo` to be able to access `bar` in the first place.
Change-Id: Id46ff28f20c67cb4971727ac52cc4811795cea2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8272
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This reports the span
1. of the code within a thunk,
2. of the place where the thunk was instantiated,
3. of the place where the thunk was first forced,
4. of the place where the thunk was forced again,
when yielding an infinite recursion error, which hopefully makes it
easier to debug them.
The spans are tracked in the ThunkRepr::Blackhole variant when putting
a thunk under evaluation.
Note that we currently have some loss of span precision in the VM loop
when switching between frame types, so spans 3/4 are currently a bit
wonky. Working on it.
Change-Id: Icbd2a9df903d00e8c2545b3fc46dcd2a9e3e3e55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8270
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This is step 1 towards being able to use all 4 spans that we know when
dealing with infinite recursion. It tracks the span at which the
force of a thunk was first requested when constructing a blackhole, so
that we can highlight the spans of the first and second forces.
These are actually the least relevant spans, but the easiest to put in
place, more coming soon.
Change-Id: I4c7e82f6211b98756439d4148a4191457cc46807
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8269
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This produces traces in which we can see what kind of native code was
run. Note that these "names" are named after the generator message, so
these aren't *really* intended for end-user consumption, but we can
give them saner names later.
Example:
https://gist.github.com/tazjin/82b24e92ace8e821008954867ee05057
This already makes the traces a little easier to parse.
Change-Id: Idcd601baf84f492211b732ea0f04b377112e10d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8268
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
When emitting an error at runtime, the VM will now use the new
`NativeError` and `BytecodeError` error kinds (which just wrap inner
errors) to create a set of diagnostics to emit.
The primary diagnostic is emitted last, with `error` type (so it will
be coloured red in terminals), the other ones will be emitted with
`note` type, highlighting the causal chain.
Example:
https://gist.github.com/tazjin/25feba7d211702453c9ebd5f8fd378e4
This is currently quite verbose, and we can cut down on this further,
but the purpose of this commit is to surface more information first of
all before worrying about the exact display.
Change-Id: I058104a178c37031c0db6b4b3e4f4170cf76087d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8266
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This actually uses coercion under the hood in C++ Nix. See the test
for an example.
Change-Id: Id56b364acf269225b6829d0b600e0222f8b3608d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8322
Reviewed-by: andi <andi@notmuch.email>
Tested-by: BuildkiteCI
This was commented out and forgotten during the generator refactor, oh
well.
Change-Id: I474b685159a955a846db462da0dd0067af177b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8321
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Passing in a &proto::node::Node into all this allows us consumers to
keep ownership of the proto::node::Node.
Change-Id: I44882a86c46826b06a8a8a0b24c18adfc7052662
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8316
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
The behaviour of this function is a bit unintuitive, and cl/8310 already
inlined the other consumer of it.
Rewrite the last consumer of the function, so we can drop it.
Change-Id: I59c8486037ce3f777667d1d9e4f4a9316d5a0cb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8311
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Instead of having two very similar match branches for the FOD and non-
FOD case, detect the FOD case while looping over all outputs.
In the case of anything other than recursive sha256 FODs, the
fingerprint and output path calculation is exactly the same.
Change-Id: Ieb6995653d008766e595cf29d7cd4fb1334e33dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8310
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Walking the arguments might encounter an `outputs` output, which might
explicitly (for whatever reason) specify the `out` output.
To prevent dropping FOD settings in this case, we have to populate
that part of the configuration after walking the other attributes.
Change-Id: Iee6a7f0a71e9c9699e79d35e6cb19e1ddb49395d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8312
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This stops using our own custom Hash structure, which was mostly only
used because we had to parse the JSON representation somehow.
Since cl/8217, there's a `NixHash` struct, which is better suited to
hold this data. Converting the format requires a bit of serde labor
though, but that only really matters when interacting with JSON
representations (which we mostly don't).
Change-Id: Idc5ee511e36e6726c71f66face8300a441b0bf4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8304
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Apparently, having multiple packages with the same path is a bad thing:
```
The bin target `tvix-store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)` has the same output filename as the lib target `tvix_store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)`.
Colliding filename is: /home/flokli/tvl/tvix/target/doc/tvix_store/index.html
The output filenames should be unique.
This is a known bug where multiple crates with the same name use
the same path; see <https://github.com/rust-lang/cargo/issues/6313>.
```
Change-Id: Ic785c0349070783baf5e8fd23f5fb60603a3c995
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8308
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
There's still some shadowing going on, but that's left for a followup
CL.
Change-Id: I02992f1eb494faca99857a3a5ee4dcd47f1b9fd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8306
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This will be published on docs.tvix.dev
Change-Id: I348e057351d5295ad20953c9e9a32c257abab089
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8298
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The website itself contains no useful information that we don't
already have published anywhere, but it's a decent landing place.
Mostly I want to have something on the root of tvix.dev, so that we
can start using it for other purposes.
Change-Id: Id2b4dabc7f6e4dd26b61484b86dbde2f39aa1719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8296
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This only trims the output paths from a Derivation struct, not the
output hashes.
Change-Id: I9250fec4602ed05bb64540c4a89ddb6fb052be1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8303
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Call this function derivation_or_fod_hash, and return a NixHash.
This is more in line with how cppnix calls this, and allows using
to_nix_hash_string() in some places.
Change-Id: Iebf5355f08ed5c9a044844739350f829f874f0ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8293
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Change-Id: I8224bf039f739c401900b5a2ddc839810c87cf6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8226
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
We settled on this being the most reasonable name for this construct.
Change-Id: Ic31c45461a842f22aa05f4446123fe3a61dfdbc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8291
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Given Rust's current lack of support for tail calls, we cannot avoid
using `async` for builtins. This is the only way to avoid
overflowing the cpu stack when we have arbitrarily deep
builtin/interpreted/builtin/interpreted/... "sandwiches"
There are only five `async fn` functions which are not builtins
(some come in multiple "flavors"):
- add_values
- resolve_with
- force, final_deep_force
- nix_eq, nix_cmp_eq
- coerce_to_string
These can be written iteratively rather than recursively (and in
fact nix_eq used to be written that way!). I volunteer to rewrite
them. If written iteratively they would no longer need to be
`async`.
There are two motivations for limiting our reliance on `async` to
only the situation (builtins) where we have no other choice:
1. Performance.
We don't really have any good measurement of the performance hit
that the Box<dyn Future>s impose on us. Right now all of our
large (nixpkgs-eval) tests are swamped by the cost of other
things (e.g. fork()ing `nix-store`) so we can't really measure
it. Builtins tend to be expensive operations anyways
(regexp-matching, sorting, etc) that are likely to already cost
more than the `async` overhead.
2. Preserving the ability to switch to `musttail` calls.
Clang/LLVM recently got `musttail` (mandatory-elimination tail
calls). Rust has refused to add this mainly because WASM doesn't
support, but WASM `tail_call` has been implemented and was
recently moved to phase 4 (standardization). It is very likely
that Rust will get tail calls sometime in the next year; if it
does, we won't need async anymore. In the meantime, I'd like to
avoid adding any further reliance on `async` in places where it
wouldn't be straightforward to replace it with a tail call.
https://reviews.llvm.org/D99517https://github.com/WebAssembly/proposals/pull/157
https: //github.com/rust-lang/rfcs/issues/2691#issuecomment-1462152908
Change-Id: Id15945d5a92bf52c16d93456e3437f91d93bdc57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8290
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This commit moves fetch_forced_with and fetch_captured_with into the
scope of their only caller (resolve_with).
Change-Id: I9a8bc27228888729d591e8cb021c431b2b6468f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8289
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This rewrites nix_cmp_ordering as an iterative loop, which
eliminates the extra pinned-boxing helper function.
Change-Id: I33d0ecc913e02affd8fd4c7bc1c9ecfdf4c7deb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8288
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
* We no longer need backtrace-on-stack-overflow, as we no longer
overflow the stack with the recent eval refactorings. This was weird
voodoo anyways, introduced earlier to debug some cases where stack
overflows occured.
* default features of genawaiter crate are not needed, as we don't use
their proc macros
Change-Id: I346fc5a18d7f117ee805909a8be8f535b96be76c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8263
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This reorders the operations in the VM's main `match` statement while
evaluating bytecode according to the frequency with which these
operations appear in some nixpkgs evaluations.
I used raw data that looks like this:
https://gist.github.com/tazjin/63d0788a78eb8575b04defaad4ef610d
This has a small but noticeable impact on evaluation performance.
No operations have changed in any way, this is purely moving code
around.
Change-Id: Iaa4ef4f0577e98144e8905fec88149c41e8c315c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8262
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The name of this was not accurate anymore after all the recent
shuffling, as noted by amjoseph. Conceptual tail calls here only occur
for Nix bytecode calling Nix bytecode, but things like a builtin call
actually push a new native frame.
Change-Id: I1dea8c9663daf86482b8c7b5a23133254b5ca321
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8256
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
... except now the tests fail, but at least it works
Change-Id: I05e86c173f40533ae65548585c1ddaa200ac5235
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8214
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This file which ships with C++ Nix is required for evaluating nixpkgs.
Like C++ Nix, we now inject a pseudo path in EvalIO from which this
will resolve as <nix/fetchurl.nix>
Change-Id: Ic948c476a2cfc6381d5655d308bc2d5fa25b7123
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8213
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This drops the usage of serde::Serialize, as the trait can not be used
to implement the correct semantics (function colouring!).
Instead, a manual JSON serialisation function is written which
correctly handles toString, outPath and other similar weirdnesses.
Unexpectedly, the eval-okay-tojson test from the C++ Nix test suite
now passes, too.
This fixes an issue where serialising data structures containing
derivations to JSON would fail.
Change-Id: I5c39e3d8356ee93a07eda481410f88610f6dd9f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8209
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This adds static strings to generator frames that describe the
generator in a human-readable fashion, which are then logged in
observers.
This makes runtime traces very precise, explaining exactly what is
being requested from where.
Change-Id: I695659a6bd0b7b0bdee75bc8049651f62b150e0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8206
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
These are serialised as the serialisation of the value of that field.
Change-Id: Ida51708b1f43ce09b0ec835f4e265918aa31dd09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8205
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
These must be serialised to a JSON string of the *result* of coercing
the function application to a string.
Change-Id: Ib7f49ccd950503ddbdbf99643cd59565e26b50da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8204
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
It turns out that this is used not just in coerceToString, but also in
toJSON.
Change-Id: I1c324b115a0b8bb6d83446d5bf70453c9b90685e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8203
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Do not print the entire value (they're likely to be thunks anyways).
This is useful because there *can* be cases where something like
`nixpkgs` itself is sent through one of these messages, in which case
the observer trying to print it will just blow up.
Change-Id: I1fa37ea071d75efa0eb3428c6e2fe4351c62be6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8202
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Print only the top 6 values of the stack, not the entire stack.
There's very few operations that deal with more values anyways, so the
rest are not likely to be useful.
This gets us one step closer to tracing VERY large executions without
blowing up.
Change-Id: I97472321b0321b25d534d9f53b3aadfacc2318fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8201
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This can actually blow up when tracing arbitrary execution, as some of
the data structures just get too large to run through a tabwriter.
Change-Id: I6ec4c30ee48655b8a62954ca219107404fb2c256
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8200
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Formals can be initialised with deferred default values (see the test
cases), in which case they need an extra thunk to have something that
can be finalised appropriately when the setup is done.
Fixes: b/255
Change-Id: I380e3770be68eaa83ace96d450c7cead32dacc9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8196
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This shaves another 8 bytes off Value. How did that type get so big?!
Change-Id: I65e9b59a1636bd57e3cc4aec5fea16887070b832
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8153
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
No longer needed, and in some cases caused some extra work.
Change-Id: I64e8e7292573bdc92a9c7a8e470e33f8c526f311
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8152
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Instead of the two different representations (which we don't really
use much), use a `Box<str>` (which potentially shaves another 8 bytes
off `Value`).
NixString values themselves are immutable anyways (which was a
guarantee we already had with `SmolStr`), so this doesn't change
anything else.
Change-Id: I1d8454c056c21ecb0aebc473cfb3ae06cd70dbb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8151
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The size of a `Vector<Value>` is 64 *bytes*, which is quite large, and
it bloated the entire Value type to this size.
This change adds an indirection for the inner vector through Rc.
Initially I tried to use a Box, but this breaks pointer equality
guarantees for the Vector when it is small enough to be inlined.
This reduces the size of Value from 64 to 32 bytes.
Change-Id: Ic3211e861b1966c78b2c3d536ba291fea92647fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8150
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Warning: This is probably the biggest refactor in tvix-eval history,
so far.
This replaces all instances of trampolines and recursion during
evaluation of the VM loop with generators. A generator is an
asynchronous function that can be suspended to yield a message (in our
case, vm::generators::GeneratorRequest) and receive a
response (vm::generators::GeneratorResponsee).
The `genawaiter` crate provides an interpreter for generators that can
drive their execution and lets us move control flow between the VM and
suspended generators.
To do this, massive changes have occured basically everywhere in the
code. On a high-level:
1. The VM is now organised around a frame stack. A frame is either a
call frame (execution of Tvix bytecode) or a generator frame (a
running or suspended generator).
The VM has an outer loop that pops a frame off the frame stack, and
then enters an inner loop either driving the execution of the
bytecode or the execution of a generator.
Both types of frames have several branches that can result in the
frame re-enqueuing itself, and enqueuing some other work (in the
form of a different frame) on top of itself. The VM will eventually
resume the frame when everything "above" it has been suspended.
In this way, the VM's new frame stack takes over much of the work
that was previously achieved by recursion.
2. All methods previously taking a VM have been refactored into async
functions that instead emit/receive generator messages for
communication with the VM.
Notably, this includes *all* builtins.
This has had some other effects:
- Some test have been removed or commented out, either because they
tested code that was mostly already dead (nix_eq) or because they
now require generator scaffolding which we do not have in place for
tests (yet).
- Because generator functions are technically async (though no async
IO is involved), we lose the ability to use much of the Rust
standard library e.g. in builtins. This has led to many algorithms
being unrolled into iterative versions instead of iterator
combinations, and things like sorting had to be implemented from scratch.
- Many call sites that previously saw a `Result<..., ErrorKind>`
bubble up now only see the result value, as the error handling is
encapsulated within the generator loop.
This reduces number of places inside of builtin implementations
where error context can be attached to calls that can fail.
Currently what we gain in this tradeoff is significantly more
detailed span information (which we still need to bubble up, this
commit does not change the error display).
We'll need to do some analysis later of how useful the errors turn
out to be and potentially introduce some methods for attaching
context to a generator frame again.
This change is very difficult to do in stages, as it is very much an
"all or nothing" change that affects huge parts of the codebase. I've
tried to isolate changes that can be isolated into the parent CLs of
this one, but this change is still quite difficult to wrap one's mind
and I'm available to discuss it and explain things to any reviewer.
Fixes: b/238, b/237, b/251 and potentially others.
Change-Id: I39244163ff5bbecd169fe7b274df19262b515699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Adds a `Value::neo_nix_eq` method (the `neo_` prefix will be dropped
when we flip over to the generator implementation of the VM) which
implements Nix equality semantics using async, generator-based
comparisons.
Instead of tracking the "kind" of equality that is being compared (see
the pointer-equality doc) through a pair of booleans, I've introduced
an enum that explicitly lists the possible comparisons.
Change-Id: I3354cc1470eeccb3000a5ae24f2418db1a7a2edc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8241
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
These functions will be used by the changes in the VM to observe the
runtime execution of generator frames, and provide a more linear view
of the execution of the Tvix VM.
Change-Id: I10b1b1933dedc065e7c61d5d6062f0aaeee0097e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8240
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
In order to implement an asynchronous builtins.sort (required for
moving builtins to generators), we need an `async` sorting algorithm
as our comparators involve invoking a Nix function.
This commit implements a fairly simple, optimised bubble sort as the
sorting algorithm used in our `async fn sort_by`.
There don't seem to be any crates providing async versions of things
like this, and they might actually be pretty hard to implement
generically due to some constraints about how `async` works.
Note that this algorithm is less efficient than the hybrid
"timsort/mergesort/insert sort" used in the Rust standard library. I
tried to write a merge sort implementation, but ran into isuses with
the sort becoming unstable because our comparators can not yield
equality. This is the simplest implementation which I know to be
correct.
Note that as of this commit this is *not* covered by the Tvix test
suite, but it will be as soon as the rest of the generator code lands.
Change-Id: Ia9a604f7dd941d6acc9212c902e0e637ed75bebc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8239
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
All code initially using this has been replaced by the simpler and more
performant implementation with StreamCDC and read_all_and_chunk.
Change-Id: I08889e9a6984de91c5debcf2b612cb68ae5072d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8265
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This was the last piece of code using BlobWriter.
We can also use `read_all_and_chunk`, it's just requires a bit more
plumbing:
- The data coming from the client (stream) needs to be mapped (we
extract the .data field).
- The stream needs to be turned into an (async) reader
- The reader needs to be made sync, and that code using the sync reader
needs to be in a `task::spawn_blocking`.
Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This moves the logic from src/import.rs that
- reads over the contents of a file
- chunks them up and uploads individual chunks
- keeps track of the uploaded chunks in a BlobMeta structure
- returns the hash of the blob and the BlobMeta structure
… into a generic read_all_and_chunk function in
src/chunkservice/util.rs.
It will work on anything implementing io::Read, not just files, which
will help us in a bit.
Change-Id: I53bf628114b73ee2e515bdae29974571ea2b6f6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8259
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Make use of the helper function here as well.
Change-Id: Ia0afd84eb3903bb897ee6aee884dc291f3e4371c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8258
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
When building tvix-store without default features, this variable doesn't
need to be mutable. Silence the warning.
Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
We currently send two warnings in case of detecting dead code - W008
inside compile_dead_code, and a more detailed warning in all places that
invoke compile_dead_code:
```
warning[W007]: useless operation on boolean: this expression is always false
--> /nix/store/qz3gjn95gazab4fkb7s8lm6hz17rdzzy-414z9nnj1wy66ymq6vgb693x9xjz6hf2-nixpkgs-src/pkgs/top-level/perl-packages.nix:12079:15
|
12079 | doCheck = false && !stdenv.isDarwin;
| ^^^^^^^^^^^^^^^^^^^^^^^^^
warning[W008]: this code will never be executed
--> /nix/store/qz3gjn95gazab4fkb7s8lm6hz17rdzzy-414z9nnj1wy66ymq6vgb693x9xjz6hf2-nixpkgs-src/pkgs/top-level/perl-packages.nix:12079:24
|
12079 | doCheck = false && !stdenv.isDarwin;
| ^^^^^^^^^^^^^^^^
```
The place invoking `compile_dead_code` has more context to why the code
is unused, so it's error message is much more useful.
Stop emitting the less informative warning inside compile_dead_code
(W008), and update the comment that we expect the caller to emit a
warning.
I kept W008 itself still around, in case we end up having places this
will get used again.
Change-Id: I2c5d84fc0cb4035872cd4b71cc3e9e34e120eb37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8024
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This removes the use of Box::new, switching fastcdc to version 3.0.2
with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed.
Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
We're using this in a bunch of places. Let's move it into a helper
function.
Change-Id: I118fba35f6d343704520ba37280e4ca52a61da44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8251
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This is useful not only in blobwriter contexts.
Change-Id: I4c584b5264ff7b4bb3b1a9671affc39e18bf4ccf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8245
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This already is a &CS, so we don't need to clone().
Change-Id: I5397a5948ae7fe4781f18df760a79047f83dca01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8243
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Look at the data that's written to us, and upload all chunks but the
rest in parallel, using rayon. This required moving `upload_chunk`
outside the struct, and accepting a ChunkService to use for upload
(which it was previously getting from `self.chunk_service`).
This doesn't speed up things too much for now, because things are still
mostly linear.
Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
The Directory service does already reject inserting invalid (wrongly
sorted) Directory messages, but our test case didn't provoke it.
Change-Id: I228e201925e8999186659a2d8da0118db184d9ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8167
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This exposes the previous default behavior at the `tvix-store daemon`
subcommand.
It also adds a `tvix-store import` command, which will ingest a given
path into the store.
Change-Id: Ide14f1d409b9364e7f98090690c744326486e470
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8166
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This provides a service using /dev/shm, that's deleted once the
reference is dropped.
Refactor all tests to use these, which allows getting rid of most
TempDir usage in the tests.
The only place where we still use TempDir is in the importer tests,
which work on a filesystem path.
Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
While we don't want to keep all of the data in memory, we want to
feed a reasonably-enough buffer to the chunking function, to prevent
unnecessarily trying to chunk over and over again.
Change-Id: I5bbe2d55e8c1c63f8f7ce343889d374b528b559e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8160
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This imports the contents at a given Path into the tvix store.
It doesn't register the contents at a Path in the store itself, that's up
to the PathInfoService.
Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This also uploaded HELLOWORLD_BLOB_CONTENTS before, but it's not
referred from anywhere in the fixture.
Change-Id: I823133afe0f08d18a59e2ac4e4d4bb7d34ce8a2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8158
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This allows hiding to tests what exact implementation we're using, when
testing things that do something with a store, but don't care what's
used for underlying storage.
Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This will moves the chunking-as-we-receive logic that so far only lived
in grpc_blobservice_wrapper.rs into a generic BlobWriter.
Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Use the FastCDC::cut function to ask fastcd for cutting points as we
receive the data. Make sure to keep the last chunk in the temporary
buffer, as we might not actually cut at the end.
Also, use rayon to calculate the blake3 hash if the input data is
> 128KiB.
Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This switches away from the less canonical "ronomon" version to the
implementation as described in the
[paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et
al., in 2020.
That version uses 64-bit hash values and tends to be faster than both
the ronomon and v2016 versions, and produces the same chunking as the
2016 version.
As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's
the recommended choice.
The crate also gained support for streaming version of chunkers:
https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be
useful.
Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
These were implementing against the (more complicated) gRPC interface,
for which we now have a wrapper.
Change-Id: I0a8284493718ab99618a1d21a76df4d173edb899
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8100
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This swaps out the implementation used in the main entrypoint.
We now use the non-gRPC aware Sled*Service, and wrap it with
GRPC*ServiceWrapper to spin up the gRPC server.
Change-Id: I5a1236c0612eee85e49891062040127c8bd95058
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8106
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This exposes a proto::pathinfo_service_server::PathInfoService for a
directoryservice::PathInfoService and a way to calculate NARs.
Change-Id: I30cd058562f83d063c78b84976ec97190de49400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8097
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This adds a PathInfoService trait, and an implementation for it using
sled, and one using a HashMap.
Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This adds a NARCalculationService trait, which will take a root node,
and provide a proto::CalculateNarResponse in return.
It also adds a NonCachingNARCalculationService implementation, that will
simply always render the NAR in memory to calculate the size and sha256,
without any caching.
Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Make sure the blob size in the current proto node matches what we get
back from the blob backend.
Change-Id: I939fa18f37c7bc86ada8a495c7be622e69ec47f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8129
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
It's much easier to just unwrap the TempDir::new() result.
Change-Id: I7b05bc18f3146401e30e1cc2bb412503c5171a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8128
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This moves the logic rendering NARs to a struct using the
previously introduced, more granular BlobService, ChunkService and
DirectoryService.
Instead of passing them around to the helper functions, they're kept as
members of a struct.
Remove the async invocations in the nar_renderer tests, there's nothing
async in here.
Change-Id: Ic6d24aaad68a1fda46ce29f2cdb5f7b87f481d5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8095
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This exposes a proto::directory_service_server::DirectoryService for a
directoryservice::DirectoryService.
Change-Id: I191a6e44e5bce10d904b5f10f0073d6adaf1f572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8094
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This adds a DirectoryService trait, and an implementation for it using
sled, and one using a HashMap.
Change-Id: Ida61524b2ca949e1b3a78089a5aa7d9f9800c8d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8093
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This takes a BlobService and ChunkService in the constructor, and
provides a [proto::blob_service_server::BlobService] trait for it.
Implementing proto::blob_service_server::BlobService is a lot of surface
to cover, and providing this wrapper will make individual
implementations taking care of how to store chunks or chunking
information much simpler.
Change-Id: Ia7b46484fb3ac9104354d496ff2922dca96ff7b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8092
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This allows adding more stuff into this namespace, from different files.
Also move tests on proto-related code from src/tests to src/proto/tests.
Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
A BlobReader can be used to read a blob, which might consist out of
multiple chunks. Chunks are fetched from a ChunkService.
Change-Id: I1806225b0052adaa4a2320b79b744e554e524ee3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8088
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
This module contains the request/response types for generators
requesting actions from the VM.
For most of these, an async helper function is added that will be used
inside of generator functions to make use of these requests/responses
instead of constructing them directly.
Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
As applies are thunked, there was no situation where OpCall could be
emitted. In practice, all calls were already tail calls.
Change-Id: Id0d441dcdd86f804d7cddd0cc14f589bbfc75e5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8147
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This adds a BlobService trait, and an implementation for it using sled,
and one using a HashMap.
Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds the simpler ChunkService trait, and an implementation for it
using sled, and one using a HashMap.
Change-Id: Icb0fdc41b37b44e9e9e4f548d0f4acae1d83b71e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8086
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This came up recently again, it makes sense to document the reasoning
behind the decision.
Change-Id: Ic51d5bc7998c70e8b070b6f42877d8e88613935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8223
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Because they do not use it, and it can not be passed with the coming
generator refactoring.
Change-Id: I0d96f2357a7ee79cd8a0f401583d4286230d4a6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8146
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Instead of using a suspended native thunk, calculate and optionally
insert the storeDir builtin when the VM is constructed.
We already have the IO handle available at this point and can just
check whether a storeDir is present, and insert its absolute value as
a builtin.
Change-Id: If966eee6ff26dc888b6e888e7c46170c0c346b05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8145
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This is a ThunkSet wrapped to be shareable, which will be required
once ThunkSets are embedded in futures.
Change-Id: I5a067b7972ac86e4d354c75ef05c86b2284c1137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8144
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Again simplifying some code down the line, where bits of code that
construct attribute sets already have the final structure available.
Change-Id: I0bb7a1daa63298122b51be73d35d695a4f73f8b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8140
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This simplifies some code down the line.
Change-Id: I58dd71e796e11479f44516cf24932f8061843d23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8139
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Some of these strings are actually just the nix hash representation of
the hash calculated earlier.
There's another one passed around via calculate_drv_replacement_str, but
that's left for a followup.
Change-Id: Id99a2a926a980d679eb49c34ee6a36bf224699b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8218
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This provides a way to construct a NixHash struct without parsing
strings.
Change-Id: I947d96e15e51e72d5b02929cda8c5fc31d81253a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8217
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
We ironically didn't add support parsing for the "native" format that
Nix uses under the hood.
This extends the from_str method to peek at the prefix of the string to
determine whether to try decoding as SRI, Nix string, or whether it
should be a bare digest.
Change-Id: I33efd24968b16f86eff18305b4ca8f112c7131d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This will be used for both Nix hash strings and hash strings without the
algo specified.
Change-Id: Iedfe5494fba5f2be00614ba0fc38bf659eafd447
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8215
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds addresses of thunk and closure chunks to the debug output
displayed when dumping bytecode.
This makes it possible to see in the dump which thunks are referenced
by constants in other thunks.
Change-Id: I2c98de5227e7cb415666cd3134c947a56979dc80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8137
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This CL removes calling into_iter on a reference, as it will
not move out it's content into resulting iterator.
Change-Id: Ifcc10b7cf33b98453570cbcec3eb82ffaba2ffcb
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8126
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This CL removes redundant clone from value which is
going to be dropped without further use.
Change-Id: Ibd2a724853c5cfbf8ca40bf0b3adf0fab89b9be5
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8125
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Instead of using an explicit closure to clone elements, use .cloned().
Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The size field already is u32, we don't need to convert here.
Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We want the address that the Rc is pointing to, not the address of the
Rc.
Change-Id: I8eba21677f242bbe4166c74d4aa4269c316076e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8045
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This CL address clippy warning which expects to
use `writeln` instead of `write` for strings with
new line.
Change-Id: Ia72a07502c60cfd489ecf1e3833b9d42d44a8b17
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8030
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This CL address clippy warning about finding the zero
length of something using `is_empty()` instead of `len() == 0`.
Change-Id: I2b36c7c7b65b733609fc0dcd33be06f9d772bc9b
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8029
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This branch was missing, and an assumption elsewhere just executed the
returned (broken) bytecode.
This fixes b/253.
Change-Id: I015023ba921bc08ea03882167f1f560feca25e50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8090
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
These should be inspectable by callers.
Change-Id: Ia9ef871aa63958d06066aaea61b2aecbd217369b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8089
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Nix accepts SRI hashes that are missing their padding characters
in base64, as seen in
7e49471316/pkgs/development/libraries/kerberos/krb5.nix .
It only seems to work in the SRI case, not with `sha256` being set to a
(nopad) base64 string.
Add regression tests for this, and document why we don't want to support
*additional* characters afterwards.
Reported in https://b.tvl.fyi/issues/252
Change-Id: I9ffc2b417501b426ced1894a9cbf95ff5f0e5159
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8037
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
Further emphasize Read() can be used to ask for blobs OR chunks, and
that clients usually want to stat and then request (smaller) chunks,
rather than reading whole blobs.
Also clarify that the chunking used to send BlobChunks over has nothing
to do with the chunk sizes communicated in a Stat() request.
Change-Id: Ia615d190aae570611de2655b11342a14d0b75976
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Plain paths like `foo/bar.nix` are also allowed, so we can not
determine this based on the prefix.
The upstream PR that is referenced in a comment here has a
significantly different interface than we expected, so I'm not
touching that comment yet in this CL before I've had more time to
digest it.
Change-Id: Iea33bbb35de9c00a7d7fedf64d02253c75c1cc9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8032
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1)
but it's not clear whether upstream will accept patches, so for now
lets point this directly at our fork.
Change-Id: Iccdcedae3e9a8b783241431787c952561d032694
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
We only use Rc in `impl EvalIO for StdIO`, which is only included when
building with the "impure" feature.
Change-Id: Id29d647c899cbfcdda11abfb9fabd5aa7e24299f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8025
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This reduces the size of `Builtin` from 88 (!) bytes to 8, and as the
largest variant of `Value`, the size of that type from 96 to 64.
The next largest type is NixList, clocking in at 64 bytes.
This has noticeable performance impact. In an implementation without
disk I/O, evaluating nixpkgs.stdenv looks like this:
Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath'
Time (mean ± σ): 1.151 s ± 0.003 s [User: 1.041 s, System: 0.109 s]
Range (min … max): 1.147 s … 1.155 s 10 runs
After this change, it looks like this:
Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath'
Time (mean ± σ): 1.046 s ± 0.004 s [User: 0.954 s, System: 0.092 s]
Range (min … max): 1.041 s … 1.053 s 10 runs
Change-Id: I5ab7cc02a9a450c0227daf1f1f72966358311ebb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8027
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
When resolving a select expression (`attrs.name` or `attrs.name or
default`), if the set compiles to a constant attribute set (as is most
notably the case with `builtins`) we can backtrack and replace that
attribute set directly with the compiled value.
For something like `builtins.length`, this will directly emit an
`OpConstant` that leaves the `length` builtin on the stack.
Change-Id: I639654e065a06e8cfcbcacb528c6da7ec9e513ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This was reflowed in a funny way. Move the whole command into its own
line, to prevent it from happening.
Change-Id: Ifba4daf418487ca4c32586820071930d29020f42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8026
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This fixes a very complicated bug (b/246). Evaluation
progresses *much* further after this, leading to several less
complicated bugs likely being uncovered by this
What was the problem?
=====================
Previously, when evaluating a thunk, we had a code path that looked
like this:
match *thunk {
ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => {
let inner_repr = inner_thunk.0.borrow().clone();
drop(thunk);
self.0.replace(inner_repr);
}
/* ... */
}
This code path created a copy of the inner `ThunkRepr` of a nested
thunk, and moved that copy into the `ThunkRepr` of the parent.
The effect of this was that the original `ThunkRepr` (unforced!) lived
on in the original thunk, without the memoization of the subsequent
forcing applying to it.
This had the result that Tvix would repeatedly evaluate these thunks
without ever memoizing them, if they occured repeatedly as shared
inner thunks. Most notably, this would *always* occur when
builtins.import was used.
What's the solution?
====================
I have completely rewritten `Thunk::force_trampoline_self` to make all
flows that can occur in it explicit. I have also removed the outer
loop inside of that function, and resorted to more use of trampolining
instead.
The function is now well-commented and it should be possible to read
it from top-to-bottom and get a general sense of what is going on,
though the trampolining itself (which is implemented in the VM) needs
to be at least partially understood for this.
What's the new problem(s)?
==========================
One new (known) problem is that we have to construct `Error` instances
in all error types here, but we do not have spans available in some
thunk-related situations. Due to b/238 we cannot ask the VM for an
arbitrary span from the callsite leading to the force. This means that
there are now code paths where, under certain conditions, causing an
evaluation error during thunk forcing will panic.
To fix this we will need to investigate and fix b/238, and/or add a
span tracking mechanism to thunks themselves.
What other impacts does this have?
==================================
With this commit, eval of nixpkgs mostly succeeds (things like stdenv
evaluate to the same hashes for us and C++ Nix, meaning we now
construct identical derivations without eval breaking).
Due to this we progress much further into nixpkgs, which lets us
uncover more additional bugs. For example, after this commit we can
quickly see that cl/7949 introduces some kind of behavioural issue and
should not be merged as-is (this was not apparent before).
Additionally, tvix-eval is now seemingly very fast. When doing
performance analysis of a nixpkgs eval, we now mostly see the code
path for shelling out to C++ Nix to add things to the store in there.
We still need those code paths, so we can not (yet) do a performance
analysis beyond that.
Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We need to distinguish explicitly between the paths used for the
scanner, and the paths that populate the derivation inputs. The full
paths must be accessible from the result of the refscanner to populate
drv fields correctly.
This was previously hidden by debug changes that masked actual IO
operations with no-ops.
Change-Id: I037af6e6bbe2b573034d695f8779bee1b56bc125
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8022
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Creates a cache of imported literal files (e.g.
`./default-builder.sh`) which avoids shelling out to Nix for each
instance of the same file.
Note that a better way to tackle this is to create memoizable thunks
for these expressions in the compiler, but we are lacking a little bit
of infrastructure for that at the moment.
Change-Id: Ibc062b20d81e97dd3986e734d225a744e1779fe7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8015
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Switch out the string-scanning algorithm used in the reference scanner.
The construction of aho-corasick automata made up the vast majority of
runtime when evaluating nixpkgs previously. While the actual scanning
with a constructed automaton is relatively fast, we almost never scan
for the same set of strings twice and the cost is not worth it.
An algorithm that better matches our needs is the Wu-Manber multiple
string match algorithm, which works efficiently on *long* and *random*
strings of the *same length*, which describes store paths (up to their
hash component).
This switches the refscanner crate to a Rust implementation[0][1] of
this algorithm.
This has several implications:
1. This crate does not provide a way to scan streams. I'm not sure if
this is an inherent problem with the algorithm (probably not, but
it would need buffering). Either way, related functions and
tests (which were actually unused) have been removed.
2. All strings need to be of the same length. For this reason, we
truncate the known paths after their hash part (they are still
unique, of course).
3. Passing an empty set of matches, or a match that is shorter than
the length of a store path, causes the crate to panic. We safeguard
against this by completely skipping the refscanning if there are no
known paths (i.e. when evaluating the first derivation of an eval),
and by bailing out of scanning a string that is shorter than a
store path.
On the upside, this reduces overall runtime to less 1/5 of what it was
before when evaluating `pkgs.stdenv.drvPath`.
[0]: Frankly, it's a random, research-grade MIT-licensed
crate that I found on Github:
https://github.com/jneem/wu-manber
[1]: We probably want to rewrite or at least fork the above crate, and
add things like a three-byte wide scanner. Evaluating large
portions of nixpkgs can easily lead to more than 65k derivations
being scanned for.
Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The README was very sparse before and we've actually had people email
us (as it says to contact us) just to ask what Tvix *is*. This should
answer some questions!
Change-Id: I0f248cb060eccfe086468afed1d648652b35dfd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8018
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
... not just a TODO.
Most use-cases of unsafeDiscardStringContext are for cases where a
string is processed in some ways and no longer contains a "physical"
reference, but still has its context attached in C++ Nix.
We don't need to do this. This does diverge in behaviour in use-cases
related to build scheduling, but that whole behaviour will be
different in Tvix.
Change-Id: I4056d4c09f62d44d6bd52b791db03fe5556672b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8016
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
... instead of a BTreeMap, as we do not need ordering guarantees here.
HashMaps are noticeably faster here (especially as we've been sorting
essentially random data!).
Change-Id: Ie92d74286df9f763c04c9b226ef1066ee8484c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8014
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
If the tvix view is cloned through josh, you don't use mg, but a
`shell.nix` is provided.
Also, add the `git clone` command, so people browsing tvix source code
in the browser know where to clone from.
Change-Id: I18483d6a52953f9f4eafd1533ea69afb0e329b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Instead of implementing `std::fmt::Display for Derivation` and relying
on the `to_string` method, introduce a `to_aterm_string()` method, which
does the same thing, but makes it clearer what we're producing, rather
than just calling `to_string()``.
Change-Id: I21823de9096a0f2c2eb6f4591e48c1aa9fd94161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7998
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This module takes care of parsing various hashes and algorithms.
It will get used to modify derivation output hashes in the next CL.
Change-Id: Idc07c401dbb7510f49883ac02b8379b9a5d930c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7990
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The module `src/derivation/derivation.rs` is a sign of module inception.
Move the Derivation struct definiton up into `src/derivation/mod.rs`,
and some of the helpers in a `util.rs`.
Change-Id: Ib24a5f8a27bdd45df8b1fa2b3482a79b33cab8d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7997
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
clippy says:
> This expression creates a reference which is immediately dereferenced
> by the compiler
Change-Id: Ic2c093b043ebee9ae80912075083107e4d216cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7995
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This doesn't require any other corresponding handling *yet*, as the
actual replacements happen in the builder logic (which we delegate to
cppnix at the moment).
Change-Id: I034147c933f05ae427c7a8794647132d108d0ede
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7972
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is an example, which uses the debug trait to print all field
values.
Silence the compiler warning about unused fields.
Change-Id: I5f1216c77819003302e83ba1af1ff13c924f3b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7971
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This adds a fake argument name to builtins.toXML which allows toXML to
serialise any value instead of panicking on functions. We do still
have to fix the value itself, eventually, though.
Change-Id: I2e330ecddcd80442b4fac5eced64431ac86123ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a function that consumes a [proto::node::Node] pointing to the
root of a (store) path, and writes the contents in NAR serialization to
the passed [std::io::Write].
We need this in various places:
- tvix-store's calculate_nar() RPC method needs to render a NAR stream
to get the nar hash, which is necessary to give things imported in
the store a "NAR-based" store path.
- communication with (remote) Nix (via daemon protocol) needs a NAR
representation.
- Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a
NAR representation.
Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Put this in its src/derivation.
Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This provides a `Directory.nodes()` function, returning an iterator over
all three node types in an ordered fashion.
Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Make the trait public, so consumers can use it.
Also, implement it for node::Node, so we can later use this to access
the name from all three if we don't care about the enum type.
Change-Id: Iae530a16b705493645e61947852c03273876cc55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Move nixbase32 and store_path into this.
This allows //tvix/cli to not pull in //tvix/store for now.
Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We only do logic here if algo and hash_mode are Some(_)
(and there's an `out` output).
The fact we don't do anything in all in other cases is a bit hidden at
the bottom. Use if let for the destructuring, and drop the other case.
Change-Id: Icc0e38e62947d52d48ef610f754749737977fca9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7966
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This shows how people can use tvix_serde to deserialise configuration
structs for their programs from Nix code.
Change-Id: I71bf4e03dce19dddafe67dd729b4e4b10719a739
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7945
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Formals can depend on each other when using another formal as a
default value.
This test ensures that the compiler's declaration and initialisation
order of formals is consistent with what actually happens in the VM.
Change-Id: Ibdabe262554e8066d67fac1ebc3b5a48ef626e18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7948
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This helper function only was created because
populate_output_configuration was hard to test before cl/7939.
With that out of the way, we can pull it in.
Change-Id: I64b36c0eb34343290a8d84a03b0d29392a821fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Replace our data_encoding usage with the implementation taken from
https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32
Also uncomment some of the unit tests, and add a regression test for a
NIXBASE32.encode with a 32 bytes sequence.
The previous implementation of NIXBASE32.encode is wrong in that case:
```
❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA=
0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk
❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C
00000000 b3 a2 4d e9 7a 8f db c8 35 b9 83 31 69 50 10 30 |..M.z...5..1iP.0|
00000010 b8 97 70 31 bc b5 4b 3b 3a c1 37 40 f8 46 ab 30 |..p1..K;:.7@.F.0|
00000020
```
Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
There's gonna be more tests coming, let's move them into a separate file
everything Directory-related into a separate module.
Change-Id: I78a0f263925528907a22724b028f75cce644b329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Also add a `NamedNode` trait. We'll later use this to access names from
all three individually.
Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is repetitive and error prone (e.g. switching around
to_string/as_str has drastic consequences) due to the ToString
overloads.
Change-Id: I9b16a2e0e05e4c21e83f43e9f603746eb42e53f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7947
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This is much less code, and makes it much easier to read.
Change-Id: I9028f226105f905c2cc2cabd33907ff493e26225
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7938
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Instead of being called with `md5`, `sha1`, `sha256` or `sha512`,
`fetchurl.nix` (from corepkgs / `<nix`) can also be called with a `hash`
attribute, being an SRI hash.
In that case, `builtin.derivation` is called with `outputHashAlgo` being
an empty string, and `outputHash` being an SRI hash string.
In other cases, an SRI hash is passed as outputHash, but outputHashAlgo
is set too.
Nix does modify these values in (single, fixed) output specification it
serializes to ATerm, but keeps it unharmed in `env`.
Move this into a construct_output_hash helper function, that can be
tested better in isolation.
Change-Id: Id9d716a119664c44ea7747540399966752e20187
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7933
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Allows easier to use these errors in test cases.
Change-Id: I8abee3b522909ed4aa4b066499cc48eececbc036
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7937
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds an implementation of this builtin which correctly calculates
paths, but does not actually write anything to the store or verify
references.
Change-Id: Ie9764cbc1d13a73d8dc9350910304e2b7cad3fe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7910
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This uses the actual upstream Nix code for
`builtins.derivation` (which is not a primop in C++ Nix) to implement
`builtins.derivation` as a wrapper around `builtins.derivationStrict`.
We're doing it this way to ensure that our thunking logic is correct.
An initial Rust-native rewrite (see e.g. cl/7363) is pretty difficult
to debug while there are still other issues to root out, but
eventually we might want to turn this into native code.
Change-Id: I5845e18073e103b8670e40648bd7fd9b511058e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7902
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Implements the logic for converting an evaluator value supplied as
arguments to builtins.derivationStrict into an actual,
fully-functional derivation struct.
This skips the implementation of structuredAttrs, which are left for a
subsequent commit.
Note: We will need to port some eval tests over to CLI to test this
correct, which will be done in a separate commit later on.
Change-Id: I0db69dcf12716180de0eb0b126e3da4683712966
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7756
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Adds a helper function which handles special parameters to
`builtins.derivation` that are not just blindly passed through to the
builder environment, but populate other specific fields of the
derivation (outside of the ones handled by other, more complex helpers
from previous commits).
Change-Id: I82d1edf9af714fc4591e9071c0b83ece83be7eee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7901
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This threads through the fields that control whether a derivation is a
fixed-output derivation or not.
Change-Id: I49739de178fed9f258291174ca1a2c15a7cf5c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7900
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a helper function which takes the output of the reference
scanner used on derivation inputs and populates the `input_sources`
and `input_derivations` field of the derivation accordingly.
Note that we have a divergence from C++ Nix here, as we do not
populate the entire FS closure of a literally referred derivation (and
our standing theory is that this is unnecessary for nixpkgs).
Change-Id: Id0f605dd8c0a82973c56605c2b8f478fc17777d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7899
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Adds a small helper function which uses a Nix value supplied to
`builtins.derivation{Strict}` to populate the `outputs` field of the
`Derivation` struct.
Change-Id: Iccc7a4f293b3d913140aed576a573a8992241e46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7898
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We only stripped one of the two uses of this string, leading to
extraneous newlines in the refscanner.
Change-Id: I25d9119be082c487352f0cf66b97ecdcc3e1de06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7932
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Main one is the its-it's mistake in the last paragraph, the rest was
suggested by LanguageTool.
Change-Id: If1b87a11f480452f312fc2759be7ded782d0a522
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7930
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
While it is in the given example, i.e. for integer addition, to claim
that they are equivalent is a bit misleading: builtins.add is less
overloaded than +, i.e. builtins.add "foo" "bar" will fail whereas
"foo" + "bar" performs string concatenation.
Change-Id: Ib52d530d1ab289b367565b286f06a76dd518d4fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7929
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This keeps the actual TotalDisplay implementation readable, as this
float formatting code suddenly made up the majority of its implementation.
Change-Id: I2c0d00e4a691e0b8ffbc72680f680e16feef4bee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7925
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML
testcase from nix_tests. It uses the `toml` crate, and the serde
integration it brings with it.
Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Apparently our naive implementation of float formatting, which simply
used {:.5}, and trimmed trailing "0" strings not sufficient.
It wrongly trimmed numbers with zeroes but no decimal point, like
`10000` got trimmed to `1`.
Nix uses `std::to_string` on the double, which according to
https://en.cppreference.com/w/cpp/string/basic_string/to_string
is equivalent to `std::sprintf(buf, "%f", value)`.
https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated
like this:
> Precision specifies the exact number of digits to appear after
> the decimal point character. The default precision is 6. In the
> alternative implementation decimal point character is written even if
> no digits follow it. For infinity and not-a-number conversion style
> see notes.
This doesn't seem to be the case though, and Nix uses scientific
notation in some cases.
There's a whole bunch of strategies to determine which is a more compact
notation, and which notation should be used for a given number.
https://github.com/rust-lang/rust/issues/24556 provides some pointers
into various rabbit holes for those interested.
This gist seems to be that currently a different formatting is not
exposed in rust directly, at least not for public consumption.
There is the
[lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate
though, which provides a way to format floats with various strategies
and formats.
Change our implementation of `TotalDisplay` for the `Value::Float` case
to use that. We still need to do some post-processing, because Nix
always adds the sign in scientific notation (and there's no way to
configure lexical-core to do that), and lexical-core in some cases keeps
the trailing zeros.
Even with all that in place, there as a difference in `eval-okay-
fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated
the fixture to a less problematic number.
With this, the testsuite passes again, and does for the upcoming CL
introducing builtins.fromTOML, and enabling the nix testsuite bits for
it, too.
Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Two main reasons:
1. Traversing the structure to do this optimisation is
actually *slower* than not optimising it.
2. There are literally hundreds of thousands of incidences of this in
nixpkgs, and with some of the weird code there some of
these (functionally) useless parens are actually required for
readability reasons.
Change-Id: I1044b1c5f9fe20df4b6085851fc3b191277c65dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7917
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
call_value in the VM expects the callable to be forced when calling
it, which was not the case for functors.
Change-Id: Id55a2fe32a9573be42aef8669e268df519a989cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7909
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This makes it easier to pass owned strings if that's what we have.
Change-Id: Ia7351ff2681292d16534ec50fe60b926b683bb9a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7908
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is used in few enough places to just inline it. It felt a bit alien
in the ATerm construction aswell.
write.rs now pleasantly only includes tokens that occur in the ATerm
representation.
Change-Id: I524f8d6c1ce9057ff7fd16c6c3efd98467040a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7911
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
All of these occurences are actually sha256 digests of some fingerprint
- which means there's not a lot of need to do the hashing outside.
Instead of passing in a digest, keep the sha256 hasher in the function,
and pass in a fingerprint.
This makes it much easier to construct fingerprints using format!() in
all consumers, because we need don't need to juggle with the hasher
anymore.
Change-Id: I2dc3af2cab6cf06f55ae6cbd9a8be95faf2a07b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7907
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This allows the calculation of a store path for a plain string that
potentially contains references. These paths are used for
`builtins.toFile` (and potentially other features of C++ Nix).
Change-Id: Ic507c7f264f362b5e6e628255869e5a4fbe4d788
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7906
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Fixes some module comments and embeds the `compress_hash` function in
the `derivation` module, as it was not used outside of this module
anyways.
Change-Id: I6c5c92b3f0c03c2cdcbcfc2f813909a968c4d44c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7905
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This makes it possible to inject builtins into the builtin set that
are written in Nix code, and which at runtime are represented by a
thunk that will compile them the first time they are used.
Change-Id: Ia632367328f66fb2f26cb64ae464f8f3dc9c6d30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7891
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is used for content-defined chunking.
Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
I omitted the link to the commit history. Most people see this README
through some git interface anyways, so it felt a bit redundant.
Change-Id: I92cf6d08b83ef680fe37df29d3d546cad020955a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7888
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
While moving the CLI out of the evaluator, we forgot to update the
README in //tvix/eval. Move this up to //tvix, so people know where
to start.
Keep the instructions on how to build only `//tvix/eval` in `//tvix/
eval/README.md`.
Change-Id: Ie2755e8b5a0056225dbf3a0ee040f70f7f6a1f27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7887
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This calls out to Directory::validate() for all received Directory
messages, and also makes sure the sizes we refer a Directory message as
matches the sizes that have been calculated.
Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We can omit sending Directory messages to clients that have already been
sent in the same stream.
We can also omit storing a Directory message if we already have it -
they're content-addressed anyways.
Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The codebase contains a lot of complexity and odd roundabout
handling for shadowing globals. I'm pretty sure none of this is
necessary, and all of it disappears if you simply make the globals
part of the ordinary identifier resolution chain, with their own
scope up above the root scope. Then the ordinary shadowing routines
do the right thing, and no special cases or new terminology are
required.
This commit does that.
Note by tazjin: This commit was originally abandoned when Adam decided
not to take away reviewer bandwidth for this at the time (eval was
still in a much earlier stage). As we've recently done some
significant refactoring of globals initialisation this came up again,
and it seems we can easily cover the use-cases of the poison tracking
in other ways now, so I've rebased, updated and resurrected the CL.
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib3309a47a7b31fa5bf10466bade0d876b76ae462
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7089
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Import more things, and use expect_err to unpack the response.
Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This forces users to pass the fully constructed set of globals to the
VM, making it harder to accidentally "lose" the set while weak
references to it still exist.
This doesn't modify any functionality, but is laying the foundation
for simplifying some of the builtins behaviour that has grown more
complex again.
Change-Id: I5120f97861c65dc46d90b8a4e2c92ad32cc53e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7877
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
These will be threaded through to eval through the new `TvixError`
variant.
Change-Id: Ia0d3f8710dcf26bb95015cd2a6a2b2911f06343f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7842
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This adds a feature to the `#[builtins]` macro which lets users
specify an additional state type to (optionally) thread through to
builtins when constructing them.
This makes it possible for builtins-macro users to pass external state
handles (specifically, in our case, known path tracking) into a set of
builtins.
Change-Id: I3ade20d333fc3ba90a80822cdfa5f87a9cfada75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7840
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
All invocations of the builtin macro had to previously filter through
the `builtin_tuple` function, but it's more sensible to directly
return these from the macro.
Change-Id: I45600ba84d56c9528d3e92570461c319eea595ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7825
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This makes it possible for users to add additional context to an
error, which will then be rendered as an additional secondary span in
the formatted error output.
We should strive to do this basically anywhere errors are raised that
can occur multiple times, *especially* during type casts. This was
triggered by me debugging a type cast error attached to a fairly
large-ish span (a builtin invocation).
Change-Id: I51be41fabee00cf04de973935daf34fe6424e76f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This allows calling validate() on a derivation that doesn't have its
output paths already calculated yet. It allows offloading some of the
error checking in builtins.derivation* to be offloaded to that function.
Change-Id: Ib4aeadc0eb6583ef8cd765f33e9a9ec32be62729
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7848
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This uses the exposed StorePath::validate_name method to check for other
invalid output names (for which it would not be possible to construct a
store path of).
Change-Id: Ia3f65e19a07ef164f9f64013a5f37cbac99eb8e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7855
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is now used in more than just validate().
Change-Id: I69c3ad6cb5f3ad60a636fe2ea05d432aebe8e53b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7851
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
`drv` is an invalid output name too, as this would cause a
`builtins.derivation` call to return an attrset with a `drvPath` key
(which already exists) and has a different meaning.
Also handle errors during store path construction, and return our own
error type, instead of the ParseStorePathError.
Change-Id: Ib7952dde1d5cf18a0e210928df7c57b5939b7678
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7850
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
No situation should be allowed in which a path is inserted into
known_paths with different types twice, which we previously enforced
only for some path types.
Change-Id: I8cb47d4b29c0aab3c58694f8b590e131deba7043
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7843
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Instead of having a representation of suspended native thunks that
involves constructing a fake code chunk, make these thunks a
first-class part of the internal thunk representation.
The previous code was not that simple to understand, and actually
contained a critical bug which could lead to Tvix crashes. This
version fixes the particular instance of that bug, but instead
uncovers another (b/238) which can still lead to Tvix crashes.
Fixes: b/237.
Change-Id: I771d03864084d63953bdbb518fec94487481f839
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7750
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is unnecessary, Rc already provides all the boxing we need.
Change-Id: I08cf0939c48da43f04c847526c7e5dae5336d528
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7749
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
This is a somewhat terrifying hack that enables us to support
`builtins.builtins`, by running a "fake compilation" inside of a
suspended native thunk that can resolve the weak pointer to the
globals.
With this implementation, the thunk at `builtins.builtins` actually
resolves to the "real" `builtins` (verified with a new test).
This is kind of ugly, and it's something users shouldn't use, but
bubbling a warning out of this is difficult at the moment due to a
little bit of trickery with how the spans in suspended native thunks
work (they don't) (see b/237, b/238)
Change-Id: I67d0e93246dd5b279c960aeda00402031aa12af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Replacement strings are some weird internal feature of Nix that is
required for calculating derivation hashes. We need to track these
like other paths, as they need to be re-used on builds with
dependencies on values from previous builds.
Change-Id: Ie955b3fb5ae3685cfadfbe4d06ea6b5e219590c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7828
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
When adding things to a C++ Nix store, ensure that the path is tracked
in the tracker.
Since the mechanism for propagating the tracker instance isn't
finalised yet, I've opted to take an Rc<RefCell> parameter for it. How
exactly that ends up there is going to become clear in the next
commits, but for now it's just instantiated in main with
Default::default.
Change-Id: I90f0b44f2d4f292dedc98ff1aa39041d279b61fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7833
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This gets very complex very quickly otherwise, as all the construction
paths for a reference scanner and all the access patterns for the
KnownPaths structure are not yet fully understood.
Change-Id: Ibadf1f18b476695f3c286fc6896ae557760edf63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7827
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Instead of going through Vec/BTreeMap for generating our internal
types, use the proptest strategies from imbl.
The one thing I couldn't figure out in the previous implementation is
where the ranges/sizes of generated collections came from. The
strategies in proptest use different types (Range, with an unknown
default value, and SizeRange with 0..100). I've opted to specify
0..100 directly, but we can probably make it configurable.
Change-Id: I749bc4c703fe424099240cab822b1642e5216361
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7791
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This module implements types used to track the set of known paths in
the context of an evaluation.
These are used to determine the build references of a derivation.
Change-Id: I81e15ae33632784e699128916485751613b231a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7816
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
These need to be sorted anyways, so let's use the correct data structure for it.
Change-Id: I009c9989d7647dc1df716170f3680c981db6e4b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7846
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Different to the go-nix implementation, we don't have a custom "writing
mode" where output paths are omitted from output, but we simply run the
serialization with these fields unpopulated (during construction).
This means, there's no need for a custom writing mode that strips
outputs, so we can drop the comments.
Change-Id: Ic0aafd4e34e0294603490cfce2b1aef4085ff34b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7845
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This variant is required for external builtins (which in our case
includes `derivation`) to thread through reasonable error messages.
This has some potential for improvement, but it's an improvement over
the status quo of panicking in the external builtins when no
appropriate error is available.
Change-Id: I7e4bdb0a156c7717092dde30aa4785192182dc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7841
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
External implementors of builtins must be able to force values, which
necessitates publishing a bunch more items from the crate.
Change-Id: I8f6b8ae88156aae417dbe630a698d123d0c1c8d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7830
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This was put together by digging through the C++ Nix git history as
described in the second paragraph. Note that this is only based on
changes given when the language version was actually increased.
There is no guarantee that there have been no other changes inbetween
that would have warranted an increase as well.
Change-Id: I4ddee0d4ecafa1b3e5e1a867e9700d6c32e936ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7836
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
When constructing derivations inside builtins.derivationStrict, we'd
have to very frequently check whether certain outputs have already
been inserted into the derivation inputs.
Using a set type is much easier, especially as this has to be ordered
and the source data that is being inserted also comes from a set,
which might let us pass this more efficiently in the future.
Note that the validate function no longer checks the order of the
entries, as that is now guaranteed by the type.
Change-Id: I2fbb984facba3e668075f6f8df8992092368c63d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7826
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This CL addresses clippy warning len_without_is_empty
which expects `.is_empty()` method to be present when
implementing `.len()` method for an item.
Change-Id: I8878db630b9ef5853649a906b764a33299bb5dc8
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7806
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This explains my current thinking on string contexts. Thanks to
everyone who gave input so far.
Change-Id: I773219402a79a9d4753b4e7cfbf3a4a751a993a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7807
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Implements `Serialize` for `tvix_eval::Value`. Special care is taken
with serialisation of attribute sets, and forcing of thunks.
The tests should cover both cases well.
Change-Id: I9bb135bacf6f87bc6bd0bd88cef0a42308e6c335
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Using yet more machinery from the pretty comprehensive aho_corasick
crate, this makes it possible to pass anything implementing `io::Read`
to the `ReferenceScanner` to accumulate matches.
Change-Id: I5b0e28eb44ea4df24010f40831e29f2cbb8c1f80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7810
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This module implements a ReferenceScanner struct which uses the
aho_corasick crate to scan string inputs for known, non-overlapping
candidates (store paths, in our case).
I experimented with several different APIs, and landed on this version
with an initial accumulator in the scanner. The scanner is
instantiated from the candidates and "fed" all the strings, then
consumed by the caller to retrieve the result.
Right now only things that look vaguely like bytestrings can be fed to
the scanner, there is no streaming support in the API yet.
Change-Id: I7782f0f0df5fc64bccd813aa14712f5525b0168c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7808
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This CL address clippy warnings related to use of 'format!' macro
to return unmodified 'String'.
Change-Id: I88726e59d8f39f6a455a8c1f48075b52d167e489
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7804
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This just shuffles the Display implementations around so that
ErrorKind itself is displayable, which is useful in some situations
where errors under construction need to be type-converted.
Change-Id: I7b633d03d0dc34f345c4f20676e0023ecb1db0c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7802
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Some of the fields in a Derivation struct stay empty, and manually
creating BTreeMap or vec for it is annoying.
Derive Default instead, so we can use the defaults instead of writing
more by hand.
Change-Id: I5d41b4b55c8187cb101eb4266451a470008e0067
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7788
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This placeholder should not live in the main crate anymore as we will
be injecting the real one from outside of eval, but there are still
language tests that depend on a (simple, mockable) version of it.
Change-Id: I68ea169db15cbdbeed320930d3069e21e376c90d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7783
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Instead of concatenating STORE_DIR manually, use
StorePath::to_absolute_path() that does it for us.
Change-Id: Ia288851a05b4e339679db268f3dd7924e7b65586
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7782
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This project was not previously covered by CI (fixed in this commit),
so we didn't catch breakage due to a renamed module.
This was noticed while rebasing a CL that has a dependency on this
crate in its Nix build.
Change-Id: Ic48570b9313e5f73e14daab50cf7ea70918c94d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7778
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This embeds the build config directly at the point where `Cargo.nix`
is imported, making it transparent to library consumers.
Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
As discussed in #tvl, this is a more common term for it.
Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This exercises the output path calculation functions like a constructing
client (an implementation of builtins.derivation) would do.
It first assembles the bar derivation, does the output path calculation
on it, then continues with the foo derivation.
The code ensures the resulting Derivations match our fixtures.
Change-Id: If93f89c6622fac1c1941085083931b6f657c04bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7775
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This provides a function returning a string starting with the store
path prefix, the counterpart of `from_absolute_path`.
Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This will return the fixed output of a derivation (and its hash), or
None if the Derivation is not fixed-output.
It will simplify the logic in the output path calculation a bit.
Change-Id: I1066cc18ee4fc419421a8c5995c93ba91b35588f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7760
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This moves all the hash compression logic into a common helper function.
Also update the docstring, which said "path" here, which could have been
confused with output paths.
Change-Id: Iedfb59aeb24f7638afac669dcd18d57b6cfaece2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7759
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is marginally more efficient and has simpler bytecode.
Change-Id: Iad37c9aeef24583e8f696911bcd83d43639f2e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7769
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a mechanism to the compiler to compile an expression without
emitting any code. This allows for detected dead code to still be
compiled to detect errors & warnings inside of it.
Change-Id: Ie78479173570e9c819d8f32ae683ce34234a4c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7767
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This optimiser can rewrite some expressions into more efficient forms,
and warn users about those cases.
As a proof-of-concept, only some simple boolean comparisons are
supported for now.
Change-Id: I7df561118cfbad281fc99523e859bc66e7a1adcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a very minimal amount of additional Rc-increments (~1 per
compilation), but makes it a lot easier to add an AST-optimising
compiler pass without incurring a lot of extra cost.
Change-Id: I57208bdfc8882e3ae21c5850e14aa380d3ccea36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7765
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
With this, tvix/cli can be run on files and only produce compiler
errors and warnings.
Change-Id: I5dd9229fc065647787daafd17d7c1540579a1d98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7764
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This would make it possible to implement something like a linter based
on the tvix-eval compiler warnings.
Change-Id: I1feb4e7c4a44be7d1204b0a962ab522fd32b93c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7763
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
There was probably a misunderstanding somewhere about the
with_stack_size being related to how far away it is from the with, but
it is about whether there is a with at all.
This broke a warning (`UselessInherit`), and may actually have let to
more inefficient codegen in some cases.
Change-Id: I08338ea59ae39dad01ca8a4e09d934a936cdea2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7762
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
... without them, using the new Builtins API is basically impossible
for library consumers.
Change-Id: Ice0557a2e55e12d812f51bf5a99e6b8c91ad1b91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7755
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Code probably rarely relies on these, but it's not hard to support them.
Change-Id: I8499fec34efaf031f9c013bbd370a13db929a2a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7772
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
This will eventually force us to have a base builtins set in common with
C++ Nix, i.e. all 2.3 builtins except the controversial
builtins.valueSize.
Change-Id: I2c767f07d6a14711911658e87da9f18ede57a143
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7747
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This allows juggling with Derivation structs in unit tests, and makes it
very easy to compare them for equality.
Change-Id: I1faf2ec1eefb1a40fcee3c29f04ec47d29f22691
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7758
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Use the #[serde(rename = "…")] field attributes to match the field names
that Nix uses in its JSON output (nix show-derivation).
This allows us to just re-use the exact same fixtures from go-nix,
without manual post-massaging.
Change-Id: Ifd5c08e43cd4f50d5e02903eccd8cb37230b70a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7757
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This conveys better if an output of a Derivation is fixed-output or not,
and provides a Hash struct that can be used to store the algo and
digest.
In case it's not, this can simply be None. The serde field attributes
have been updated to still accept the same JSON.
We currently still store the hash algo and digest as strings, mostly
because the only thing populating it so far is the example JSONs.
We might want to update this, once actual Nix code populates this.
While updating write.rs, I pushed some of the Vec<String> to [&str]
conversions inline, and made it a Vec<&str>, because it was annoying to
juggle with.
Change-Id: Ia9cd0568fe179ac22a4a636237f22ab4ad92b95b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7746
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Allow others to peek into values. We should probably still restrict
this further at a later point.
Change-Id: I2831432038aa87c3c7dcc85af4fa76a4fe0eadff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7745
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
There's now a common constant for this, so we don't need to redefine
it here.
Change-Id: I48b096f5e623e5fc5bbd355db83b674766935f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7753
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This allows constructing a NixPath from an absolute path. It pops off
the STORE_DIR prefix and the trailing slash and returns an error if it
couldn't be found.
Change-Id: Ib540e353c63cc247ac15e20414b0db2caf695ef4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7751
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Implements externally tagged enum deserialisation. Other serialisation
methods are handled by serde internally using the existing methods.
See the tests for examples.
Change-Id: Ic4a9da3b5a32ddbb5918b1512e70c3ac5ce64f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7721
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Only missing enums at this point, but they're a bit of a beast.
Change-Id: I4ad47c034851f9a8794c81f39a5149a8ac1826e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7716
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
With this is_valid_nix_identifier should line up with the upstream lexer
definition:
ID [a-zA-Z\_][a-zA-Z0-9\_\'\-]*
While we're working on this, add a simple test checking the various
formatting rules. Interestingly, it would not be suitable as an identity
test, since you have to write
{ "assert" = null; }
in order to avoid an evaluation error, but C++ Nix is happy to print
this as
{ assert = null; }
– maybe should be considered to be a bug.
Change-Id: I0a4e1ccb5033a80f3767fb8d1c4bba08d303c5d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7744
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Previously the construction of globals (a compiler-only concept) and
builtins (a (now) user-facing API) was intermingled between multiple
different modules, and kind of difficult to understand.
The complexity of this had grown in large part due to the
implementation of `builtins.import`, which required the notorious
"knot-tying" trick using Rc::new_cyclic (see cl/7097) for constructing
the set of globals.
As part of the new `Evaluation` API users should have the ability to
bring their own builtins, and control explicitly whether or not impure
builtins are available (regardless of whether they're compiled in or
not).
To streamline the construction and allow the new API features to work,
this commit restructures things by making these changes:
1. The `tvix_eval::builtins` module is now only responsible for
exporting sets of builtins. It no longer has any knowledge of
whether or not certain sets (e.g. only pure, or pure+impure) are
enabled, and it has no control over which builtins are globally
available (this is now handled in the compiler).
2. The compiler module is now responsible for both constructing the
final attribute set of builtins from the set of builtins supplied
by a user, as well as for populating its globals (that is
identifiers which are available at the top-level scope).
3. The `Evaluation` API now carries a `builtins` field which is
populated with the pure builtins by default, and can be extended by
users.
4. The `import` feature has been moved into the compiler, as a
special case. In general, builtins no longer have the ability to
reference the "fix point" of the globals set.
This should not change any functionality, and in fact preserves minor
differences between Tvix/Nix that we already had (such as
`builtins.builtins` not existing).
Change-Id: Icdf5dd50eb81eb9260d89269d6e08b1e67811a2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7738
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds a function to generate the derivation path. The computation
is based on the Go implementation.
Change-Id: Iae89db4976f5fd9208f0453f73688689a245cd66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7729
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is just to clarify that the formatted result is the ATerm representation
Change-Id: I98fd0b1d2daa3cf6fdbe526ae1e0bd100ff62df3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7742
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This adds the implementation of fmt::Display for Derivation so that we can
easily store the formatted content as a string. Internally, we use the
serialization function to generate the string.
Change-Id: I6caca0d6c1bea3ca44b6c535c5b1d5d66d8413b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7741
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Rename PATH_HASH_SIZE to DIGEST_SIZE.
It's a digest, not hash (we don't necessarily have the internal hash
state anymore), and the fact it's about (Nix)Paths is already visible
from the module name.
Also expose ENCODED_DIGEST_SIZE, so we don't need to do the calculation
inside from_string() method, and it becomes more clear this is a
constant.
Change-Id: I0e7577dd7a6e503039037b986313b214e995d826
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7725
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Tested-by: BuildkiteCI
These can be accessed directly.
Change-Id: I71dc84f982820d53f319efefbed9b9055034954d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7724
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This provides validation of PathInfo messages, and ensures the output
hashes are properly parsed from the root node names.
NixPath already has a more extensive test suite for various wrong
NixPaths, so it's omitted from here.
Change-Id: I5d69118df5816daabb521ddb19d178bddd1caacf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7684
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows other crates to import tvix_store.
Rename the bin crate to tvix-store-bin, to avoid having multiple crates
with the same name (https://github.com/rust-lang/cargo/issues/6313)
Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This will make it possible fairly easily use Nix to represent
arbitrary data structures, e.g. for using Nix as a config language.
Only pure Nix (i.e. no `import` etc.) is supported for now.
Not all types, specifically no struct traversal, are implemented in
this commit.
Change-Id: I9ac91a229a0d12bf818e6e3249f3e5a691599a2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7712
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This makes it easier to interface this error with other crates.
Change-Id: I4947ea6097608f8c0427fb94a819ef748d94ea4b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7711
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This refactors the code to serialize a derivation. The original code
has beed moved to seperate crates for better code structure.
Change-Id: I3b1a6b134428fcbc9930c330bced8ec3610cfb4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7733
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/
tracing.
Annotate all method handlers with an instrument macro, and warn! a
message for them being unimplemented.
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools
like evans automatically discover available services, without having to
specify the path to the .proto files client-side.
It's behind a reflection feature flag, which is enabled by default.
Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This refactors how the original digest type (Vec<u8>) is converted
to [u8; PATH_HASH_SIZE].
Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This replaces the hello world example from tvix-store with an actual
gRPC endpoint, implementing all of BlobService, DirectoryService and
PathInfoService.
All RPC methods currently respond with the unimplemented gRPC status.
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Similar to cl/7682, we also want to make that error configurable.
Change-Id: I64f1a4570b3d75af4741abe10c2855959766e107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7708
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This implements the NixPath structure. NixPath allow to parse a string to a nix path. If the parsing fails, a DecodeError will be raised.
Change-Id: I28363cdcfb27f04bf21a11c0d130b461667e3720
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This implements the nix-specific base32 encoding and decoding, exposing
a subset of the API that the data-encoding crate provides.
Nix uses a custom alphabet, no padding, and encodes bytes in reverse
order. The latter one is the reason we can't just use the data-encoding
crate directly.
Three odd corner case tests ported over from go-nix failed. We opened
b/235 to further investigate.
Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We'll be using validate_node_name in other places in a bit, where
returning a ValidateDirectoryError is not appropriate.
Accept a function mapping a string to error as a second argument, and
pass ValidateDirectoryError::InvalidName at the current call sites.
Change-Id: I45cbb0deb4763061ad912c6b18a112c727795a17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7682
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
The `im::OrdMap` is already small and cheap to copy while sharing
memory, so this is not required anymore.
Only the `KV` variant may have slightly larger content, but in
practice this doesn't seem to make a difference when comparing the two
variants and this one is less complicated.
Change-Id: I64a563b209a2444125653777551373cb2989ca7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7677
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory
between different iterations of a map.
This slightly speeds up eval, but not significantly. Future work might
include benchmarking whether using a `HashMap` and only ordering in
cases where order is actually required would help.
This switches to a fork of `im` that fixes some bugs with its OrdMap
implementation.
Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
The conversion from im::Vector -> Vec is cheaper for NixList
construction (of course), so where possible we should make use of
that.
This updates most builtins dealing with lists to use Vector directly,
and marks the function constructing NixList from Vec as deprecated so
that we get appropriate warnings in places where it's still in use.
These places are currently inside of JSON serialisation logic which is
in flux right now, so lets leave them as-is until it's stabilised.
Change-Id: I037f12a2800f2576db4d9526bd935efd079163f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7671
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This adds a Derivation structure and allows to write it to a structure that implements std::fmt:Write.
The implementation is based on the go-nix version.
Change-Id: Ib54e1202b5c67f5d206b21bc109a751e971064cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7659
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is a persistent, structurally sharing data structure which is
more efficient in some of our use-cases. I have verified the
efficiency improvement using `hyperfine` repeatedly over expressions
on nixpkgs.
Lists are not the most performance-critical structure in Nix (that
would be attribute sets), but we can already see a small (~5-10%)
improvement.
Note that there are a handful of cases where we still go via `Vec`
that need to be fixed, most notable for `builtins.sort` which can not
currently be implemented directly using `im::Vector` because of a
restrictive type bound.
Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The only warnings left are unused warnings, but that'll change once we
have a real implementation, and not just tests.
Change-Id: I28912281b5e66735be37e999cc8ef4b8b09028fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7669
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
lazy_static is only used in tests, and anyhow isn't used at all (yet).
This can be dropped.
Change-Id: Ic41ff3f9bb93cfa600c3485e85464f78a3976504
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7668
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Move them from the bottom of src/proto.rs to its own src/tests/mod.rs.
Also drop the test_ prefix, this is not golang.
Change-Id: I2e0b6b9812264f3d9721c0766936f08157fadc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7667
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Stat exposes metadata about a given blob,
such as more granular chunking, baos.
It implicitly allows checking for existence too, as asking this for a
non-existing Blob will return a Status::not_found grpc error.
The previous version returned a Status::not_found error on the Get
request too, but there was no chance to prevent the server from starting
to stream (except sending an immediate cancellation).
Being able to check whether something exists in a BlobStore helps to
prevent from uploading in first place.
The granular chunking bits are an optional optimization - if the
BlobStore implements no more granular chunking, the Stat response can
simply contain a single chunk.
Read returns a stream of BlobChunk, which is just a stream of bytes -
not necessarily using the chunking that's returned in the reply of a
Stat() call. It can be used to read blobs or chunks.
Change-Id: I4b6030ef184ace5484c84ca273b49d710433731d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7652
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This changes the RPC methods to return/consume a stream of chunks, instead of a
very big message containing the whole blob, to keep message sizes in manageable
sizes (less than 4MiB).
Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Expose the NAR calculation to a separate `CalculateNAR` method, which
responds with the NAR size and sha256 hash.
Contrary to what cl/7618 and cl/7620 initially did, don't add different
other request types.
In the CalculateNARResponse message, there's now some duplication in the
(optional) `narinfo` field of a PathInfo, but I'm not entirely sure if
we want to drop the fields from there yet.
Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is gonna get used in another place in a second.
Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I347c11c8d24379628b7ed09d2c90670c576e686a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7617
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
While there's currently nothing in here checking the size of the digest,
we should use something that passes the to-be-introduced validate()
function.
Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Validate thecks the Directory message for invalid data, such as:
- violations of name restrictions
- invalid digest lengths
- not properly sorted lists
- duplicate names in the three lists
Change-Id: I8d43a13797793c64097e526ef3bd482c9606c87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7648
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds Size() and Digest() functions for the golang version.
Change-Id: If71445a9bb26100bb4076ac4f5c96945b33919f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7325
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This allows importing the generated .pb.go files into other go projects.
I initially looked at buildGo.protos, but it doesn't work for multi-.proto
files, and actually having LSP support for the generated structs is nice, too.
Change-Id: Idbd448008010790a10a0ea42e4059dbb609eaf1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7322
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Change-Id: I0898b8a0a78e704219da38e5acaabef1e640d4e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7321
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
These are intended to help digest the protocol definitions for tvix-
store, and how they tie into the whole concept.
Co-Authored-By: Vincent Ambo <mail@tazj.in>
Change-Id: Ic1ba3ba41ef599209453f15d0ac2e07a6144bcca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7439
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
While prost-build already exposes protobuf message types as structs, we
actually need tonic-build too, to be able to get traits for all the RPC
services defined in the proto files.
Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
It's been a while since the last time, so quite a lot of stuff has
accumulated here.
Change-Id: I0762827c197b30a917ff470fd8ae8f220f6ba247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7597
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Introduces continuation-passing-based trampolining of thunk forcing to
avoid recursing when forcing deeply nested expressions.
This is required for evaluating large expressions.
This change was extracted out of cl/7362.
Co-authored-by: Vincent Ambo <tazjin@tvl.su>
Co-authored-by: Griffin Smith <grfn@gws.fyi>
Change-Id: Ifc1747e712663684b2fff53095de62b8459a47f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7551
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
... if they are known. We currently do not propagate names correctly
for curried functions.
Change-Id: I19d57fb30a5c0000ccdf690b91076f6b2191de23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7596
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This value creates a human-readable explanation of a value. This can
be used to implement documentation related functionality.
For some values, the amount of information displayed can be expanded
quite a bit.
Change-Id: Ie8c400feae909e7680af163596f99060262e4241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7592
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This type allows for temporarily compatibility with the C++ Nix store,
specifically (for now) it gives us the store directory used by Nix and
imports files the same way.
Change-Id: I4767794ef2863eba49661315c63c4e17de946d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7587
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
`mg run //tvix:crate2nixGenerate --` runs crate2nix generate and
depotfmt's it afterwards.
This removes a frequent point of friction, because the Cargo.nix emitted
by crate2nix needs another formatter.
Change-Id: I649495980718cd3847d4cff77c9d4bfcb599387c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7612
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This provides an interface to retrieve and upload single Directory
messages, or a DAG of them.
Change-Id: Id9e7084bd697d843649a122da2c992a3e36d808c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7137
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
This defines a service that can be used to upload and retrieve metadata
of nix paths.
Change-Id: Id86eb531ce4ae316adb15934b0d1386a14ba2132
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Having a multi-line docstring yields multiple doc-attributes in order,
however we were previously discarding all but the first one.
This reduces them into a single string instead, which can then be
displayed as multi-line documentation.
Change-Id: I1f237956cdea2e4c746d3f13744e0373c1c645a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7594
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This "ties the knot" of importing files into a store when referring
to them through path literals, e.g. inside of strings.
I'm not yet sure if this interface is sufficient for
builtins.path (which we haven't implemented at all yet), but it's
enough to wire up eval & store initially.
In the default implementations nothing interesting happens in this
function at all.
Change-Id: Ie01ff4161617d1e743a68dbd1a5e54c1b40c0990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7582
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Returns the store directory through EvalIO::store_dir.
Note that this is _optional_ in Tvix, as an evaluation can occur in a
context where there simply is no store directory. In those contexts,
`builtins.storeDir` returns `null` in Tvix.
This would only happen in contexts like Tvixbolt (or completely
unrelated use-cases) in practice.
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: I5a752c7e89b2f75bd7efb082dbfa5b25e3b1ff3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7452
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This shouldn't be available if we've built a "pure" crate.
Change-Id: I7c85827ee212890252ff7e0b6242e2c52618cba5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7572
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
... until we have a store-I/O layer, or something that intercepts the
store-related stuff appropriately.
Change-Id: I22f63435b3f9e118e3faeb2924fda8373a23ea7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7568
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
With this change, the behaviour of reading a string from a file path
is controlled by the provided `EvalIO` structure.
This is a huge step towards abstracting away I/O behaviour correctly.
Change-Id: Ifde8e46cd863b16e0301dca45a434ad27560399f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7567
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This lets users set the `io_handle` field on an `Evaluation`, which is
then propagated to the VM.
Change-Id: I616d7140724fb2b4db47c2ebf95451d5303a487a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7566
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This trait is going to be used to abstract filesystem interactions in
Tvix.
For now, it only contains a `read_to_string` method that closely
mirrors `std::fs::read_to_string`.
As a first step, to see how this works in practice, we will thread
through only this function to the various relevant parts.
Two implementations are provided in tvix-eval itself: A dummy
implementation (which just returns ErrorKind::NotImplemented for all
operations), and a std implementation which delegates to `std`
functions.
Change-Id: Ied3e3bf4bd0e874dd84e166190e3873a0f923ddb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7565
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This type carries the information required for calculating a
span (i.e. the chunk and offset), instead of the span itself. The span
is then only calculated in cases where it is required (when throwing
errors).
This reduces the eval time for
`builtins.length (builtins.attrNames (import <nixpkgs> {}))` by *one
third*!
The data structure in chunks that carries span information reduces
in-memory size by trading off the speed of retrieving span
information. This is because the span information is only actually
required when throwing errors (or emitting warnings).
However, somewhere along the way we grew a dependency on carrying span
information in thunks (for correctly reporting error chains). Hitting
the code paths for span retrieval was expensive, and carrying the
spans in a different way would still be less cache-efficient. This
change is the best tradeoff I could come up with.
Refs: b/229.
Change-Id: I27d4c4b5c5f9be90ac47f2db61941e123a78a77b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7558
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Having thunks which, when forced, execute native Rust code rather
than interpreted opcodes lets us avoid having to bundle
`src/libexpr/primops/derivation.nix` like cppnix does by implementing
it in Rust instead.
Change-Id: If91d77a6736234321eee87ba4b4777eed5a3fe1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7450
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Instead of finding locals by doing 2x O(n) walks over the compiler's
locals list, use a secondary name-based index for resolving locals by
name.
Previously, almost 60% (!!) of eval time on some expressions over
nixpkgs was spent in `Local::has_name`. This function doesn't even
exist anymore now, and eval speed about doubles as a result.
Note that this doesn't exactly make the locals code easier to read,
but I'm also not sure what we can simplify in there in general.
This fixes b/227.
Change-Id: I29ce5eb9452b02d3b358c673e1f5cf8082e2fef9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7560
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This is done in the form of some comments on the functions, as well as
a functional doctest that writes a single file to a NAR.
Change-Id: Ic97ebd439e91d6b076685807fe70de098ec02575
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7599
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Users can again pass flags for dumping the AST, bytecode, and runtime
trace.
With this commit the CLI is at feature-parity with what we had before,
but entirely through the new API.
Change-Id: I30fe26f243224b25d1e4f828fec607325ef88306
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7550
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
With this change, it should be possible to have both existing
use-cases (CLI & Tvixbolt) use the same API.
Change-Id: I2195264f08cc892177b559a28660dc5f98e48e41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7545
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This is required for passing through NIX_PATH from the CLI.
Change-Id: If129df79ef9c3ffab31408adb85679909276c4f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7544
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This should make no difference in Nix builds, but allows running tests
locally again with `cargo test` for //tvix/eval.
Change-Id: I97d61840143d5c14db61d5862781bf635f9a28e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7590
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In //tvix/eval:
* criterion bumped to 4.0, which at least depends on clap 3.x instead
of 2.x, which is less incompatible
In //tvix/cli:
* no changes required
In //tvix/nix_cli:
* some minor changes for compatibility with clap 4.0, no functionality
changes
Change-Id: If793f64b59fcaa2402d3d483ddbab4092f32df03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7588
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
The tvix-eval project is independent from any *uses* of the evaluator,
such as the tvix-repl.
This functionality has been split out into a separate "tvix-cli"
crate. Note that this doesn't have to mean that this CLI crate is the
"final" CLI crate for tvix, the point of this is not "getting the CLI
structure right" but rather "getting the evaluator structure right".
This reshuffling is part of restructuring the way that functionality
like store communication is injected into language evaluation.
Note that at this commit the new CLI crate is not at feature-parity.
Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This removes internal uses of the previous crate::eval module, which
is being removed.
Change-Id: I5fb3c53460a9c5381853d0258f9ed074ab23c630
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7543
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
A step towards something more like how I imagine the future public API
for tvix-eval. Please note that this is definitely not the final
version yet, but it's better than the previous API that either exposed
a side-effecting blackbox, or a very low-level "interface".
The basic idea is that an evaluation of some Nix code is requested by
a caller with various parameters, but not all callers are interested
in all of these parameters.
There are also some bits of information that are returned from an
evaluation that are not necessarily relevant to all callers.
To support this somewhat ergonomically, the API is built around an
`Evaluation` struct that is configured by the caller with the various
parameters and then "executed".
Change-Id: I71826f3897126898adc2873d31c44d3eaf5c2be0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7542
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This defines a service that can be used to get and put content-addressed
chunks of data.
Change-Id: I36cf2278ed1daf71848c04fdfd14450b2268c5de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Manually maintained shell target based on the previous "loose"
`shell.nix` file.
We might want to have a function that combines the dependencies of all
the targets automatically, but at a quick glance that was actually
non-trivial so I'm leaving it as an exercise for someone else.
Change-Id: I74754940088f1b58e3b6754fb782470c80ea4292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7540
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Introduces granular dependency builds using crate2nix, bootstrapped
off the generated configuration from the newly introduced
workspace (see cl/7533).
This commit checks in the generated Cargo.nix file which can be
regenerated with a parameterless invocation of `crate2nix generate` in
`//tvix`. I tried generating this in IFD, but it turned out to be
harder than what seemed worthwhile for now.
In this setup, the various build targets for Rust projects end up
being attributes of the imported `Cargo.nix` file at the `tvix.crates`
attribute. These still lack configuration, however, which has been
fixed in the various `default.nix` files of individual projects.
Note that we (temporarily) lose the ability to build tvix-eval's
benchmarks in CI. I haven't figured out what magic incantation summons
them from the void again ...
The `eval-okay-readDir` tests from both test suites have been disabled
because they fail for unknown reasons when run in this new derivation.
Somebody will have to debug it!
Change-Id: I2014614ccb9c8951aedbd71df7966ca191a13695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7538
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This relates to the (abandoned) cl/7256.
Introduces a Cargo workspace at //tvix that is primarily intended to
be used as a workaround for the annoying Nix+Rust tooling while having
a consistent set of dependencies.
This is driven in part by a desire to adopt crate2nix and get more
granular Nix builds for Tvix's Rust projects, and in part by a need to
split //tvix/eval into something providing the CLI (REPL etc.), and a
library providing eval, without significantly altering the structure
of build targets.
To accomplish this the workspace has been designed to allow projects
to remain independent build targets. I want to avoid lumping all the
projects together - something like //tvix/eval should always be
independent of other parts of tvix.
A helper function in //tvix/default.nix lets downstream naersk
projects construct a sparse root for the project which combines the
workspace's `Cargo.lock` with the project's own `Cargo.toml`.
Note that cargo commands in the workspace itself require the build
dependencies of _all_ projects to be present, which is currently a bit
annoying to accomplish.
This introduces some breakage:
1. It breaks usage of rust-analyser without being in a shell with the
dependencies of *all* Tvix projects, as it is not capable of
respecting only the subset of dependencies for a part of the
workspace.
2. It is no longer possible to run tests using `cargo test`, as the
test generation crate we use does not work with workspaces:
https://github.com/frehberg/test-generator/issues/6
This still works in the Nix build as we construct a Cargo project
that looks like it's not in a workspace there. Until somebody fixes
that crate / writes a new macro / does something else with the test
suite, the way to run the tests is through the Nix build.
Long-term we'll probably want to get rid of cargo completely, it's
just a big wart and most tooling works without it if correctly
configured, but we don't have time for that now.
Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This .envrc file causes additional software to be loaded, but there is
no way to opt-out even if the software is not desired:
1. If I opt-in (i.e. `direnv allow`) the file, additional stuff that I
don't need is injected into my environment in a blocking way.
2. If I opt-out, *all* of the depot configuration (including `mg`) is
unloaded, as direnv configurations do not trivially nest.
I have to work around this constantly by making the file contain just
the line `source_up` and then setting `--assume-unchanged` on the git
index to avoid accidentally committing the file changes.
This is kind of silly, the people who *want* this stuff to be loaded
should devise a mechanism that loads it automatically but is opt-in.
This could be done e.g. by gating something on environment variables,
or having a shell hook, or whatever. Breaking expectations that hold
elsewhere in depot is not okay, however.
If you manually want a shell for a project, run `mg shell` either with
a target specification for that project (e.g. `mg shell //tvix/store`)
or in the project's folder.
You can also just use standard nix-shell invocations.
Change-Id: I0de43378424d0cb1e1279c72c47940fecf497bf0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7531
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
A few weeks ago, oberblastmeister did a release to crates.io so we can
stop importing it via GitHub.
Change-Id: I9d5fa5cd281685779c71b12fed45ed201a1db17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7532
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This uses prost-build to build the proto files.
Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
This adds the PathInfo message, which hosts information about a Nix
Store path, mapping to either of a {Directory,File,Symlink}Node.
Change-Id: I79d871b6fad450d6a4ae4101fb72c51f9a83471f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7132
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This moves the castore.proto file into the //tvix/store/protos directory. Per-
component protos might make more sense, than a "tvix-wide" proto directory.
Change-Id: Ie728210174b041e9285a0e2ac605d715d0f0cbda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7285
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Fixes b/212. Based on feedback in https://cl.tvl.fyi/c/depot/+/7492, all
uses of `NixAttrs::from_map` have been removed. Only `from_iter` and
`from_kv` remain.
Change-Id: I52e25f73018c2aa1843197427516b7a852503e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7500
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: IslandUsurper <lyle@menteeth.us>
Allows for the removal of some BTreeMap usage when constructing NixAttrs
by allowing any iterator over 2-tuples to build a NixAttrs. Some
instances of BTreeMap didn't have anything to do with making NixAttrs,
and some were just the best tool for the job, so they are left using the
old `from_map` interface.
Change-Id: I668ea600b0d93eae700a6b1861ac84502c968d78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7492
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Maybe counter-intuitively the inner elements of a list or the
attribute values of an attribute set will be forced despite
pointer equality (but only one layer deep).
Change-Id: I485d96452fb56f5fb342d39039c9137725b33d3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7371
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
This came up in the Nix Language channel today and I thought it
warranted a test case.
We did actually implement this correctly.
Change-Id: I4b37c92d06eb6e3a7f59ea3d10af38f2b0a93d53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7493
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Rust doesn't do tail-call elimination (still!) so the best we can
hope for here is to inline non-recursive invocations.
Change-Id: I78949967e48b006fcbf31786d8f6281cd122f36f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7360
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Before this, tvix was spending most of its time furiously re-parsing
and re-compiling nixpkgs, each time hoping to get a different result...
Change-Id: I1c0cfbf9af622c276275b1f2fb8d4e976f1b5533
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7361
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
With this change, the test introduced by cl/7370 passes.
Change-Id: Ie7d2f02a59d61151f14ebd328e6cfa5892cacfb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7375
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: Adam Joseph <adam@westernsemico.com>
This passes all the function/thunk-pointer-equality tests in
cl/7369.
Change-Id: Ib47535ba2fc77a4f1c2cc2fd23d3a879e21d8b4c
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7358
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The backtrace-on-stack-overflow create provides best-effort stack
traces when a stack overflow happens. Since it's running on the
(usually tiny) signal alternate stack this isn't easy.
This is guarded by a new `backtrace_overflow` feature flag and never
enabled (even if that feature is selected) for release builds. This
is strictly for debugging; there's crazy unsafe voodoo in there.
https://lib.rs/crates/backtrace-on-stack-overflow
Example output:
```
Stack Overflow:
0: backtrace_on_stack_overflow::handle_sigsegv
at /home/amjoseph/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-on-stack-overflow-0.2.0/src/lib.rs:93:40
1: <unknown>
2: __rust_probestack
3: tvix_eval::vm::VM::run_op
at src/vm.rs:399
4: tvix_eval::vm::VM::run
at src/vm.rs:388:23
5: tvix_eval::vm::VM::enter_frame
at src/vm.rs:360:22
6: tvix_eval::value::thunk::Thunk::force
at src/value/thunk.rs:116:25
7: tvix_eval::vm::VM::run_op
at src/vm.rs:801:37
8: tvix_eval::vm::VM::run
at src/vm.rs:388:23
9: tvix_eval::vm::VM::enter_frame
at src/vm.rs:360:22
10: tvix_eval::value::thunk::Thunk::force
at src/value/thunk.rs:116:25
...
```
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I1d8a2017f836be7bf91a2223e7adacb86fa1dbb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7354
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
See cl/7368
Change-Id: I97630994c3d65f4d16414a0da236ce000a5b6d33
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7374
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
See cl/7372; Nix equality semantics require the ability to track
pointer equality of upvalue-sets.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I82ba517499cf370189a80355e4e46a5caaab7153
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7373
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This test case checks two things:
* A sanity check that "pointer equality for functions" means not
just the lambda, but also the upvalues.
* To be pointer-equal, it is not enough for the upvalues to be
normal-form equal (i.e. `nix_eq()`-equal); the upvalues must be
*pointer*-equal. The second part of the test case checks for
this.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I4e59327a6f199b8212e97197b212e3c3934bb3f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7372
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
The "dep:" syntax in Cargo.toml is very new; crate2nix master has
support for it, but they have not yet made a release with this
update, and therefore the crate2nix in nixpkgs does not yet support
it.
Could we avoid using "dep:" for a few weeks to give crate2nix a
chance to release so I can bump the version in nixpkgs? I've opened
an issue asking crate2nix to make a release:
https://github.com/kolloch/crate2nix/issues/264
I propose that if they haven't acted within a month we stop waiting
and revert this at that time.
Change-Id: I999a72429db667bedf4b2cdba27cb63b3f3d9657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7350
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
When we start unrecursivifying (sp?) things, Rust's borrow checker
is going to be a headache; its magic only works when you use the CPU
stack as your call stack.
Fixing the borrow checker issues usually involves adding lots of
`clone()`s. Right now `NixList` is the only variant of `Value` that
isn't cheap to clone() -- all the others are either a wrapper around
Rc or else are of bounded size.
Note that this requires dropping the `DerefMut for NixList` instance
and using `Vec<Value>` instead in those situations.
Change-Id: I5a47df66855342aa2064f8f3cb7934ff422d26bd
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7359
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
When comparing Nix values for equality, an issue can occur where
recursive values contain thunks to themselves which causes borrow
errors when forcing them for comparison later down the line.
To work around this we clone the values for now. There might be some
optimisations possible like checking for thunk equality directly and
short-circuiting on that (we have to check what Nix does).
Change-Id: I7e75c992ea68f100058f52b4b46168da7d671994
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7314
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
When passing multiple arguments, every intermediate callable needs to
be forced as this is expected by the VM's call_value function.
Also adds a debug assertion for this which makes it easier to spot
exactly what went wrong.
Change-Id: I3aa519cb6cdaab713bd18282bef901c4cd77c535
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7312
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This function covers builtins.genericClosure, seemingly including
weird behaviour around the order in which the work set is processed.
For some reason, in C++ Nix the test expectation is written in XML
which we do not yet support, so I have created a new expectation file
using `nix-instantiate --eval --strict` on the file (yes, using C++
Nix).
Change-Id: Id90e7117d120dc66d963a51083c4d8e8f2d9f181
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7311
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This implementation closely follows the original implementation in
Nix, including the use of an equality-based "set" structure to track
keys that have already been processed.
Note that this test does not yet enable the `notyetpassing` test for
builtins.genericClosure because (for as of yet unknown reasons) this
test compares against XML output (however, evaluating the test case
actually does work).
This takes us one step closer to nixpkgs eval.
This commit was written somewhere in the North Sea.
Co-Authored-By: Griffin Smith <root@gws.fyi>
Change-Id: I450a866e6f2888b27c2fe7c7f77ce0f79bfe3e6c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7310
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Both //tvix/eval and //tvix/nix_cli have need to for rust tooling available
in $PATH.
Move this one level up, so it's accessible in all subdirectories.
Change-Id: I0763bbe9cefdc962f3a8f86c51e8f67cde8b4b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7248
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This detects if the second argument of a division is a zero (either as integer
or as float). If so, an error message is displayed.
This fixes b/219.
Change-Id: I50203d14a71482bc757832a2c8dee08eb7d35c49
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7258
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
As per the discussion in
https://cl.tvl.fyi/c/depot/+/7128/2..5/tvix/proto/castore.proto#b39, ref
sounds more like an external, stateful ID. Call this `digest`, to make
clear it's precisely this.
Change-Id: I81dd3769e2ce017de470ae92f72a38fb72015f10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7134
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
`buf` doesn't like protos with different package names in the same
directory.
Change-Id: I30806b46b88f103779faa40466461091a4a01e06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7130
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Make it clear these are symlinks, not hardlinks.
The term "link" is too heavily correlated to other meanings in IPFS/IPLD
world, and calling this symlink removes this confusion.
Change-Id: Id3f1eaa32098510b05f3e1a1348344503bcb4d5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7129
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Add a new `documentation: Option<&'static str>` field to Builtin, and
populate it in the `#[builtins]` macro with the docstring of the builtin
function, if any.
Change-Id: Ic68fdf9b314d15a780731974234e2ae43f6a44b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7205
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Refactor the arguments of a Builtin to be a vec of a new BuiltinArgument
struct, which contains the old strictness boolean and also a static
`name` str - this is automatically determined via the ident for the
corresponding function argument in the proc-macro case, and passed in in
the cases where we're still manually calling Builtin::new.
Currently this name is unused, but in the future this can be used as
part of a documentation system for builtins.
Change-Id: Ib9dadb15b69bf8c9ea1983a4f4f197294a2394a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7204
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Some new top-level re-exports (specifically VM, Builtin, and ErrorKind)
were added to lib.rs in tvix/eval to allow the builtin-macros tests to
work - we should be clear which of these are part of the public
interface (I think it's reasonable for ErrorKind to be) and which
aren't (specifically I'm not sure VM and Builtin necessarily should be,
at least yet).
Change-Id: I3bbeaa63cdda9227224cd3bc298a9bb8da4deb7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7203
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Similar to what we did with pure builtins, define the impure builtins
within a module at the top-level using the new #[builtins] attribute
macro
Change-Id: Ie5d5135d00bb65e651531df6eadba642cd4eb08e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7202
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Break out all pure builtin functions to top-level functions defined
within the `pure_builtins` module in `builtins/mod.rs`.
Change-Id: I9a10660446d557b1a86da4c45a463e9a1a9b4f2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7201
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Mostly as a proof-of-concept of the new proc-macros for defining
builtins, define a single builtin (the first in the list, `abort`) at
the top-level of a child module within builtins/mod.rs, and add it to
the list of builtins returned from `pure_builtins`.
If this works nicely, we can start breaking out the rest of the builtins
into the top-level too, in addition to introducing additional sets of
builtins (to differentiate between pure and impure builtins).
Change-Id: I5bdd57c57fecf8d63c9fed4fc6b1460f533b20f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7199
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Add a single new proc macro to a new proc-macro crate,
`tvix-eval-proc-macros` for defining an inline module containing nix
builtins, and automatically generating a function within that module
which returns a list of those builtins as `tvix_eval::value::Builtin`.
Change-Id: Ie4afae438914d2af93d15637151a49b4c68aa352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7198
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
This commit adds a markdown document which explains how the
thread-local VM infrastructure works, in case it is useful in the
future.
Change-Id: Id10e32a9e3c5fa38a15d4bec9800f7234c59234a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7193
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
This is more generally useful than just inside the VM, until it is
stabilised in Rust itself.
Change-Id: Id9aa3d5b533ff38e3d2c6b85ad484394fdd05dcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7186
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
This fixes a mistake I made in d978b556e6.
Change-Id: I88db697105a7149e9785f6aface03bff68566d2b
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7085
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Scope_depth and with_stack_depth were being reset to zero for nested
function abstractions. Fortunately nothing depends on them being
computed correctly in these cases, but it sure was confusing.
Change-Id: I59980b6a5aff043f60079f97211220b0086eb97d
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7091
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
It is very confusing that this opcode is called DataLocalIdx, but it
carries a StackIdx rather than a LocalIdx. It seems like this
really ought to be called DataStackIdx, but maybe I've
misunderstood; if so please explain it to me.
Change-Id: I91f6ffa759412beef0b91d3c19ec0d873fe51b99
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7088
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This implements builtins.split, and passes eval-okay-regex-split.nix
(which is moved out of notyetpassing).
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ieb0975da2058966c697ee0e2f5b3f26ccabfae57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7143
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
We have to be careful implementing `builtins.groupBy`, since the
list may contain thunks, and tvix's to_xxx() functions do not work
on thunks.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I182b6fc2d4296f864ed16744ef70b153e8e6978a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7039
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The impl Display for NixAttrs needs to wrap double quotes around any
keys which are not valid Nix identifiers. This commit does that,
and adds a test (which fails prior to this commit and passes after
this commit).
Change-Id: Ie31ce91e8637cb27073f23f115db81feefdc6424
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7084
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The variable name `local_idx` is used here for a StackIdx, which invites
confusion.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I2e22db90acdc0d29586ee5b72ea18d42d93badcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7086
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
If self.depth > other.depth then self is deeper than other, so self
is *below* other, not above it. Let's just inline the function.
Change-Id: I8dda3d90cbc86c8a6fa01bc4a5e506a2e403bd20
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7090
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
It isn't possible to implement PartialEq properly for Value, because
any sensible implementation needs to force() thunks, which cannot be
done without a `&mut VM`.
The existing derive(PartialEq) has false negatives, which caused the
bug which cl/7142 fixed. Fortunately that bug was easy to find, but
a silent false negative deep within the bowels of nixpkgs could be a
real nightmare to hunt down.
Let's just remove the PartialEq impl for Value, and the other
derive(PartialEq)'s that depend on it.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Iacd3726fefc7fc1edadcd7e9b586e04cf8466775
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7144
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I think we should bring this into $PATH too.
Change-Id: Ie31ac558355b7c4ed9dcd3dd60e1b03f141d1178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7166
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
In ripple, this is used for the internal on-disk format, but it's not
suitable for remote consumption.
Change-Id: I327361a2254566ac9216e23eaed36dba8fdd283b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7127
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
This is
0d4906cbabb183caa96e763671810fb39bd0c935:ripple/fossil/src/store.proto,
from https://src.unfathomable.blue.
It is not pulled in as a subtree, as some of the commits in there break
index-pack:
```
remote: error: object 2f487c3bf7cd8efd64f1d217edac732db98ff1c0: badDateOverflow: invalid author/committer line - date causes integer overflow
remote: fatal: fsck error in packed object
error: remote unpack failed: index-pack abnormal exit
```
Co-authored-by: edef <edef@unfathomable.blue>
Change-Id: I3369044090a3192e2322775a335887c37536a942
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7126
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
The current implementation of nix_eq will force one level of thunks
and then switch to the (non-forcing) rust Eq::eq() method. This
gives incorrect results for lists-of-thunks.
This commit changes nix_eq() to be recursive.
A regression test (which fails prior to this commit) is included.
This fix also causes nix_tests/eval-okay-fromjson.nix to pass, so it
is moved out of notyetpassing.
Change-Id: I655fd7a5294208a7b39df8e2c3c12a8b9768292f
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7142
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We're getting close to the finish line, folks.
I went through the list of builtins and there are only 33 that
remain unimplemented. I've marked them, and indicated which are
ready to be implemented vs which are waiting for other things.
We can delete this column from the table once everything is
implemented.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Idfaef93283536288b12e59aef5c3e1cd139bd133
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7140
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
I believe that the currentTime, findFile, hashFile, pathExists,
readDir, path (unless ?sha256), and readFile builtins are impure.
This commit marks them as such in docs/builtins.md.
Change-Id: Ib1b59fe643dde73cb2b00050b4ef9d3401ad22eb
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7139
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is a bit tricky because the comparator can throw errors, so we
need to propagate them out if they exist and try to avoid sorting
forever by returning a reasonable ordering in this case (as
short-circuiting is not available).
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: Icae1d30f43ec1ae64b2ba51e73ee467605686792
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7072
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Lists are compared lexicographically in C++ nix as of [0], and our
updated nix test suites depend on this. This implements comparison of
list values in `Value::nix_cmp` using a very similar algorithm to what
C++ does - similarly to there, this requires passing in the VM so we can
force thunks in the list elements as we go.
[0]: 09471d2680#
Change-Id: I5d8bb07f90647a1fec83f775243e21af856afbb1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7070
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
CL/7034 looks great, except that for a length-N target string it
will perform N deep copies of each of the from and to-lists. Let's
use references instead of clones.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Icd341213a9f0e728f9c8453cec6d23af5e1dea91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7095
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I assumed that AttrsRep::KV represented attrsets with a single
attribute as a Key-Value pair. That is not the case. Let's warn
other people about this.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ie3d2765fcc1ab705c153ab94ffe77bbd6d4ab39e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7093
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Rustc uses wasm32-unknown-unknown, which is rejected by config.sub,
for wasm-in-the-browser environments. Rustc should be using
wasm32-unknown-none, which config.sub accepts. Hopefully the rustc
people will change their triple before stabilising this triple. In
the meantime, we fix it here in order to unbreak tvixbolt.
https://doc.rust-lang.org/beta/nightly-rustc/rustc_target/spec/wasm32_unknown_unknown/index.html
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I941fd8d6f3db4e249901772fd79321ad88cd9cc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7107
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This commit contains two search-and-replace renames which are broken
out from I04131501029772f30e28da8281d864427685097f in order to
reduce the noise in that CL:
- `is_thunk -> is_suspended_thunk`, since there are now
OpThunkClosure and OpThunkSuspended
- `compile_lambda_or_thunk` -> `compile_lambda_or_suspension`
Change-Id: I7cc5bbb75ef6605e3428c7be27e812f41a10c127
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7037
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
CL/6867 added support for builtins.import, which required a cyclic
reference import->globals->builtins->import. This was implemented
using a RefCell, which makes it possible to mutate the builtins
during evaluation. The commit message for CL/6867 expressed a
desire to eliminate this possibility:
This opens up a potentially dangerous footgun in which we could
mutate the builtins at runtime leading to different compiler
invocations seeing different builtins, so it'd be nice to have
some kind of "finalised" status for them or some such, but I'm not
sure how to represent that atm.
This CL replaces the RefCell with Rc::new_cyclic(), making the
globals/builtins immutable once again. At VM runtime (once opcodes
start executing) everything is the same as before this CL, except
that the Rc<RefCell<>> introduced by CL/6867 is turned into an
rc::Weak<>.
The function passed to Rc::new_cyclic works very similarly to
overlays in nixpkgs: a function takes its own result as an argument.
However instead of laziness "breaking the cycle", Rust's
Rc::new_cyclic() instead uses an rc::Weak. This is done to prevent
memory leaks rather than divergence.
This CL also resolves the following TODO from CL/6867:
// TODO: encapsulate this import weirdness in builtins
The main disadvantage of this CL is the fact that the VM now must
ensure that it holds a strong reference to the globals while a
program is executing; failure to do so will cause a panic when the
weak reference in the builtins is upgrade()d.
In theory it should be possible to create strong reference cycles
the same way Rc::new_cyclic() creates weak cycles, but these cycles
would cause a permanent memory leak -- without either an rc::Weak or
RefCell there is no way to break the cycle. At some point we will
have to implement some form of cycle collection; whatever library we
choose for that purpose is likely to provide an "immutable strong
reference cycle" primitive similar to Rc::new_cyclic(), and we
should be able to simply drop it in.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I34bb5821628eb97e426bdb880b02e2097402adb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7097
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds a comment noting that StackIdx is an offset relative to
the base of the current CallFrame, whereas UpvalueIdx is an absolute
index into the upvalues array.
It also removes the confusing mention of StackIdx in the descriptive
comment for LocalIdx. They index into totally different structures;
one exists at runtime and the other exists at compile time.
Change-Id: Ib932b1b0679734c15001e8c5c95a08293fa016b4
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7017
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds a function NixList::force_elements() which forces each
element of the list shallowly. This behavior is needed for
`builtins.replaceStrings`, and probably a few other builtins as
well.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I3f0681acbbfe50e781b5f07b6a441647f5e6f8da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7094
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Unfortunately we have to mangle test case filenames into rust-valid
symbols, since test-generator doesn't use `r#"..."` (deliberately?).
This means that when a test fails, there's nothing on the console
you can copy-and-paste in order to view/edit the code of the failing
test case.
This commit (partially) fixes it by including the unmangled name in
the panic!() string. However failures due to panic!()s inside the
vm (including deliberate panics due to panic!()-debugging) still
won't display an unmangled filename.
Maybe we should reconsider the use of test-generator?
Change-Id: I2208a859ffab1264f17f48fd303ff5e19675967e
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7092
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Rather than implementing all of the interesting semantics of value
comparison with a macro bound to the VM, implement the bulk of the logic
with a method on Value itself that returns an Ordering, and then use the
macro to implement the comparison against that Ordering. This has no
functional change, but paves the way to implementing lexicographic
comparison of list values, which is supported in the latest version of
upstream nix.
Change-Id: I8af1a020b41577021af5939f5edc160c407d4a9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7069
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
I played around a little bit with doing this in-place, but ended up
going with this perhaps slightly clone-heavy approach for now because
ideally most clones on Value are cheap - but later we should benchmark
alternate approaches that get to reuse allocations better if necessary
or possible.
Change-Id: If998eb2056cedefdf2fb480b0568ac8329ccfc44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7068
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The last bump in langVersion (5->6) in C++ Nix was due to making lists
comparable (commit `09471d2680292af48b2788108de56a8da755d661`), which
we support in Tvix with cl/7070.
Change-Id: Id3beed5150b8fb6e0a46a4d1b7e3942022a65346
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7074
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This commit implements builtins.currentSystem, by capturing the
cargo environment variable `TARGET` and exposing it to rustc as
`TVIX_CURRENT_SYSTEM` so it can be inserted into the source code
using `env!()`.
The resulting value needs to be massaged a bit, since it is an "LLVM
triple". The current code should work for all the platforms for
which cppnix works (thanks qyliss for generating the list!). It
does *not* reject all of the triples that cppnix's configure.ac
rejects -- it is much more forgiving. We can tighten this up in a
future commit.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I947f504b2af5a7fee8cf0cb301421d2fc9174ce1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6986
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Since we push arguments onto a stack when calling multi-argument
functions, we actually were ending up calling `call_with` with the
arguments in the *reverse order* - we patched around this by passing the
arguments in the reverse order for `foldl'`, but it makes more sense to
have them just be the order that the function would be called with in
user surface code instead.
Change-Id: Ifddb98f46970ac89872383709c3ce758dc965c65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7067
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
When compiling a lambda, take the name of the outer slot (if
available) and store it as the name on the lambda.
These names are then shown in the observer, and nowhere else (so far).
It is of course common for these things to thread through many
different context levels (e.g. `f = a: b: c: ...`), in this setup only
the outermost closure or thunk gains the name, but it's better than
nothing.
Change-Id: I681ba74e624f2b9e7a147144a27acf364fe6ccc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7065
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
There are some rare scope cases with deferred access where this
doesn't behave correctly otherwise.
Change-Id: I6c774f5e62c1cb50b598026c54727017a52cd22d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7064
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The warning needs to consider whether it is occuring inside of a
thunk, i.e. the dynamic ancestry chain of lambda contexts must be
inspected and not just the current scope.
Change-Id: I5cf5482d67a8bbb9f03b0ecee7a62f58754f8e59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7063
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
This check is now actually simply equivalent to checking whether the
target has been initialised or not.
Change-Id: I30660d11073ba313358f3a64234a90ed81abf74c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7062
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
These are builtins which can be basically implemented as the identity
function from the perspective of pure evaluation, and which help us
get closer to evaluating nixpkgs.
For now, builtins added here will be "usable" and just emit a warning
about not being implemented yet.
Change-Id: I0fce94677f01c98c0392aeefb7ab353c7dc7ec82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7060
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Checking the computed depth and stack slot against the computed depth
and stack slot is equivalent to just checking the indices into the
locals vector against each other (i.e. "is the slot we're compiling
into the slot we're accessing?")
Change-Id: Ie85a68df073e3b2e3d9aba7fe8634c48eada81fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7059
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
When capturing an upvalue, return a detailed TvixBug error that
contains metadata about what exactly was missing.
This particular thing helps with debugging some scope issues we still
seem to have.
Change-Id: I1089a1df4b3bbc63411a4907c3641a5dc3fad984
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7058
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Using the same method as in Thunk::deep_force, detect cycles when
printing values by maintaining a set of already seen thunks.
With this, display of infinite values matches that of Nix:
> nix-instantiate --eval --strict -E 'let as = { x = 123; y = as; }; in as'
{ x = 123; y = { x = 123; y = <CYCLE>; }; }
> tvix-eval -E 'let as = { x = 123; y = as; }; in as'
=> { x = 123; y = { x = 123; y = <CYCLE>; }; } :: set
Change-Id: I007b918d5131d82c28884e46e46ff365ef691aa8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7056
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This can be raised in cases where some panics lurk (e.g. indexed
accesses).
Change-Id: I93f4d60568277e9c5635aa52f378645626d68c5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7057
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Note that this test (ironically) fails if the observer is used (e.g.
with --trace-runtime), but that's a separate issue.
Change-Id: I952eaeac8b5a7acce9c66cd4744ec570280748e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7055
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This is done via a new `deepForce` function on Value. Since values can
be cyclical (for example, see the test-case), we need to do some extra
work to avoid RefCell borrow errors if we ever hit a graph cycle:
While deep-forcing values, we keep a set of thunks that we have
already seen and avoid doing any work on the same thunk twice. The set
is encapsulated in a separate type to stop potentially invalid
pointers from leaking out.
Finally, since deep_force is conceptually similar to
`VM::force_for_output` (but more suited to usage in eval since it
doesn't clone the values) this removes the latter, replacing it with
the former.
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: Iefddefcf09fae3b6a4d161a5873febcff54b9157
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7000
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
When forcing thunks in `force_with_output`, the call stack of the VM
is actually empty (as the calls are synthetic and no longer part of
the evaluation of the top-level expression).
This means that Tvix crashed when constructing error spans for the
`fallible` macro, as the assumption of there being an enclosing span
was violated.
To work around this, we instead pass the span for the whole top-level
expression to force_for_output and set this as the span for the
enclosing error chain. Existing output logic will already avoid
printing the entire expression as an error span.
This fixes b/213.
Change-Id: I93978e0deaf5bcb0f47a6fa95b3f5bebef5bad4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7052
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Failures to resolve a nix search path lookup in angle brackets can be
caught using tryEval (if it reaches the runtime). Resolving relative
paths (either to the current directory or the current user's home) can
never be caught, even if they happen inside a thunk at runtime (which is
currently the case for home-relative paths).
Change-Id: I7f73221df66d82a381dd4063358906257826995a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7025
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Adding these to the C++ Nix CI ensures that it is possible for the
tests in that directory to pass at all. Mainly this would catch
situations like fixed in a previous CL when moving the tests around
would break them so that they wouldn't even pass in C++ Nix.
For this to work, we need to track skips by basename to be
directory-independent (assuming that every skipped test name is unique
is hopefully okay).
Change-Id: If6cb63ebdef0fc19b082b6a04e79ada2e47c658e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7048
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
cl/7036 moved these paths around, but neglected to update the relative
paths they contain. Without these updated, they will never start
passing!
Change-Id: Ib16468611af59729883e501be8486f43d850fd58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7046
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
eval-okay-eq.nix is actually an ancient test case that used the ATerm
syntax for the test result. It was disabled for a while, since the
behavior got reverted for a bit, but works on any reasonably modern
Nix implementation. This change matches my [C++ Nix PR].
[C++ Nix PR]: https://github.com/NixOS/nix/pull/7196
Change-Id: I602fd7c83a0bc104ab502c8b6a74e4591272be1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7045
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The language test suite actually doesn't require flakes and the
new features are mostly sensible (added builtins) as well as some
tests for regressions the C++ implementation experienced.
The path interpolation test is not included in this update because there
is no way to construct an location-independent .exp file for it (the C++
repo also doesn't have one). We may still want to implement that feature
eventually (in case rnix adds support for it).
The C++ Nix revision used is ac0fb38e8a5a25a84fa17704bd31b453211263eb.
Change-Id: I75f1e780ddeeee6f6b1f28cf3c66c288dca2c20c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7043
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Since cl/7036 we have a mechanism for dealing with the nix_tests we do
not pass yet.
Change-Id: I246c52963ae7f2500253f4035a77d7006dd35307
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7049
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reimplement the test discovery of the lang tests script in Nix which
allows for a more flexible skipping logic that can e.g. react to the C++
Nix version used. This allows us to run the test suite against both
C++ Nix 2.3 and the latest C++ Nix version 2.11. The latter is mainly
useful, so we can implement newer Nix features and still verify them
against the C++ implementation.
Change-Id: I30c802844133b86b5e49f5e4f4fefacdb6215e0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7042
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
It is helpful to be able to use the test suite as a regression test:
make a change to the compiler/vm, re-run the tests, and if there are
any failures you know it's your fault.
Right now we can't do that, because the expected-to-fail tests are
mixed in with the expected-to-pass tests. So we can't use them as a
regression test.
Change-Id: Ied606882b9835a7effd7e75bfcf3e5f827e0a2c8
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7036
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit deduplicates the Thunk-like functionality from Closure
and unifies it with Thunk.
Specifically, we now have one and only one way of breaking reference
cycles in the Value-graph: Thunk. No other variant contains a
RefCell. This should make it easier to reason about the behavior of
the VM. InnerClosure and UpvaluesCarrier are no longer necessary.
This refactoring allowed an improvement in code generation:
`Rc<RefCell<>>`s are now created only for closures which do not have
self-references or deferred upvalues, instead of for all closures.
OpClosure has been split into two separate opcodes:
- OpClosure creates non-recursive closures with no deferred
upvalues. The VM will not create an `Rc<RefCell<>>` when executing
this instruction.
- OpThunkClosure is used for closures with self-references or
deferred upvalues. The VM will create a Thunk when executing this
opcode, but the Thunk will start out already in the
`ThunkRepr::Evaluated` state, rather than in the
`ThunkRepr::Suspeneded` state.
To avoid confusion, OpThunk has been renamed OpThunkSuspended.
Thanks to @sterni for suggesting that all this could be done without
adding an additional variant to ThunkRepr. This does however mean
that there will be mutating accesses to `ThunkRepr::Evaluated`,
which was not previously the case. The field `is_finalised:bool`
has been added to `Closure` to ensure that these mutating accesses
are performed only on finalised Closures. Both the check and the
field are present only if `#[cfg(debug_assertions)]`.
Change-Id: I04131501029772f30e28da8281d864427685097f
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7019
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
In `a++b`, the previous implementation would move `b` (i.e. memcpy
its elements) twice. Let's do that only once.
We sure do call NixList.clone() a whole lot. At some point in the
future we probably want to do a SmolStr-type split for NixList into
a two-variant enum where one side is an Rc<Vec<Value>> for lists
longer than a certain length.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I32154d18785a1f663454a8b9d4afd3e78bffdf9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7040
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This commit adds a comment for the benefit of non-Lua-users
explaining what an upvalue is. It also adds a comment explaining
what `with_stack` does, including a brief reminder of nix's
slightly-unusual-but-well-justified handling of this construct.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: If6d0e133292451af906e1c728e34010f99be8c7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7007
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Without this change it was possible to cause situations (see the new
test) in which a `with`-namespace was forced prematurely.
Change-Id: I879ea7763b43edc693feace2c73c890d426fafd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7031
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Now that we're tracking formals on Lambda this ends up being quite easy;
we just pull them off of the Lambda for the argument closure and use
them to construct the result attribute set.
Change-Id: I811cb61ec34c6bef123a4043000b18c0e4ea0125
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7003
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Validate "closed formals" (formal parameters without an ellipsis) via a
new ValidateClosedFormals op, which checks the arguments (in an attr set
at the top of the stack) against the formal parameters on the Lambda in
the current frame, and returns a new UnexpectedArgument error (including
the span of the formals themselves!!) if any arguments aren't allowed
Change-Id: Idcc47a59167a83be1832a6229f137d84e426c56c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7002
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
In preparation for both implementing the `functionArgs` builtin and
adding support for validating closed formals, record information about
the formal arguments to a function *on the Lambda itself*. This may seem
a little odd for the purposes of just closed formal checking, but is
something we have to have anyway for builtins.functionArgs so I figured
I'd do it this way to kill both birds with one stone.
Change-Id: Ie3770a607bf352a1eb395c79ca29bb25d5978cd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Since we already have infra for forcing arguments to builtins, this ends
up being almost *too* simple - we just return the second argument!
Change-Id: I070d3d0b551c4dcdac095f67b31e22e0de90cbd7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6999
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This resolves a TODO.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: If4d2124648ac88094e547e1ad7f1b446feb26182
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7010
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The previous serialisation format kind of lost the information about
what AST node we're dealing with (e.g. `1234` would serialise to an
AST with a literal `1234`).
That's great for pretty-printing the _code_, but we explicitly want to
serialise how rnix-parser parses something.
To that end, literals are now instead serialised into a structure like
all the other ones (`kind: literal` and appropriate value fields).
Change-Id: I586c95d7db41820b8ec43565ba4016ed3834d1b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7030
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: j4m3s <james.landrein@gmail.com>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Home relative paths depend on the environment to be resolved. We have
elected to do everything that depends on the environment, e.g. resolving
SPATH expressions using NIX_PATH, at runtime, so tvix evaluation would
continue to behave correctly even if we separated the compilation and
execution phases more, e.g. via serializing bytecode. Then the value of
HOME, NIX_PATH etc. could reasonably change in the time until execution,
yielding wrong results if the resolution results were cached in the
bytecode.
We also take the opportunity to fix the broken path concatenation
previously found in the compiler, fixing b/205.
Another thing we could consider is emitting a warning for home relative
path literals, as they are by nature relatively fragile.
One sideeffect of this change is that home path resolution errors
become catchable which is not the case in C++ Nix. This will need to be
fixed up in a subsequent change.
Change-Id: I30bd69b575667c49170a9fdea23a020565d0f9ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7024
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
To assert that OpFindFile is only emitted for specially compiled SPATH
expressions, as well as make sure it doesn't accidentally operate on
“ordinary values”, introduce an UnresolvedPath internal value. If
OpFindFile sees a non-UnresolvedPath value, it'll crash.
Note that this change is not done purely for OpFindFile: We may want to
compile SPATH expressions as function calls to __findFile (like C++ Nix
does) in the future, so the UnresolvedPath value would definitely need
to be an ordinary string again then. Rather, this change is done in
preparation for resolving home dir relative paths at runtime (since they
depend on the environment) for which we'll need a similar mechanism to
OpFindFile.
Change-Id: I6acf287f35197cd9e13377079f972b9d36e5b22e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7023
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This implements serde::Serialize for the rnix AST through a wrapper
type, and exposes a function for serialising the AST into
a (pretty-printed JSON) string representation.
This can be used to debug issues with the AST, and to display an AST
reprsentation in tools like tvixbolt.
Serialize is implemented manually because we don't own any of the
structs and the way to traverse them is not easily derived
automatically, and this is quite verbose. We might be able to condense
it a little bit, but at the same time it's also fairly straightforward.
Change-Id: I922df43cfc25636f3c8baee7944c75ade516055c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6943
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Lambda has a quite large and variable-sized runtime representation,
unlike Rc<Lambda>. It would be easy to accidentally call clone() on
this and create input-dependent performance regressions.
Nothing in the codebase is currently using Lambda.clone(). Let's
remove the derived instance. If it's really needed it is very easy
to add it back in, but whoever does that will have to look at the
struct they're adding Clone to, which will hopefully prompt them to
think about whether or not that's really what they want to do.
Change-Id: I7806a741862ab4402229839ce3f4ea75aafd6d12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7029
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I believe this variant is left over from a previous implementation.
If not, please let me know.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I02a3bf2f63794d09e96a5a92a034c0ad3d1ff221
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7027
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Implement an *initial* version of builtins.match, using the rust `regex`
crate for regular expressions. The rust regex crate definitely has
different semantics than nix's regular expressions - but we'd like to
see how far we can get before the incompatibility starts to matter.
This consciously leaves out any sort of memo for compiled regular
expressions (which upstream nix also has) for the sake of expediency -
in the future we should implement that so we don't have to compile the
same regular expression multiple times.
Change-Id: I5b718635831ec83397940e417a9047c4342b6fa1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6989
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Using `serde_json` for parsing JSON here, plus an `impl FromJSON for
Value`. The latter is primarily to stay "dependency light" for now -
likely going with an actual serde `Deserialize` impl in the future is
going to be way better as it allows saving significantly on intermediary
allocations.
Change-Id: I152a0448ff7c87cf7ebaac927c38912b99de1c18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6920
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Working on https://github.com/NixOS/nix/pull/7158, I discovered that C++
Nix actually is strict in the accumulator, just not in the first value.
This seems due to the fact that in the C++ evaluator, function calls
don't seem to be thunked unconditionally and foldl' just elects not to
wrap it in a thunk (don't quote me on this summary, even though it seems
to line up with the code for primop_foldlStrict and testable behavior).
It doesn't seem worth it to risk breaking the odd Nix expression just to
be strict in one more value per invocation of foldl' (i.e. the initial
accumulator value `nul`), so let's match the existing C++ Nix behavior
here.
Change-Id: If59e62271a90d97cb440f0ca72a58ec7840d1690
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7022
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This commit causes the test eval-okay-builtins.nix to pass.
It also adds tests/tvix_tests/eval-okay-dirof.nix which has better
coverage than the nix tests for this builtin.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I71d96b48680696fd6e4fea3a9861742b35cfaa66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6987
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit implements builtins.toPath. Like OP_ADD, it currently
does not handle string contexts.
This commit allows the
tests::nix_eval_okay_src_tests_nix_tests_eval_okay_pathexists_nix
test to pass.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Iadd4f7605f8f297adbd0dba187b8481c21370b6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6996
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Maybe I misunderstood this part of the code, but the use of `unsafe`
appears unnecessary here? In any event it is the one and only
`unsafe` in the codebase.
Hopefully getting to "no `unsafe` anywhere" is worth the extra
never-taken branch caused by unwrap() instead of unwrap_unchecked()?
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I33fbd5aad9d8307ea82c24b6220412783e1973c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7011
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The case branch in vm.rs for OpResolveWithOrUpvalue is
unreachable/deadcode.
I believe this opcode is unnecessary, since it should always be
statically detectable (at parse-time) whether a reference is to an
upvalue (i.e. enclosing binding); otherwise, and only then, is
with-resolution applicable.
Perhaps I've misunderstood how with-resolution works. If so, please
explain it to me and -1/-2 this CL.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I4a90b9eb6cb3396df92a6a943d42ecc301871ba0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7009
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This makes it easier to compare currently implemented ones with the full
list.
Change-Id: Ibaffd99d05afa15fc9ab644fd101afa24fc7a1b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7008
Tested-by: BuildkiteCI
Autosubmit: j4m3s <james.landrein@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
When I first read the `README.md` I didn't know what this `mg build`
was. Some grepping around turned up `/tools/magrathea`. Apparently
this thing "helps me build planets". Awesome! Let's tell other
people where to find it. It seems that this tool is (currently)
specific to depot, so people who arrive at depot because of tvix
probably won't know where to find `mg`.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Icb8c51087fd41b696794516abcfee24a4b3b4a14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6948
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
This commit implements builtins.baseNameOf and adds a test case
eval-okay-basenameof.nix to the test suite.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib8bbafba2ac9ca0e1d3dc5e844167f94890d9fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6997
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
builtins.parseDrvName should not coerce its argument to a string.
This commit fixes that oversight in my previous commit, and adds an
xfail test to cover this condition.
Thanks to @sterni for noticing this.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I76bc78f1a82e1e08fe5c787c563a221d55de2639
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6991
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This commit adds support for running the "expected failure" tests in
both the nix and tvix test suites.
I have disabled the eval-fail-blackhole.nix test because it gets
stuck running forever.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Iba75ce6c8f2becab3c834fcfdd9f4fdc5a4bdb9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6990
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
cppnix has merged #7149, so let's update our copy of their tests:
https://github.com/NixOS/nix/pull/7149
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I0be3bf9da937abd24102e1997daa2087ecfafd98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6992
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This path normalisation business causes runtime panics on WebAssembly
because those operations are unsupported.
Maybe this shouldn't be happening in the compiler anyways, not sure,
but for now this commit adds a workaround based on the target to
disable the normalisation if we're compiling for wasm.
Change-Id: I908a84fbdffc3401f8d443e2c73ec673e9f397ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7004
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Cppnix immediately absolutizes pathnames at parse time; if you write
`./foo`, it is immediately converted to `$(pwd)/foo` and manipulated
as an absolute path at all times.
To avoid having to introduce filesystem access operations in the
implementation of otherwise-pure builtins, let's guarantee that the
`root_dir` of the VM is always an absolute path.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I7cbbae2cba4b2716ff3f5ff7c9ce0ad529358c8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6995
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Right now we're pretending that the Rust library path_clean does the
same thing that cppnix's canonPath() does. This is not true. It's
close enough for the test suite, but may come back to bite us.
Let's create our own canon_path() function and call that in all the
places where we intend to match the behavior of cppnix's
canonPath(). That way when we fix this we can fix it once, in one
place.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ia6f9577f62f49ef352ff9cfa5efdf37c32d31b11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6993
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
On Debian machines (and most FHS distros) /bin is now a symlink to
/usr/bin, which causes this test to fail. Let's use /etc as a
root-relative test case, since it is less likely to be a symlink.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I1c94de0d2a242394182442fe1c895dc17eb04a4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6994
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
The nix_tests test suite produces lots of warnings. We can't fix
these, since they are kept in sync with upstream, so there's little
point in cluttering up the console with them every time the tests
are run.
Let's add a clap flag "warnings" and TVIX_WARNINGS environment
variable. The default is "true". The test runner overrides this
default and mutes the warnings.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I4b065f96fe15838afcca6970491a54e248ae4df7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6985
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This commit passes nix_eval_okay_src_tests_nix_tests_eval_okay_versions_nix.
See also: https://github.com/NixOS/nix/pull/7149
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I24605c2a0cd0da434f37f6c518f20693bfa1b799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6913
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Since NixString is the Rust type for nix strings, people might
mistake NixPath for the Rust type of nix paths, which it is not.
Let's call it NixSearchPath instead.
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib2ea155c4b27cb90d6180a04ea7b951d86607373
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6927
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
I broke the build.
I didn't understand that before hitting "submit" you need to re-test
your changes on latest HEAD (and that CI doesn't do that for you); I
failed to re-test cl/6912 following the merge of cl/6907.
This commit fixes the build by removing the overlapping instances.
Change-Id: I2a720d2c60cc7103b350f78102a8998f93bac828
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6965
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
When investigating discrepancies between foldl' in tvix and C++ Nix,
I discovered that C++ Nix's foldl' doesn't seem to be strict at all.
Since this seemed wrong, I looked into Haskell's foldl' implementation
which doesn't force the list elements (`val` in our code), but the
accumulation value (`res` in our code). You can look at the code here:
https://hackage.haskell.org/package/base-4.17.0.0/docs/src/GHC.List.html#foldl%27
This actually makes a lot of sense: If `res` is not forced after each
application of `op`, we'll end up thunks nested as deeply as the list is
long, potentially taking up a lot of space. This can be limited by
forcing the `res` thunk before applying `op` again (and creating a new
thunk).
I've also PR-ed an equivalent change for C++ Nix at
https://github.com/NixOS/nix/pull/7158. Since this is not merged nor
backported to our Nix 2.3 fork, I've not copied the eval fail test yet,
since it wouldn't when checking our tests against C++ Nix in depot.
Change-Id: I34edf6fc3031fc1485c3e714f2280b4fba8f004b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6947
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This *maybe* should do something to check that we don't pass both a file
and an expr, but for now this is useful enough to cut corners (plus
we're probably due for a CLI revamp eventually anyway).
Change-Id: Id44357074150b336b6215ba596cc52d01d037dbd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6941
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is currently implemented with a simple println inline, but in the
future we could hook into this via something pluggable on the VM.
Change-Id: Idd9cc3b34aa13d6ebc64c02aade81ecdf439656a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6938
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Previously, the VM assumed that if an error was returned from `run()`,
the evaluation was "finished" and the state of the VM didn't matter.
This used to be a reasonable assumption, but now that we've got
`tryEval` around we need to actually make sure that we clean up after
ourselves if we're about to return an error. Specifically, if the *last*
instruction in an evaluation frame returns an error, we previously
wouldn't pop that evaluation frame, which could cause all sorts of
bizarre errors based on what happened to be in the stack at the time.
This commit splits out a `run_op` method from `VM::run`, and uses that
to check the evaluation frame return condition even if the op we're
running is about to return an error, and pop the evaluation frame if
we're at the last instruction.
Change-Id: Ib40649d8915ee1571153cb71e3d76492542fc3d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6940
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Currently, the span on *all* thunk force errors is the span at which the
thunk is forced, which for recursive thunk forcing ends up just being
the same span over and over again. This changes the span on thunk force
errors to be the span at which point the thunk is *created*, which is a
bit more helpful (though the printing atm is a little... crowded). To
make this work, we have to thread through the span at which a thunk is
created into a field on the thunk itself.
Change-Id: I81474810a763046e2eb3a8f07acf7d8ec708824a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6932
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Previously the various call functions either returned `EvalResult<()>`
or `EvalResult<Value>`, which was confusing.
Now only vm::call_with returns a Value directly, and other parts of
the API just leave the stack top in the post-call state.
This makes it easier to reason about what's going on in non-tail-call
cases (which are making a comeback).
Change-Id: I264ffc683a11aca72dd06e2220a5ff6e7c5fc2b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6936
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Instead of just printing the number of errors (useless!) actually emit
separate diagnostics for each nested error.
Change-Id: I97b53c3276c906af5def89077b5b6ba6ec108b37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6933
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Despite this not being documented, `tryEval` is empirically able to
catch errors caused by a <...> path not resolving (and nixpkgs depends
on this).
Change-Id: Ia3b78a2d9d2d0c603aba829518b351102dc55396
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6926
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Since the body of an `if` expr can refer to deferred upvalues, it needs
to be thunked so when we actually compile those deferred upvalues we
have something for the finalize op to point at. Without this all sorts
of weird things can happen due to the finalize op being run in the wrong
lambda context, up to and including a panic.
Change-Id: I040d5e1a7232fd841cfa4953539898fa49cbbb83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6929
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
With asserts compiled using conditional jumps, this ends up being quite
straightforward - the only real tricky bit is that we have to know
whether an error can or can't be handled.
Change-Id: I75617da73b7a9c5cdd888c0e26ae81d2c5c0d714
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6924
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This commit implements (lazy) resolution of `<...>` paths via either the
NIX_PATH environment variable, or the -I command-line flag - both
handled via EvalOptions. As a result, EvalOptions can no longer derive
Copy, meaning we have to clone it at each line of the repl - this is
probably not a huge deal as repl performance is not exactly an inner
loop and we're not cloning very much.
Internally, this works by creating a thunk which pushes a constant
containing the string inside the brackets to the stack, then a new
opcode to resolve that path via the `NixPath`. To get that opcode to
work, we now have to pass in the NixPath when constructing the VM.
This (intentionally) leaves out proper implementation of path resolution
via `findFile` (cppnix just calls whatever identifier called findFile is
in scope!!!) as that's widely considered a bit of a misfeature, but if
we do decide to implement that down the road it likely wouldn't be more
than a few extra ops within the thunk introduced here.
Change-Id: Ibc979b7e425b65cbe88599940520239a4a10cee2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6918
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Add a simple struct implementing both the string parsing and path
resolution rules of Nix's `NIX_PATH` environment variable, for use in
resolving `<...>`-style paths
Change-Id: Ife75f39aa5c12928278d81fe428fbadc98bac5cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6917
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Implement adding paths and strings via OpAdd. Since the nix rules are
quite obscure, I'm electing to test this one with an oracle test to
avoid the danger of getting the actual asserted result wrong.
Change-Id: Icdcca3690ca2e8459e386c1f29cc48eaaa39e9a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6914
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
In order to behave nicely with tryEval, asserts need to leave the
instruction pointer in a reasonable place even if they fail - whereas
with the previous implementation catching a failed assert would still
end up running the op for the *body* of the assert. With this change, we
compile asserts much more like an `if` expression with conditional jumps
rather than having an OpAssert op.
Change-Id: I1b266c3be90185c84000da6b1995ac3e6fd5471b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6925
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The process of calling a function from a builtin, especially if it's got
more than 1 arrgument, is reasonably involved and easy to get wrong due
to having to interact directly with the stack - instead of having that
done entirely manually in builtins, this wraps it up in a new
`call_with` function which handles pushing arguments onto the stack and
recursively calling the (partially applied) function.
Change-Id: I14700c639a0deca53b9a060f6d70dbc7762e9007
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6910
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Implement the listToAttrs builtin, which constructs an attribute set
from a list of attribute sets with keys name and value.
This is tested using an adaptation of the nix `eval-ok-listtoattrs.nix`,
with the utilities from `lib.nix` inlined.
Change-Id: Ib5bf743466dda9722c2c1e00797df4b58448cf0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6894
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is generally more idiomatic (over just delegating to Debug), and
also allows us to avoid intermediate allocations if we ever end up
using error messages as part of larger strings (because we don't have to
allocate a full String for the return value).
Change-Id: I67e48b44570c72761ed0fcaded9ae4bf3fcbaacf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6896
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Similar to what we do for import, push on a `default.nix` to the path
that the top-level is invoked with (if any) if it's a directory.
Change-Id: I281bd44e3c8803b6765c886ae5fd08f549e2e563
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6895
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
When contrasting the compilation of the desugared version to the
"sugared" version, this was the noticeable difference.
This fixes b/203.
Change-Id: Iae02ffc56e06de1de091b84cdc59d8fe83a17d69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6898
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This requires actually passing the source directory into `interpret` in
the eval tests, but otherwise this is fairly straightforward - if we're
trying to import a directory, just push `default.nix` onto it and import
that instead.
Change-Id: I0b7d4234f81977e78d14dfa651bf0cf9721017e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6893
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Factor out the construction of Value::Attrs (including the Rc) into a
new `attrs` constructor function, to abstract away the presence of the
Rc itself.
Change-Id: I42fd4c3841e1db368db999ddd651277ff995f025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6892
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Thunks correctly force when comparing for equality against other thunks,
but weren't being forced correctly when comparing against non-thunk
values, in either direction.
Change-Id: Ia03702895ec4d70aed3445c1b0a9a7a641d1a300
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6897
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Adds secondary spans for errors that occur deeply nested within a
thunk.
This is pretty raw right now, there's technically nothing stopping one
of these error chains from being a hundred thunks deep into code,
producing unmanageable error output. We should trim these down
according to some heuristics (e.g. when crossing file boundaries, o r
just - for starters - beginning and end).
Change-Id: Ia73892512737850b6fa3e07cabc37fa9c534c4d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6872
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This change is quite verbose, so a little bit of explaining:
1. To correctly format parse errors, errors must be able to return
more than one annotated span (the parser returns a list of errors
for each span).
To accomplish this, the structure of how the `Diagnostic` struct
which formats an error is constructed has changed to delegate the
creation of the `SpanLabel` vector to the kind of error.
2. The rnix structures don't have human-readable output formats by
default, so some verbose methods for formatting them in
human-readable ways have been added in the errors module. We might
want to move these out into a submodule.
3. In many cases, the errors returned by rnix are a bit strange - so
while we format them with all information that is easily available
they may look weird or not necessarily help users. Consider this CL
only a first step in the right direction.
Change-Id: Ie7dd74751af9e7ecb35d751f8b087aae5ae6e2e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6871
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This logic was duplicated between the two rnix types before, but more
crucially - it is also needed for correctly displaying the text ranges
contained in syntax errors.
Change-Id: Ifc6a521de1594d6ced9cba6274f1931df99b6634
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6870
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This is also useful for error-handling related logic, outside of just
the compiler module.
Change-Id: I5c386e2b4c31cda0a0209b31136ca07f00e39e45
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6869
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This enables the use of string paths (and, in the future,
derivations), as long as their string values represent an absolute
path.
Change-Id: I4b198efeb70415ed52f58bd1da6fa79a24dad14c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6866
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This lets the VM emit warnings when it encounters situations that
should only be warned about at runtime.
For starters, this is used to pass through compilation warnings that
come up when `import` is used.
Change-Id: I0c4bc8c534d699999887c430d93629fadfa662c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6868
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Adding `import` to builtins causes causes a bootstrap cycle because
the `import` builtin needs to be initialised with the set of globals
before being inserted into the globals, which also must contain
itself.
To break out of the cycle this hack wraps the builtins passed to the
compiler in an `Rc` (probably sensible anyways, as they will end up
getting cloned a bunch), containing a RefCell which gives us mutable
access to the builtins.
This opens up a potentially dangerous footgun in which we could mutate
the builtins at runtime leading to different compiler invocations
seeing different builtins, so it'd be nice to have some kind of
"finalised" status for them or some such, but I'm not sure how to
represent that atm.
Change-Id: I25f8d4d2a7e8472d401c8ba2f4bbf9d86ab2abcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6867
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This adds an initial working version of builtins.import which
encapsulates the entire functionality of `import` within the builtin
itself, without requiring any changes in the compiler or VM.
The key insight that enables this is that we can simply return a Thunk
from `import` that is constructed from the output of running the
compiler and - ta-da! - no other component needs to know about it.
A couple of notes:
* builtins.import needs to capture variables like the SourceCode
structure. This means it can not currently be constructed the same
way as other builtins and has special handling, which leaks out to
`eval.rs`. I have postponed dealing with that until we have this
working a bit more.
* the `globals` are not yet passed through
* the error representation for the new variants is absolutely not done
yet, we probably want to switch to something that supports
cause-chaining now (like miette)
* there is no mechanism for emitting warnings at runtime; we need to
add that
Change-Id: I3117a7ae3ff2432bf44f5ff05ad35f47faca31d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6857
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
For some upcoming builtins (notably, import) we need to capture
arguments in the builtin's implementation.
To allow this, we can no longer use function pointers for builtins,
but must use a reference-counted closure object instead.
Unfortunately this adds an extra pointer operation to every builtin
call. We should benchmark this later against having a split builtin
representation.
Change-Id: I109d98d0e25998870542f47573eb1ec2e546f2a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6856
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
nixpkgs has hardcoded references to Nix versions, we need to provide
it with something that looks like a Nix version while actually being a
Tvix version.
For now, we do this by stealing a trick out of the browser book and
constructing a version that looks like a Nix version to Nix, but like
a Tvix version to people who know what they are looking for.
Nevermind that we don't actually have any kind of versioning for
Tvix (yet?), other than depot revisions.
Change-Id: I7ce8079dd8164a2079891d38e707f09a45f0bbc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6858
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This type hides away the lower-level handling of most codemap data
structures, especially to library consumers (see corresponding changes
in tvixbolt).
This will help with implement `import` by giving us central control
over how the codemap works.
Change-Id: Ifcea36776879725871b30c518aeb96ab5fda035a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6855
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
There's basically nothing that needs *ownership* of an AST
node (which is just a little box full of references to other things
anyways), so we can thread this through as references all the way.
Change-Id: I35a1348a50c0e8e07d51dfc18847829379166fbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6853
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
There are actually two different types of observers, the ones that
observe the compiler (and emitted chunks from different kinds of
expressions), and the ones that trace runtime execution.
Use of the NoOpObserver is unchanged, it simply implements both
traits.
Change-Id: I4277b82674c259ec55238a0de3bb1cdf5e21a258
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6852
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Until we can display a chained representatino of errors in thunks, it
is most useful to forward the error code from the innermost error to
the user.
Change-Id: I8d67254d52313be40387f080e57966c001e0d51c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6854
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Returns time since epoch in seconds.
This has a slight behaviour difference from Nix, in that we don't pin
the time between REPL entries (Nix pins it for the program lifetime),
but this is probably inconsequential as long as it is pinned during an
evaluation.
Change-Id: I010c02e93097a209d8ad69e278397c7e30e54c86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6846
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Allows impure builtins that have a different shape than a Rust
function pointer; specifically this is required for
builtins.currentTime which does not work in WASM.
Change-Id: I1362d8eeafe770ce4d1c5ebe4d119aeb0abb5c9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6849
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Concatenates (but not flattens) a list of lists.
Change-Id: I692e0b3e7b5a5ff93d5768d3a27849b432ec5747
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6843
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is the same code as before, just moved into a trait impl to gain
access to stuff that needs IntoIterator
Change-Id: Iff9375cd05593dd2681fa85ccc7f4554bf944a02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6842
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This can actually legitimately be emitted by the compiler currently
when compiling formals with default values. See the scope6 test from
the Nix test suite for an example.
We should restructure this slightly to be able to reintroduce a
runtime error here in case something was compiled incorrectly.
Change-Id: Ib81f0f58ae0e850db9fbc459458b7bd0d3ac6f23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6841
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This implements __functor calling in situations where `OpTailCall` is
used, but not yet for `OpCall`.
For some reason I have not yet figured out, this same implementation
does not work in call_value, which means that it also doesn't yet work
in builtins that apply functions.
Change-Id: I378f9065ac53d4c05166a7d0151acb1f55c91579
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6826
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Add a new derivation target to the passthru of tvix.eval that builds the
benchmark binaries, *and* copies them to the outupts of the derivation
via the (somewhat arcane) `copyBinsFilter` jq script arg to naersk. This
is a bit annoying because (as far as I can tell) the derivations
returned by naersk aren't directly overridable, so we have to explicitly
fixpoint the attrs we're passing.
Also, since this is now a separate target to build the benchmarks, we
can remove `--all-targets` from the build of `tvix-eval` itself since
that was only added to build benchmarks in CI, and make
regular (non-benchmark) builds a bit faster.
Change-Id: I136b8526790545e93b1ae666abaefb51cbbee390
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6847
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The idea is that we can keep track of the more unexpected behavior,
behavior that maybe should not be a thing at all and behavior we are not
sure about yet.
Change-Id: I70933f00af1230a7ab9d30e917b61199fe571caf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6803
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This function previously kept a borrow in the form of the
`Thunk::value` result alive while performing arbitrary actions in the
VM, which caused a borrowing error in the test case attached.
The `Ref` value must never be used in cases where control flow is
passed to other parts of the VM.
Change-Id: I41d10aa1882a2166614b670e8ba77aab0e67deca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6825
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This finishes up the implementation of nested keys after the key
insight that the nesting level does not need to be tracked, and
instead the attribute iterator can simply be retained inside the
structures as is (in an advanced state).
With this implementation, when encountering a nested key, the Tvix
compiler will first analyse whether there is already a matching
binding that can be merged (i.e. a binding that is a literal attribute
set), and perform the merge, or otherwise create a new recursive set
of bindings in which the entry is inserted with the path iterator
advanced beyond the first name component.
With this, all the logic simply applies recursively until there are no
more nested bindings (i.e. until all iterators are "empty").
Note that this has one (potentially insignificant) deviation from Nix
currently: If a non-mergable value is supplied (e.g. `a.b = 1; a =
2;`), Tvix will emit a *runtime* error (whereas it is *parse* time in
Nix) as the branch which could statically analyse this is currently
unreachable. There's a TODO for this, so we can fix it up later.
Change-Id: I53df70e09614ff4281a70b80eac7da3beca12da9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6806
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is actually quite useless, as we can just pass
`AstChildren<ast::Attr>` around after partially consuming it.
Change-Id: If0aefa2b53fc801fced1ae0709bff93966bf19f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6804
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
When encountering a nested binding for the first time, cleanly flip
the representation to `Binding::Set` in `Binding::merge` before
proceeding with the actual merge.
This reduces the number of points where we have to deal with the (soon
to be slightly more complex) construction of the nested binding
representation.
Change-Id: Ifd43aac7b59ebd15a72c3ec512386a5bcf26ec13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6802
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This adds the scaffolding required for tracking the nesting level (and
appropriately skipping the correct amount of attrpath entries when
inserting nested sets).
In order for all of this to work correctly, we can no longer track
`AttrpathValue` directly in the entries vector as rnix does not allow
us to construct values of that type - so instead we have to track its
inner components.
Change-Id: Icb18e105586bf6c247c2e66c302cde5609ad9789
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6801
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is a significant step towards correctly implemented nested
bindings. All attribute sets defined within the same binding scope
will now be merged as in Nix, if they use the same key.
Change-Id: I13e056693d5e73192280043c6dd93b47d1306ed6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6800
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This sets up the required logic for finding and merging attribute sets
into nested bindings if they exist. This is absolutely not complete
yet and can, at this commit, probably cause undefined runtime
behaviour if nested attributes are specified.
The basic idea is that a new helper function on the `TrackedBindings`
struct is called with each encountered attribute and determines
whether the new entry can be merged into an existing attribute or not.
Right now the only effect this has in practice is that a new error
becomes available if somebody attempts to cause a merge into an
inherited key.
Change-Id: Id010df3605055eb1ad7fa65241055889dd21bab0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6798
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This needs to move here so that we can reuse compile_bindings for the
nested attribute sets we're about to start constructing.
Change-Id: Ie83f52f7e1d128886e96a1da47792211fa826f21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6796
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This struct will be the key to correctly compiling nested bindings, by
having insertions flow through some logic that will attempt to bind
attribute-set-like things when encountering them.
Change-Id: I8b5b20798de60688f3b6dc4526a460ebb2079f6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6795
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This implementation, which only ever worked for non-recursive
attribute sets, is no longer needed and thus removed here.
We have a new implementation of these nested keys coming up instead.
Change-Id: I0c2875154026a4f5f6e0aa038e465f54444bf721
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6783
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
As of this commit, all three types of bindings scopes are compiled the
same way (i.e. compilation of non-recursive attribute sets has been
switched over to the new code paths).
This sets us up for doing the final implementation of nested attribute
sets.
HOWEVER, this breaks the existing implementation of nested attributes
in non-recursive attribute sets. That implementation is flawed and
unworkable in practice, so we need to do this dance to be able to
implement it correctly.
Change-Id: Iba2545c0d1d6b51f5e1a31a5d005b8d01da546d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6782
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This wires up the new bindings setup logic to be able to thread
through & compile dynamic attributes in recursive attrs.
It seems like we don't actually need to retain the phasing of Nix
exactly, as we can use the phantom mechanism to declare all locals
without making the dynamic ones accessible.
Change-Id: Ic2d43dd8fd97d7ccd56d8c6adf2ff97274cd837a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6781
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Another slice of the salami, but no functionality changes yet (other
than opening a code path that can reach a `todo!()`, but this will be
removed soon).
Change-Id: I56b4ed323f70754ed1ab27964ee3c99cf3bf3292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6780
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The previous way of sanitising dynamic keys is going away as we're
slowly introducing the new nested key logic.
While touching this stuff, I've also changed all the related string
types to SmolStr as that is more sensible for identifiers.
Change-Id: If30c74151508719d646d0e68e7d6f62c36f4d23f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6779
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Removes the `compile_inherit_attrs` logic which was only used for
BindingsKind::Attrs (i.e. non-recursive attrs).
This brings us a step closer to fully merging all the binding logic
into one block that can dispatch based on the kind of bindings (and
thus giving us a good point to introduce the final logic for nested
bindings).
Change-Id: If48d7a9497fc084a5cc03a130c2a7da5e2b8ef0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6776
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This helper is responsible for declaring the bindings in the
compiler's scope tracking structures.
It is almost equivalent to the previous logic, but also accounts for
`BindingsKind::Attrs` - though those code paths do not end up here
yet.
Change-Id: I44f617b99b10f2a7b9675f7b23e2c803a4a93d29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6775
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.
* runCommandNoCC -> runCommand
* gmailieer -> lieer
We also need to work around the fact that home-manager hasn't catched
on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy
Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.
Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
This is responsible for actually setting up `TrackedBinding`s on the
stack, i.e. in some sense "actually compiling" values in bindings.
There is no functionality change to before, i.e. this is a salami
slice.
Change-Id: Idb0312038e004470a7d130c020ae0fe87c55c218
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6774
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Splits the large `compile_inherits` function which
previously *compiled* plain inherits and *declared* namespaced
inherits into `compile_plain_inherits` and
`declare_namespaced_inherits`.
This is supposed to make more sense than before, but is still not
consistently used (notably, non-recursive attribute sets still
duplicate most of this logic).
Another salami slice.
Change-Id: Id97fac1cbd5ee97b24d047e7728655e6b7734153
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6773
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
... but do not use it yet.
This refactoring is pretty complicated, so I'm applying salami-slicing
tactics here.
Change-Id: I66e04ee10548f68bf67dc842f3f14cc279426c22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6772
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
As part of the unification of binding logic between different carriers
of bindings, we need to track which kind of bindings we are dealing
with (attribute set? recursive scope? ...) to correctly emit keys and
declare identifiers in the locals stack.
Right now this changes no functionality as `BindingsKind::Attrs` is
not yet used (only RecAttrs and LetIn, which was previously
represented by the `rec_attrs` boolean).
Change-Id: Id2ac27894079ab584521cb568d75c124f7bf2403
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6771
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This helper will gain the ability to compile both kinds of inherits,
but it is kind of tricky to get right so I am doing it in smaller
steps. Right now there is no change in functionality.
Change-Id: Ie990b88dd90a5e0f9fd79961ee09a6c83f2c872d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6770
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This just describes a binding, and we do need a good name for the kind
of binding*s*, which is going to be introduced soon.
Change-Id: I53900ee52da8a07dae8b918fa6a4cb308e627efb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6768
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Changes the module structure of the compiler to have a module
dedicated to the logic of setting up bindings. This logic is in the
process of being merged between attribute sets and `let`-expressions,
and the structure of the modules makes more sense when ecapsulating
that specifically.
(Other bits of code related to e.g. attribute sets are pretty
straightforward and can just live in the main compiler module).
Change-Id: I9469b73a7034e5b5f3bb211694d97260c4c9ef54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6766
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This was added in preparation for some builtins that don't exist yet
and is producing some noise during compilation.
Change-Id: I51fb0d14c3edf0bd6d9a288d50e44dacf35166c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6769
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is a little ugly because the plain Iterator::filter method can
not be used (it does not support fallible primitives), so we need to
resort to an `Iterator::filter_map` and deal with the wrapping in
Options everywhere.
This prevents use of `?` which introduces the need for some matching,
but it's not *too* bad.
Change-Id: Ie2c3c0c9756c4c627176f64fb4e0054e717c26d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6765
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The previous version had a bug where we assumed that the number of
entries in an attribute set AST node would be equivalent to the number
of entries in the runtime attribute set, but due to inherit nodes
containing a variable number of entries, this did not work out.
Fixes b/199
Change-Id: I6f7f7729f3512b297cf29a2e046302ca28477854
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6749
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Fairly straightforward, only thing of note is that we coerce (weakly) to
string here as well.
Change-Id: I03b427e657e402f1f9eb0f795b689bbf5092aba1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6745
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
The simplest solution seems to be to pass references to arithmetic_op!()
which avoids the moving annoyance we had to deal with in the
builtins (no more popping!). We then use .force() to force the values
and dereference any Thunks (which arithmetic_op! doesn't do for us).
Change-Id: I0eb8ad60e80a0b3ba9d9f411e973ef8bcf136989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6724
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Because `inherit` can take string identifiers, we can access
arbitrarily weird identifiers in scopes using it.
Change-Id: Ic868233221befa160538dd2ffaff1cc7f566585a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6748
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This updates rnix-parser to a version where inherits provide an
iterator over `ast::Attr` instead of `ast::Ident`, which mirrors the
behaviour of Nix (inherits can have (statically known) strings as
their identifiers).
This actually required some fairly significant code reshuffling in the
compiler, as there was an implicit assumption in many places that we
would have an `ast::Ident` node available when dealing with variable
access (which is then explicitly only not true in this case).
Change-Id: I12f1e786c0030c85107b1aa409bd49adb5465546
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6747
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
We delegate to Rust std's Ord trait, so we should at least do a sanity
check here.
Change-Id: I9596068f8ab3e51b79602de0bac79af6d558086e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6723
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Sketch out a new set of "impure" builtins, which supplement the existing
set of "pure" builtins but are gated behind a feature flag, which allows
them to be omitted by crates depending on tvix-eval that only want pure
evaluation, such as tvixbolt.
Change-Id: I2736017b5c9b4776bbba8758e108ec84887abd66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6655
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
This was derived from
else if (c1 == "" && n2) return true; // true implies c1 < n2
However, this has no effect since Word always looses out against Number
anyways and the `pre` rules are also unaffected by this change – since
this only affects comparison of an empty Word part with a Number.
Change-Id: Ia04e42ac726352b688c87674b0fdb355f06edbcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6722
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This asserts the not-quite lexicographical property of the comparison.
Change-Id: Iad68081e4b3a7106513f479643de87065dc47739
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6721
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is necessary because builtins.compareVersions compares versions in
a subtly not-quite-but-still-lexicographical way: `pre` for example can
have an effect if it is post-fixed: `2.3 < 2.3pre`. This is a violation
of the rule that in a lexicographical ordering, the longer string is
considered greater if they are otherwise equal. builtins.compareVersion
is comparing lexicographically though, if you do the following
transformation beforehand:
2.3 --split--> [ "2" "3" ] --append--> [ "2" "3" "" ]
2.3pre --split--> [ "2" "3" "pre" ] --append--> [ "2" "3" "pre" "" ]
Comparing the transformed version is then done lexicographically:
2.3 < 2.3.0pre since [ "2" "3" "" ] < [ "2" "3" "0" "pre" ]
Here, the `pre` rule never comes into effect because no comparison on it
happens, instead we use the longer string rule of a lexicographical
comparison.
In the C++ codebase, the reason for this behavior is that the
iterator-esque construct they use always yields the empty string before
it exposes it has been fully consumed. This is probably intentional to
support the postfixed `pre` which is, for example, used by NixOS
versions (e.g. unstable post 22.05 is 22.11-pre). We replicate this
behavior using the `Chain` iterator in `VersionPartsIter::new_for_cmp`.
Change-Id: I021c69aa27b0b7deb949dffe50ed18b6de3a7b1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6720
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is based on the [relevant code] in C++ Nix. Our version has more
branches because the C++ one only checks if it is less than or not, so
can save handling a few cases. We on the other hand, can avoid calling
the algorithm twice. It'd be nice to implement proptests for this in the
future and to make sure that this weird little algorithm doesn't violate
the Ord laws.
[relevant code]: cd35bbbeef/src/libstore/names.cc (L81-L94)
Change-Id: I46642e6da5eac7c0883cdce860622cdba04cd12b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6719
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This was fairly easy, thanks to the work already done by Thomas Frank.
However, the implementation is suboptimal because we parse number parts
only to convert them back to strings afterwards. I've chosen to tackle
this problem in the future, since having an (inefficient) implementation
of splitVersion will be helpful for debugging the slight discrepancies
between C++ Nix and Tvix in the case of compareVersions.
Change-Id: Id6ed8eeb77663ff650c8c53ea952875b1fb7ea84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6688
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This makes it possible to call a callable value (builtin or
closure/lambda) directly, without unwrapping it first. This is needed
for pretty much all higher-order functions to work correctly.
This is mostly equivalent to the previous code in coerce_to_string for
calling `__toString`, except it expects the argument(s) to already be
placed on the stack.
Note that the span for the `NotCallable` error is not currently
guaranteed to make any sense, will experiment with this.
Change-Id: I821224368d438a28900858b343defc1817e46a0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6717
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Generate smaller recursive values for generated Values, and run fewer
cases for the attrs proptests which are particularly egregious.
Change-Id: Ia35c7c120270feaf045be1deb440c87ebb185c27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6716
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
As we already have a VM passed to the builtins, we can simply execute
the provided closure/lambda in it for each value.
The primary annoyance with this is that we have to clone the upvalues
for each element, but we can try making this cheaper in the
future (it's also a general problem in the VM itself).
Change-Id: I5bcf56d58c509c0eb081e7cf52f6093216451ce4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6714
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Previously we only matched the outer constructor after forcing which
would mean that we would always return `false` if the inspected value
was a thunk, regardless what value would be present inside.
Change-Id: I361ea6e855e23ef8e5b59098a50b9cd59253803f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6692
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Previously, this would almost always crash because list items are
thunked more often nowadays and selecting from a thunk would fail. Also
we no longer pop from args, accessing it by index should avoid an
unnecessary clone here.
Change-Id: I4410c4c2e28cc255a2c7cf2a5322db3d2c556a0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6693
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
And insert the missing newline the C++ Nix test script needs.
Change-Id: I04ddd7268f9caa1414fd23314c281bb7c1e854cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6689
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Instead of arity, we pass a array reference to Builtin::new that
describes how many arguments there are and which of them need to be
forced, eliminating the need to force manually.
Note that this change doesn't fix some of the instances where the the
Builtin doesn't consider that the value could be a Thunk.
Change-Id: Iadb58bb79886c30dc6b09dcf0ffad8abf28036a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6662
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
TL;DR:
- support `builtins.tail`
- define `ErrorKind::TailEmptyList` and canonical error code
- support basic unit tests
Unsure whether or not the error should be a dedicated `ErrorKind`...
Change-Id: Iae90fda1bb21ce7bdb1aaa2aeb2b8c1e6dcb0f05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6545
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Refactor the `force!` macro to a method on `Value` which returns a
smart-pointer-esque type, which simplifies the callsite and eliminates
rightward drift, especially for high-arity builtins.
Change-Id: I97a7837580accfb4bbd03b24f2acdbd38645efa5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6656
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Previously only the first one was guaranteed to be forced, but we need
to do this for all of them.
Fixes b/190
Change-Id: I76b5667dbfb2f3fde3587e7b91d268cbf32aca00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6645
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Document the OpAttrs op, paying special attention to the (perhaps
confusing) behavior of taking the number of *pairs*, not the number
of *values*, which will be popped off the stack into the resulting attr
set.
Change-Id: I64df0290308ecae7a5c7e14ead37091d32701507
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6654
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Refactor the environment variable and argument parsing for the tvix repl
to use Clap instead of doing things ad-hoc, and thread through options
obtained from environment variables via explicit arguments rather than
obtaining them from the environment as they're needed. This makes adding
more flags more sustainable, and also makes the binary fully
self-documenting, including supported env vars, via `--help`.
Change-Id: Ib1f6a0cd20056e8c9196760ff755fa5729667760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6653
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Thunks might be encountered deep in equality comparison (eg nested
inside a list or attr-set), at which point we need to force them in
order to compare them for equality (or else we panic when trying to get
at their value).
Fixes: b/192
Change-Id: I912151085f8298f30d5214c7965251c9266443f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6652
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Pass in, but ignore, a mutable reference to the VM to the `nix_eq`
functions, in preparation for using that VM to force thunks during
comparison.
Change-Id: I565435d8dfb33768f930fdb5a6b0fb1365d7e161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6651
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Using rust's PartialEq trait to implement Nix equality semantics is
reasonably fraught with peril, both because the actual laws are
different than what nix expects, and (more importantly) because certain
things actually require extra context to compare for equality (for
example, thunks need to be forced). This converts the manual PartialEq
impl for Value (and all its descendants) to a *derived* PartialEq
impl (which requires a lot of extra PartialEq derives on miscellanious
other types within the codebase), and converts the previous
nix-semantics equality comparison into a new `nix_eq` method. This
returns an EvalResult, even though it can't currently return an error,
to allow it to fail when eg forcing thunks (which it will do soon).
Since the PartialEq impls for Value and NixAttrs are now quite boring,
this converts the generated proptests for those into handwritten ones
that cover `nix_eq` instead
Change-Id: If3da7171f88c22eda5b7a60030d8b00c3b76f672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6650
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This disconnects ownership of the `File` reference in a compiler from
the calling scope, which is required for when we implement `import`.
`import` will need to carry an `Rc<RefCell<CodeMap>>` (or maybe, in
the future, Arc) to give us the ability to add new detected code
files at runtime.
Note that the choice of `Arc` over `Rc` here is not ours - it's the
codemap crate's.
Change-Id: I3aeca4ffc167acbd1701846a332d93550b56ba7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6630
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
As before, this limits the cases to a relatively small number because
otherwise things get quite large.
Change-Id: I5371dc56418fca52e1dd1d905b20868f647091ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6649
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Only running 20 cases for now, since Value can get quite big if you let
it run for a while.
Change-Id: I09ef19da22c789c4869793836c98937c44595340
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6648
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
this was found by proptests!
Change-Id: I16d6a6ece3b20cdddd6f78c94cc87befb1b651e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6647
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Add an optional config argument to the `<trait>_laws` macros, to allow
configuring the generated tests with a ProptestConfig struct (to limit
the number of cases run)
Change-Id: I2143ddb72c6a870e8be4a9058135b6f9a703039e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6646
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Impl Arbitrary for Value (and NixAttrs and NixList) in the same way we
did for NixString. Value currently only generates non-"internal"
values (no thunks, AttrNotFound, etc.) and can't generate
functions (builtins or closures), because those'd require full
generation of tvix bytecode, which is a bit more work than I'd like to
do now - there's a `todo!` left in the code for a place where we could
allow opting-in to internal values and functions later.
Change-Id: I07a59e2b1d89cfaa912d4ecebd642caf4ddb040a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6627
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is pretty boring at the moment, but mostly serves as a foot in the
door in the direction of writing more tests
Change-Id: Id88eb4ec7e53ebb2d5b5c254c8f45ff750238811
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6637
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Invalid integers (eg integers that're too long) end up as error returns
on the `.value()` returned from the literal in the AST - previously we'd
unwrap this error, causing it to panic the compiler, but now we've got a
nice error variant for it (which just unwraps the underlying
std::num::ParseIntError).
Change-Id: I50c3c5ba89407d86659e20d8991b9658415f39a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6635
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This bumps rnix-parser to a commit that should be unaffected by the
Nix >= 2.4 bug that prevents it from cloning repositories with filters.
Change-Id: Ie01da95245ec6740fa889eb710819e512202f665
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6634
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Just want to note those down somewhere before I forget them again,
we can delete them later if we have it all figured out.
Change-Id: Icafa2d8fc7ca39e38e9637b7eca6f2bbf487c2b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6632
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Add a suite of proptests covering the laws of the handwritten stdlib
trait impls (Eq, Ord, and Hash) for String, generated from a new set of
macros for generating those tests which can be applied to other types.
Change-Id: Ib3276c9e96fca497aece094e5612707d3dc77ccd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6626
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
The From<String> impl for NixString only generates StringRepr::Heap
strings, but we want to make sure we're testing StringRepr::Smol too
Change-Id: I6d04b9cf12ef8462fe2788e0c6414b165f40311d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6629
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
It's proptest time!
Add deps on the proptest and test_strategy crates, gated behind a
new (default-enabled) "arbitrary" feature flag so that they don't affect
dependencies of things like tvixbolt that depend on tvix.
These are going in dependencies, not dev-dependencies, so that we can
impl Arbitrary for stuff outside of test modules (which will be
important for integration suites which want to run proptests)
Change-Id: I1613bd3ea9a835e22986ad4e59700e8736007963
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6624
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This matches the name of the AST node from which it was compiled.
Suggested by sterni in cl/6231
Change-Id: Ia51525158d2f47467c01fce2282005b1a8417a47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6623
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Add the start of a test suite that compares tvix eval results against
nix, using the string repr of the value as the comparison. This shells
out to a nix-instantiate binary, which is configurable as an environment
variable, to eval - there's some extra machinery there to setup a new
nix store as a tempdir to allow running this test inside the nix build
for tvix-eval itself.
Currently this has a macro that'll allow writing lots and lots of
hardcoded tests, but going forward I'm also going to be looking into
adding proptest-based generation of expressions to compare.
Change-Id: I9f4895fab1e668ed2b7dfd6f92f8c80de1bbb16b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6307
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
... it would be nice if we could thread it through to the `Scope`
stuff (declaring phantoms & locals).
Change-Id: Id3b84e79032b8fbb12138b719e657565355fbc79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6616
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This trait can be used to convert most structures from rnix-parser
into a codemap::Span. It uses a macro to implement the trait for the
various expression types in the rnix AST, as Rust's silly semantic
versioning restriction stops us from doing a blanket implementation.
This will be used in the next commit to clean up the span handling in
the compiler a bit.
Change-Id: I0a437034e5fa203b5a49c6f25c45932a9f3b2bca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6615
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
... and emit a warning if anyone decides to use.
Change-Id: Iaa6fe9fa932340e6d0fa9f357155e78823702576
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6611
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Yep.
This is kind of ugly right now. The idea is that the recursive_scope
compilation function is used for recursive sets as well by emitting
the keys. The rest of the logic is pretty much identical.
There is quite a lot of code here that can be cleaned up (duplication
between attrs and let, duplication inside of the recursive scope
compilation function etc.), but I'd like to get it working first and
then make it nice.
Note that nested keys are *not* supported yet.
Change-Id: I45c7cdd5f0e1d35fd94797093904740af3a97134
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6610
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This type is used in the list temporarily populated by the *second*
pass over all identifiers in a recursive scope. This first pass only
serves to make all bindings known to the compiler, without populating
their values yet.
Having a type here is going to be useful once we implement `rec`,
which needs to thread through slightly more information.
Change-Id: Ie33e0f096c5fcb6c864c991255466748b6f0d1eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6609
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This needs to be reused between let & `rec` attrs.
Change-Id: I4a3bb90af4be32771b0f9e405c19370e105c0fef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6608
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This makes the phases of attribute set construction that Nix has very
explicit (inherits, static keys, dynamic keys).
This change focuses on the split between dynamic/static keys by
collecting all dynamic ones while compiling the static ones, and then
phasing them in afterwards. It's possible we also need to do some
additional splitting inside of the inherits.
Change-Id: Icae782e2a5c106e3ce0831dda47ed81c923c0a42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6530
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This extends various ideas around the store and its slightly different
internal model.
Thanks to store composition, lazy substitution and more efficient
transfer, we can also simplify some complicated logic while building.
Change-Id: Ib3380af650fe06e114f54e8dc2df231f18af876b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6585
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This function is necessary for all builtins that expect some form of
path as an argument. It is merely a wrapper around coerce_to_string that
can shortcut if we already have a path. The absolute path check is done
in the same way as in C++ Nix for compatibility, although it should
probably be revised in the long term (think about Windows, for example).
Since coercing to a path is not an operation possible in the language
directly, this function can live in the builtins module as the only
place it is required.
Change-Id: I69ed5455c00d193fea88b8fa83e28907a761cab5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6574
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Define `.len()` method on `NixAttrs` to preallocate the capacity of the result
vector.
Also anchor an errant comment to its context (I think).
Change-Id: I268f15025d453d7b3ae1146558c80e51433dd2a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6546
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
This allows us to get rid of the count local variable which was a bit
confusing. Calling parts.len() multiple times is fine, since the length
doesn't need to be computed.
Change-Id: I4f626729ad1bf23a93cb701385c3f4b50c57456d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6584
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
With this puzzle piece of string compilation in place, `compile_str`
becomes less redundant, as every part now needs to be compiled the same.
The thunking logic becomes a bit trickier, since we need to thunk even
in the case of `count == 1` if the single part is interpolating.
Splitting the inner (shared) code in a separate function turned out to
be easier for making rustc content.
Change-Id: I6a554ca599926ae5907d7acffce349c9616f568f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6582
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
If we have multiple string parts, we need to thunk assembling the
string. If we have a single literal, it is strict (like all literals),
but a single interpolation part may compile to a thunk, depending on how
the expression inside is compiled – we can avoid forcing to early here
compared to the previous behavior.
Note that this CL retains the bug that `"${x}"` is erroneously
translated to `x`, implying e.g. `"${12}" == 12`.
The use of `parts.len()` is unproblematic, since normalized_parts()
builds a `Vec` instead of returning an iterator.
Change-Id: I3aecbfefef65cc627b1b8a65be27cbaeada3582b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6580
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Implement C++ Nix's `EvalState::coerceToString` minus some of the Path
/ store handling. This is currently only used for `toString` which does
all possible coercions, but we've already prepared the weaker coercion
variant which is e.g. used for builtins that expect string arguments.
`EvalState::coerceToPath` is still missing for builtins that need a
path, but it'll be easy to build on top of this.
Change-Id: I78d15576b18921791d04b6b1e964b951fdef22c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6571
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
OpAttrsIsSet and OpAttrsTrySelect will fail silently if the attribute
set value on the stack is actually a thunk, so we need to make sure to
force at every step of the way.
Emitting the force instructions in the compiler because it is easier to
add, but maybe the VM should do this when handling the relevant opcodes?
Comments welcome.
Change-Id: I65c5ef348d59b2d07c9bb06abb24f9f3e6a0fdb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6540
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This prevents Nix programs to observe the "internal" type of thunks.
Possibly .type_of() is also an area of the runtime where we should panic
if "internal" would ever be returned.
Change-Id: I9f358044c48ad64896fb6a1b1a42f00a29efac00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6539
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The expression inside ${…} may return arbitrary values, including
thunks, so we need to make sure to force them just in case.
Change-Id: Ic11ba00c4c92a10a83becd91233db5f57f6e59c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6541
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Avoids accidentally dropping one on the floor if we add more, pointed
out by sterni in cl/6372
Change-Id: Ib7bb0ce9c8331c8337003d20c4d5240dfae1c32a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6570
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Pointed out by sterni in cl/6370
Change-Id: I324d8049a2702ced8f30ad43a64d63ae79dd0eab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6569
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Pointed out by sterni in cl/6389
Change-Id: I648056a760266a8cfd7adcdc478c7ff2132991f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6568
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Pointed out by sterni in cl/6395
Change-Id: I2dda2bb11fef702df05fd7a4fd93b9e717a85dad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6567
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is more useful than pointing it at the entire assert expression,
as that includes the body as well which is not going to be relevant in
the error.
Pointed out by sterni in cl/6391
Change-Id: I95a5d1edf90df65e7fa53d4d04502afd6e99e89a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6566
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is no longer required, resolution is now more sane. Pointed out
by sterni in cl/6422.
Change-Id: Icc8983c648f864e66813948df6e2d4bad6a7f312
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6565
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This field no longer needs to be directly accessible by the compiler.
Addresses a sterni lint from cl/6466
Change-Id: I5e6791943d7f0ab3d9b7a30bb1654c4a6a435b1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6564
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
As suggested by sterni in cl/6453.
Change-Id: I3cf80d97c11fd7d085ab510f6be4b5f937c791ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6562
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This very closely follows the way it's done for warnings, but errors
have a lot more information available in some cases which we do not
surface yet.
Note also that due to requiring the `CodeMap`, this is not yet called
from eval.rs as the way that is threaded through needs to be
refactored, so only the method for reporting these errors as strings
is implemented so far.
Next steps for this will be to add a generic diagnostics module that
reduces some of the boilerplate for this between warnings & errors,
and which will also give us a good point in the future to switch to a
fancier diagnostics crate.
Change-Id: If6bb209f8e7a568d866e516a90335b9b2afbf66d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6534
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This implements an initial fancy display for warnings emitted by the
tvix compiler, using the codemap_diagnostic crate.
Each warning variant has an associated message, and optionally an
associated annotation for the span displayed to the user.
In theory we could get a lot more fancy with the display for specific
variants if needed (e.g. re-parse the AST and actually add multiple
semantic spans based on context), but this is already a good start.
Example:
tvix-repl> let toString = https://tvl.fyi; in let inherit toString; in ({}: 42) rec {}
warning[W004]: declared variable 'toString' shadows a built-in global!
--> [tvix-repl]:1:5
|
1 | let toString = https://tvl.fyi; in let inherit toString; in ({}: 42) rec {}
| ^^^^^^^^ variable declared here
warning[W001]: URL literal syntax is deprecated, use a quoted string instead
--> [tvix-repl]:1:16
|
1 | let toString = https://tvl.fyi; in let inherit toString; in ({}: 42) rec {}
| ^^^^^^^^^^^^^^^
warning[W002]: inherited variable already exists with the same value
--> [tvix-repl]:1:40
|
1 | let toString = https://tvl.fyi; in let inherit toString; in ({}: 42) rec {}
| ^^^^^^^^^^^^^^^^^
warning[W999]: feature not yet implemented in tvix: recursive attribute sets
--> [tvix-repl]:1:70
|
1 | let toString = https://tvl.fyi; in let inherit toString; in ({}: 42) rec {}
| ^^^^^^
warning[W999]: feature not yet implemented in tvix: closed formals
--> [tvix-repl]:1:62
|
1 | let toString = https://tvl.fyi; in let inherit toString; in ({}: 42) rec {}
| ^^
warning[W003]: variable 'toString' is declared, but never used:
--> [tvix-repl]:1:5
|
1 | let toString = https://tvl.fyi; in let inherit toString; in ({}: 42) rec {}
| ^^^^^^^^ variable declared here
=> 42 :: int
These are coloured when output to a terminal.
Change-Id: If315648a07e333895db4ae1d0915ee2013806585
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6532
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This is a crate for source-span based error reporting. Since all of
our spans are already codemap spans, it is a good starting point.
We have to figure out quite a bit of logic for neat error printing;
later on if we want fancier presentation we might want to look at one
of the other libraries in this space like miette.
Change-Id: I4e28886af1ed199b7112d9dbf063c9f29b612bf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6531
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
It is impossible for tvixbolt to recover from panics, so the user
experience of typing an expression using an unsupported feature was
that it would get sad and stop responding to input.
Instead, raise a normal value-level error of a new variant and
continue where possible.
Change-Id: Ibe016c92cacb87b85095c0f83758eddc6468053e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6528
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
only run test_nix_store_add() when the feature integration_tests is
enabled.
Change-Id: I600f08ecaefe1ce77651ae07a58d7987107ab969
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6084
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is used to drop an already emitted operation from a chunk again
and clean up its span tracking. This is required in cases where the
compiler has to backtrack.
Change-Id: I8112da9427688bb2dec96a2ddd12390f6e9734c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6499
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Refactors the methods used for determining whether an identifier in a
binding (i.e. an `rnix::Attr` node) is a static string, and extracting
it.
Previously all uses of this logic were for `let`-expressions, where
dynamic attributes are always an error. However, we need the same
logic to properly implement the phase separation of attribute set
compilation.
To facilitate this, the actual core logic of these methods now return
`Option`, and are only converted to errors in cases where the errors
are actually required.
Change-Id: Iad7826eff2cb428182521c6f92276310edeae1eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6498
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
There's about to be a lot more code for attrsets (hopefully
temporarily as part of an expand&contract cycle), while nested
attribute logic is being refactored in preparation for recursive
attribute sets.
This does not change any functionality.
Change-Id: I667565cd810ca7d9046120d1721c2ceb9095550b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6497
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
We need to make sure that we compile all plain inherits in a let
expression before declaring any other locals. Plain inherits are special
in the sense that they can never be recursive, instead resolving to a
higher scope. Thus we need to compile their value, before declaring
them. If we don't do that, before any other local can be declared,
we cause a situation where the plain inherits' values are placed into
other locals' stack slots.
Note that we can't integrate the plain inherit compilation into the
regular 2-3 phase model where we defer the compilation of the value or
we'd compile `let inherit x; in …` as `let x = x; in …`.
Change-Id: I951d5df3c9661a054e12401546875f4685b5bf08
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6496
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
This test shows that let bindings' dependencies can form a cyclical
graph, so we need to use thunking to break this cycle.
Change-Id: I2a4de71fd7024f3d3d1166154784139a82f39411
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6495
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
As the new test case demonstrates, asserts need to be evaluated lazily.
Change-Id: I808046722c5a504e9497855ca5026d255c7a4c34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6494
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
The recent change that split declaration of let based locals and the
compilation of their values did not touch locals bound by inherit in
let. These were previously declared and compiled immediately before
starting to work on the other locals introduced in a let.
In the case of plain inherits, this behavior is kept in this change,
because there's nothing wrong with it: The value of a plain inherit will
always resolve to a higher scope, either statically or dynamically.
Since inherit (from) expression might refer to other locals bound in the
same let, we need to handle them in the same three steps as ordinary let
based locals:
1. We need to declare the (uninitialised) locals.
2. We need to compile the expression that obtains their value. For this,
we create a new thunk, since the from expression may very well return
a thunk which we need to force before selecting the value we are
interested in.
3. Thunks need to be finalised.
For 1., we create an extra pass over the inherits that already declares
and initialises plain inherits and notes inherit (from) expressions in
the entries vector after declaring them. 2. only needs a bit of adapting
to create the thunks for selecting if appropriate, the rest of the
existing code can be reused.
Change-Id: Ie4ac1c0f9ffcbf7c07c452036aa8e577443af773
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6490
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
With this all other "weird scope" logic starts working for `with` as
well.
Change-Id: I0ea1d8c5fbd9cec5084bd574224f77b71ff2b487
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6487
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This completely rewrites the handling of "dynamic upvalues" to,
instead of resolving them at thunk/closure instantiation time (which
forces some values too early), capture the entire with stack of parent
contexts if it exists.
There are a couple of things in here that could be written more
efficiently, but I'm first working through this to get to a bug
related to with + recursion and the code complexity of some of the
optimisations is distracting.
Change-Id: Ia538e06c9146e3bf8decb9adf02dd726d2c651cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6486
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This struct will be responsible for tracking upvalues (and is a
convenient place to introduce optimisations for reducing value clones)
instead of a plain value vector.
The main motivation for this is that the upvalues will have to capture
the `with`-stack fully and I want to avoid duplicating the logic for
this between the two capturing types.
Change-Id: I6654f8739fc2e04ca046e6667d4a015f51724e99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6485
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Instead of the reference to the Rc, print the address of the Rc
itself.
Change-Id: I4560598924db7d2864d5c4ae9af847aee2ea7eff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6471
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Similarly to attribute sets, list elements can be arbitrary
expressions and their (temporary) stack slots during construction must
be accounted for by the compiler.
Change-Id: I3b6f7927860627fd867c64d0cab9104fd636d4f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6470
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The temporaries left on the stack as operands to `OpAttrs` must be
accounted for in the locals array in order for operations within them
to receive correct slots.
Some test cases that were previously broken have been added.
Change-Id: Ib52b629bbdf7931f63fd45a45af1073022da923c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6468
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
There are more upcomming uses of declare_phantom where this will come
in handy to avoid some code bloat.
Change-Id: I75cad8caf14511c519ab2f56e87e99bcbf0a082e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6467
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Moves the logic for removing tracked locals from a given scope from
the compiler's locals list, and leaves only the actual
compiler-related stuff (emitting warnings, cleaning up locals at
runtime) in the compiler itself.
Change-Id: I9da6eb54967f0a7775f624d602fe11be4c7ed5c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6466
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This will be re-used between the code paths for
recursive/non-recursive sets, and it might even be possible to unify
it with the logic for compiling `let inherit ...`.
Change-Id: I960a061048ac583a6e932e11ff6e642d9fc3093e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6464
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The comment explains how this works fairly well.
Note that this does not yet have the ability to check "closed
formals", i.e. without an ellipsis Tvix will *NOT* fail if unexpected
attribute set keys are provided.
Change-Id: I0d2b77e893243093d2789baa57f876d35d0a32ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6463
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This doesn't actually do anything other than causing errors for the
current state of this folder.
Change-Id: I0af6410e9eb1700cd0b2b105c8adde077d931a69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6492
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
... even if the code is broken.
Change-Id: I5898bceaebf201b97e8988c94c90e7fafff82529
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6462
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
As pointed out by sterni in cl/6205, this is actually possible in
syntactically valid expressions like
{ ${12 + 13} = 12; }
Change-Id: Id8a1e3aceb551f288f9050c4eea563eb6572f1a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6461
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
As pointed out by grfn on cl/6091
Change-Id: I28308577b7cf99dffb4a4fd3cc8783eb9ab4d0d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6460
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
When the last instruction in a chunk is OpCall, make it an OpTailCall instead.
Change-Id: I2c80a06ee85e4abf545887b1a79b6d8b5e6123e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6458
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
If the last operation within a chunk is a function call, the call can
be executed in the same call frame without increasing the depth of the
call stack.
To enable this, a new OpTailCall instruction (similar to OpCall) is
introduced, but not yet emitted by the compiler.
Change-Id: I9ffbd7da6d2d6a8ec7a724646435dc6ee89712f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6457
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This introduces a macro to do the forcing, but this solution isn't
very nice and also does not work in all cases yet.
Change-Id: Icd18862ec47edb82c0efc3af5835a6cb6126f629
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6456
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This produces similar output to the previous tracing feature, but can
redirect the output somewhere else.
Change-Id: I9493c260f480904f3932cb74809b622c24d7be96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6453
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
These methods make it possible to trace the runtime execution of the
VM through an observer.
Change-Id: I90e26853ba2fe44748613e7f761ed5c1c5fc9ff7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6452
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
These are required for tvixbolt to work. This interface is definitely
not stable yet, though.
Change-Id: I4076498e8f42311de74ee4f33c93a3ee0c5f8d3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6450
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This type implements an observer that is called whenever the compiler
emits a chunk (after the toplevel, thunks, or lambdas) and prints the
output of the disassembler to its internal writer.
This replaces half of the uses of the `disassembler` feature, which
has been removed from the Cargo configuration.
Note that at this commit runtime tracing is not yet implemented as an
observer.
Change-Id: I7894ca1ba445761aba4ad51d98e4a7b6445f1aea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6449
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This trait will enable library users of tvix-eval to observe internal
happenings of the compilation and runtime processes.
The initial methods of the observer will be called whenever the
compiler emits a chunk.
Change-Id: I668f6c2cfe3d6f4c1a1612c0f293831011768437
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6448
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is a step towards hiding the internal fields of thunk, and making
the interface of the type more predictable.
Part of the preparation for implementing observers.
Change-Id: I1a88a96419c72eb9e2332b56a2dd94afa47e6f88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6447
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
With this change, it becomes possible to compile tvix-eval to
webassembly if the `repl` feature is disabled.
Change-Id: Icc0a059964cd0bea2054110c682d50fc5c87ec01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6446
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
With this, most cases of `fix` in attribute sets will work correctly.
A simple test exercising both has been added.
Change-Id: I70fd431177bb6e48ecb33a87518b050c4c3d1c09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6437
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
At the point where control flow exits Thunk::force (which may be due
to recursing), it is vital that there is no longer a borrow to the
inner thunk representation, otherwise this can cause accidental
infinite recursion (which will be detected, but cause failures on
valid code).
Change-Id: I2846f3142830ae3110a4f5d2299e9d7928634504
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6436
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This makes it easier to track exactly which lambda is which when
inspecting e.g. the concrete representation of a thunk.
At runtime all lambdas live in an Rc. To make this print the right
address, the construction of these Rcs had to be moved up right to the
point where the lambda is first emitted (and disassembled).
Change-Id: I6070e6c8ac55f0bd697966c4e7c5565c20d19106
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6435
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
There can be different spans on the same line, so the previous
implementation would duplicate line numbers unnecessarily.
Change-Id: I8d8db77177aee0d834a6ec3584641e1bd5f31c3e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6434
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This was fixed by some of the previous commits around scopes. It's
somewhat similar to a few other tests, but I had this one failing
earlier and everything else succeeding, so it is useful to keep it
around for sure.
Change-Id: Ie6cf372b5c805daf992cd87aeb3dfe91542c381c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6431
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Deferred local upvalues can *only* occur at the same depth as the
thing that is closing over them, but there are various situations with
scope nesting where the actual stack indexes of the local and the
closer look like a deferred value is being accessed.
To fix this, simply compare the depth as well.
Change-Id: Ice77424cc87ab0a2c4f01379e68d4399a917b12b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6429
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is the same as `eval-okay-attrs-simple-inherit`.
Change-Id: I23878accc6cd62c16ec96601239838a385d31306
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6428
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
The condition here was extremely hard to read prior to this change.
As the locals vector is now guaranteed to never be empty (there is
always at least a phantom for the current chunk's root expression),
the logic here can be simplified to just dropping tailing locals
entries while their depth matches that of the scope being closed.
Change-Id: I24973e23bc2ad25e62ece64ab4d8624e6e274c16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6427
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Similar to setting up a phantom slot when compiling the root value of
a file, closures and thunks need to have a phantom stack slot for the
root of the expression yielded by their thunk to make all accounting
work correctly.
The tricky thing here is that closures & thunks *escape* their inner
lambda context (that's the point!), so the functions emitting them
need to know both the *inner* slot (to resolve everything correctly
while compiling the slot) and the *outer* slot (to correctly emit
instructions for closing over upvalues).
Change-Id: I62ac58e2f639c4b9e09cc702bdbfd2373e985d7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6426
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Instead of using a sentinel LocalIdx which potentially points to a
value in the locals stack that does not actually exist, set up an
initial uninitialised phantom value representing the result of the
root expression.
Change-Id: I82ea774daab83168020a3850bed57d35ab25c7df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6424
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
When deciding whether an upvalue needs to have a deferred resolution
step, the *stack* indexes should be compared - not the locals indexes.
The results are almost always the same, but there are tricky
situations where this can cause errors.
It's difficult to reproduce these errors in isolation, as they depend
on other scope behaviour, so this is one in a series of commits to
address the combination of issues which will gain some tests at the
end.
Change-Id: Iaa400b8d9500af58f493ab10e4f95022f3b5dd21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6423
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Instead of using sentinel values and an additional bool, this tracks
the identifier of a local as an enum that is either a statically known
name, or a phantom.
To make this work correctly some more locals related logic has been
encapsulated in the `scope` module, which is a good thing (that's the
goal).
Phantom values are now not initialised by default, but the only
current call site of phantoms (`with` expression compilation) performs
the initialisation right away.
This commit changes no actual functionality right now, but paves the
way for fixing an issue related to `let` bodies.
Change-Id: I679f93a59a4daeacfe40f4012263cfb7bc05034e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6421
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
The slot is now always known (at the root of the file it is simply
stack slot 0 once the scope drops back down to 0), so it does not need
to be wrapped in an `Option` and accessed in cumbersome ways anymore.
Change-Id: I46bf67a4cf5cb96e4874dffd0e3fb07c551d44f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6420
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
With this change the runtime trace contains much more exact
information about the context of the computation (entering/exiting
calls etc.)
This is in large part due to moving the tracer to be a field on the VM
itself, which enables consistent ordering of traces across the
execution, and tracing an execution with its *input* instead
of *output* stack.
Change-Id: Ibe525e6e7d869756501e52bef1a441619ce7332c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6419
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This makes it much easier to figure out what happened while debugging
this sort of thing.
Change-Id: I2e0e8096709adc647d63c04f213c547c415e5f44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6418
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This could occur when the disassembler is enabled and tracing the
runtime while a thunk is being evaluated, as it would not be possible
for the *tracer* to borrow the thunk at this exact moment.
However, we know that if the borrowing fails here we are dealing with
a not-fully evaluated thunk (blackhole), which should just print the
internal representation.
Change-Id: I4bdb4f17818d55795368e3d28842048f488f0a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6416
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Previously, "calling" (setting up the VM run loop for executing a call
frame) and "running" (running this loop to completion) were separate
operations.
This was basically an attempt to avoid nesting `VM::run` invocations.
However, doing things this way introduced some tricky bugs for exiting
out of the call frames of thunks vs. builtins & closures.
For now, we unify the two operations and always return the value to
the caller directly. For now this makes calls a little less effective,
but it gives us a chance to nail down some other strange behaviours
and then re-optimise this afterwards.
To make sure we tackle this again further down I've added it to the
list of known possible optimisations.
Change-Id: I96828ab6a628136e0bac1bf03555faa4e6b74ece
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6415
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
If the disassembler feature is enabled, make sure that an Rc of the
codemap is available through the chunk.
Change-Id: I700f27ab665a704f73457b19bd2d7efc93828a16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6414
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
The casting methods of `Value` are pretty verbose, and actually
incorrect before this commit as they did not account for inner thunk
values.
To address this, we first attempt to make them correct by introducing
a standard macro to generate them and traverse the inner thunk(s) if
necessary.
This is likely to be a performance hit as it will now involve more
cloning of values. We can do multiple things to alleviate this, but
should do some measurements first.
Change-Id: If315d6e2afe7b69db727df535bc6cbfb89a691aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6412
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This makes it possible for builtins to force values on their own,
without the VM having to apply a strictness mask to the arguments
first.
Change-Id: Ib49a94e56ca2a8d515c39647381ab55a727766e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6411
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Scope poisoning must be inherited across lambda context boundaries,
e.g. if an outer scope has a poisoned `null`, any lambdas defined on
the same level must reference that poisoned identifier correctly.
Change-Id: I1aac64e1c048a6f3bacadb6d78ed295fa439e8b4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6410
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
With this, if an error occurs while forcing a thunk (which is very
likely) it is threaded through to the top by wrapping it in the
ErrorKind::ThunkForce variant.
We could use this to generate "stacktrace-like" error output if we
wanted, or simply jump through and discard everything except the
innermost error.
Change-Id: I3c1c8708c2f73ae062815adf490ce935b1979da8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6409
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Previously error spans were optional because the information about
code spans was not available at runtime. Now that this information has
been added, the error type will always carry a span.
This change is very invasive all throughout the codebase. This is due
to the fact that many functions that are called *by* the VM expected
to return `EvalResult`, but this no longer works as the span
information is not available to those functions - only to the VM
itself.
To work around this the majority of these functions have been changed
to return `Result<T, ErrorKind>` instead and an accompanying macro in
the VM constructs the "real" error.
Note that this implementatino currently has a bug where errors
occuring within thunks will yield the location at which the thunk was
forced, not the location at which the error occured within the code.
This will be fixed soon, but the commit is large enough as is.
Change-Id: Ib1ecb81a4d09d464a95ea7ea9e589f3bd08d5202
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6408
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Get the length of a list
Change-Id: I41d91e96d833269541a1b3c23b7cc879f96d1e5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6407
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
First pass at supporting `builtins` for tvix. The following tests appear to be
WAI:
```shell
$ cd tvix/eval
$ cargo build
$ cargo test
```
Change-Id: I27cce23d503b17a886d1109e285e8b4be4264977
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6405
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Another step towards being able to report accurate errors. The codemap
spans contain strictly more accessible information, as they now retain
information about which input file something came from.
This required some shuffling around in the compiler to thread all the
right information to the right places.
Change-Id: I18ccfb20f07b0c33e1c4f51ca00cd09f7b2d19c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6404
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
As of this commit, the source spans of all emitted bytecode are fully
tracked.
Change-Id: I4c83deee0fc3f5e6fd6acad5a39047aec693b388
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6403
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
These source spans will always point to the *value* that is being
forced, not the instruction that caused the force to be emitted. This
makes sense so that errors during forcing point at the value and not
the surrounding expression.
Change-Id: I4694414a3281a0de878f71634105b92148ec61f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6402
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
With this change, the upvalue data instructions used by finalisers for
thunks and closures track the source span of the first identifier that
created the upvalue (if the same value is closed over multiple times
the upvalue will be reused, hence only the first one).
To do this the upvalue struct used by the compiler's scope now carries
an identifier node, which had to be threaded through quite a few
places.
Change-Id: I15a5fcb4c8abbd48544a2325f297a5ad14ec06ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6400
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This separation makes it possible to annotate the upvalue itself with
the span that created it, which (due to upvalue reuse) is only the
first one for an instance of the given UpvalueKind.
Change-Id: I9a991da6a3e8d71a92f981314bed900bcf434d44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6399
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
These are again a bit tricky in terms of emitted errors. The main
error is that the condition is not a boolean, which means that the
jump inspecting the condition must derive from the condition itself to
return an error at the correct position.
For other parts of the expression, it is simply the node itself.
Change-Id: I72411630e5d57dfc199f4c3c48afe443fe966322
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6392
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This one is tricky, specifically the span used for the final jump. I
decided that it makes sense to use the attrpath node, as the final
jump is the one that jumps *over* the default value, so the effect of
this is more closely related to the selector than the default.
It might be more correct to pass through the `or` token itself and
point to this for the jumps, but it depends a bit on what shape of
errors we could end up producing from this.
Change-Id: I29fbc97ba6b9e14e1a0e5f3a7759ddc299dd9c0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6390
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
These are not actually used yet; this is in preparation for a
multi-commit chain for emitting all the right spans in the right
locations.
Change-Id: Ie99d6add2696c1cc0acb9ab928917a10237159de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6379
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This instantiates a codemap outside of the compiler and passes a
reference to the file currently under compilation to it. Note that the
"file" might just be a REPL line.
Change-Id: I131ae1ddb6d718e1374750da9ba0b99608c6058d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6378
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is currently just a wrapper around Chunk::push_op, but will gain
the span resolution logic in a moment.
Change-Id: I862bf9ecff0932f8da6708401ea044b9442c5d5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6377
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This adds a new vector to the chunk data structure which tracks spans
into a codemap. The compiler will emit this information to the chunk
when adding instructions.
The internal representation of the spans is slightly optimised to
avoid storing duplicate spans, as there are cases where many
instructions might be derived from the same span.
Change-Id: I336f8c912e7eb50ea02ed71e6164f651ca3ca790
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6376
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Updated the rnix hash manually, and ran `cargo update` for the rest.
Change-Id: I457262625d648e25d745efa4d33ae44cb8f21326
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6375
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This will be used to track source spans when emitting bytecode.
The codemap is a data structure which tracks *all* the source files
visited by an evaluation, and makes it possible to represent locations
across all of the files using a simple span (i.e. pair of offsets).
When reporting errors, this even contains enough information to
reconstruct the rnix AST to create fancier reporting in certain cases
if desired.
Change-Id: I4ae98620b9b150fb5a389bd7f1e12670e3192c62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6374
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Note that I've allowed `needless_lifetimes` for the attribute set
iterator, as I find the type easier to understand with these
annotations present.
Change-Id: I33abb17837ee4813076cdb9a87f54bac4a37044e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6373
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This essentially makes the VM behave like `nix-instantiate --eval
--strict`, i.e. data structures are traversed strictly and thunks are
forced. Thunks embedded in closures are not forced.
This allows us to re-enable tests that were disabled because they
needed to output nested thunk contents, but is overall a behaviour
that must be configurable later on, as it is not cmopatible with e.g.
an evaluation of nixpkgs.
Change-Id: I5303a5c8e4322feab1384fdb7712fecb950afca5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6372
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This does not require a custom iterator type (for now?)
Change-Id: I5beb194bd8629571bd4040c69c977c27149807fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6371
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
If a thunk is already evaluated, there are cases where due to the
memoisation implementation something might observe a value wrapped in
a thunk.
In these cases, the implementation of `Display` and `PartialEq` must
delegate to the underlying value.
Note that there are a handful of other cases like these which we need
to cover.
It is a little tricky to write integration tests for these directly,
especially as some of the open-upvalue optimisations coming down the
pipe will reduce the number of observable thunks.
One test that covers a part of this behaviour is currently
disabled (needs some more machinery), but it's being brought back in
the next commits.
Change-Id: Iaa8cd338c12236af844bbc99d8cec2205f0d0095
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6370
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Implementing iteration over NixAttrs requires a custom iterator type
in order to encapsulate the different representations. The BTreeMap
for example has its own iterator type which needs to be encapsulated.
This is mostly boilerplate code, but for a change some simple unit
tests have been added in.
Change-Id: Ie13b063241d461b810876f95f53878388e918ef2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6367
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
These static strings show up a bunch when dealing with the internals
of attribute sets, and having them available as static references is
required.
Due to the way const expressions are evaluated, taking a reference to
the existing NixString::NAME / NixString::VALUE items does not work
and the references themselves need to be const-evaluated.
Change-Id: If6e75847af978118a3b266fe6a3242321722434d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6366
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The VM previously took care of repeatedly forcing a thunk until it
reached an evaluated state. This logic is now encapsulated inside of
the `Thunk::force` implementation.
In addition, force no longer returns a reference to the value by
default, leaving it up to callers to decide whether they want to
borrow the value or not (a helper is provided for this).
Change-Id: I2aa7da922058ad1c57fbf8bfc7785aab7971c02b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6365
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This thunks the construction of attribute sets. Because Tvix does not
currently have a "strict output" mode, a test had to be disabled that
now displays a thunk representation.
The test will be re-enabled once that is available.
Change-Id: I360332be64cd5c154f9caea21828f6f1b37a265c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6363
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This lets us create a release build with debug info, for use with e.g.
perf + hotspot
Change-Id: I03897de36c872d318abf1332ca0c1aeabe344ec6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6362
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
As noticed by sterni in cl/6195
Change-Id: Ie9c1e80e2e709284fa8412334af9188d999f64dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6361
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This should not have grown a second implementation of the identifier
resolution logic, but it somehow did.
This implementation ended up being incorrect because it did not
account for upvalues inside of thunks.
Change-Id: Ieb1364d8fe43c96aaf4b125fd4b8a522aedff167
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6360
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
All attribute set *key* related operations strictly evaluate all key
fragments, including during construction of an attribute set.
Change-Id: I3519e5e9b0886c2cdc8615ea7dcb5f7be0c59b3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6358
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The arguments of all unary/binary operators that are built in to Nix
are forced when encountered. This emits the necessary OpForce operations.
Change-Id: I691fcdbebfe7586cfe217c68d44b10b1192f82d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6357
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
With this change any compilation of an expression is aware of its own
stack slot if it is leaving identifiers on the stack.
Change-Id: I0c9f148ae06b078a46b25180c4961686d5f2e166
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6356
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This threads the current local slot through the `compile_attr`
function and all of its callers. At the moment this does not improve
any user-facing behaviour, just internally changes the way in which
some correct expressions would fail to run.
Eventually this slot will need to reach everywhere ...
Change-Id: Iba73123dd1ced421093d8fc18ebeeffc16efacf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6355
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is the simplest kind of thunk that can be created (and so far the
only one the compiler knows how to create), in which an identifier
inside a `let` encounters a value that is bound *after* it is
initialised.
Change-Id: I6ea4408a3baef1e7d5137365d70804283f2dbf8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6354
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This operation forces the evaluation of a thunk.
There is some potential here for making an implementation that avoids
some copies, but the thunk machinery is tricky to get right so the
first priority is to make sure it is correct by keeping the
implementation simple.
Change-Id: Ib381455b02f42ded717faff63f55afed4c8fb7e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6352
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The capacity (i.e. number of builtins) is known from the lambda, so we
can size it correctly right away.
Change-Id: Iab0b5a3f47d450fa9866c091ebbbed935b934907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6351
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Implements an operation very similar to `OpClosure` which populates a
thunk's upvalues and leaves it on the stack.
Change-Id: I753b4dfeeaae6919316c7028ec361aaa13d87646
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6350
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This function is reusable between thunks & closures.
Change-Id: I44d5f9897b087a385c8e75027d2ff39c48a096f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6349
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
CallFrame has to work for both thunks & closures (as a thunk is
basically a "weird 0-argument closure").
We opt to store the common, relevant fields directly in the frame to
avoid having to dereference through the nested structures
constantly (which would be especially annoying in the case of thunks).
Change-Id: I47781597b84ec5cd55502dba1713e92cf2592af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6348
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This trait abstracts over the commonalities of upvalue handling
between closures and thunks.
It allows the VM to simplify the code used for setting up upvalues,
without duplicating between the two different types.
Note that this does not yet refactor the VM code to optimally make use
of this.
Change-Id: If8de5181f26ae1fa00d554f1ae6ea473ee4b6070
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6347
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
For now, do not distinguish between closing and non-closing thunks, it
will make the initial implementation easier. See Knuth etc.
Change-Id: I0bd51e0f89f2c77e90bac63b507e5027b649e3d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6346
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
When resolving a value on the same level that is known but not yet
defined, emit a thunk.
Consider for example:
let
# v--- requires a thunk
a = 1 * b;
b = 10;
in a
Change-Id: I922cb50973ebe05e335a7bc7cb851960cf34733b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6345
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The logic in this method is *very* similar to `compile_lambda`. It is
intended to be called around any expression that should be
thunked (such as function applications, attribute set values, etc.).
Change-Id: Idfbb2daa9f4b735095378fb9c39a2fd07c8cff91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6344
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Introduces the representation of runtime thunks, that is lazily
evaluated values. Their representation is very similar to closures.
Change-Id: I24d1ab7947c070ae72ca6260a7bbe6198bc8c7c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6343
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This exact same logic is reused for thunk creation.
Change-Id: I731db9cc659a1f2ca87db55d58d6ff632f417812
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6342
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This test case was previously broken by the bug introduced by
confusing local and stack indexes.
Change-Id: Ibef299dad266c6105deac1da5dde112fe9f640b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6341
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Previously the functions in the scope module returned usize values,
which - sometimes from the same function - were either indexes into
the runtime stack *or* indexes into the compiler's local stack.
This is extremely confusing because it requires the caller to be aware
of the difference, and it actually caused subtle bugs.
To avoid this, there is now a new LocalIdx wrapper type which is used
by the scope module to return indexes into the compiler's stack, as
well as helpers for accounting for the differences between these
indexes and the runtime indexes.
Change-Id: I58f0b50ad94b28a304e3372fd9731b6590b3fdb8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6340
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
With this change, the runtime can correctly capture deferred upvalues.
Change-Id: I1e43b7b1ac2553b1812424adfc8bd08ef77bf1ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6339
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
When encountering a deferred local upvalue, the compiler will now mark
the corresponding local as needing a finaliser which makes it possible
to emit the OpFinalise instruction for this stack slot a little bit
down the line.
Change-Id: I3962066f10fc6c6e1472722b8bdb415a811e0740
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6338
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This instruction finalises the initialisation of deferred upvalues in
closures (and soon, thunks).
The compiler does not yet emit this instruction, some more accounting
is needed for that.
Change-Id: Ic4181b26e19779e206f51e17388559400da5f93a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6337
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This will leave a sentinel value in the upvalue slot in which the
actual value is to be captured after resolution once a scope is fully
set up.
Change-Id: I12b37b0dc8d32603b03e675c3bd039468e70b354
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6336
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Uses the threaded through slot offset to determine whether
initialisation of a captured local upvalue must be defered to a later
point where all values of a scope are available.
This adds a new data representation to the opcode for this situation,
but the equivalent runtime handling is not yet implemented. This is in
part because there is more compiler machinery needed to find the
resolution point.
Change-Id: Ifd0c393f76abfe6e2d91483faf0f58947ab1dedc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6329
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
While compiling local bindings, we now know the stack slot of the
value currently being compiled.
This will let us determine whether an upvalue can be captured directly
or whether it needs to wait for a synchronisation point at which the
upvalue can be instantiated.
This machinery lets us avoid unnecessary work at runtime when
instantiating closures that actually do not need complicated recursive
resolution.
This change itself introduces no new functionality, but since the
threading is noisy it is split out as a separate change.
Change-Id: I847c677ee8f6725fda1d2efd689b6a58bdccb779
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6328
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Instead of looking up the local to be initialised by its name again,
we can simply track the index at which it was declared from the point
where the declaration was made.
This reduces some string cloning and removes unnecessary logic. It
also theoretically makes the *current* index available during locals
compilation, which can be used to optimise some recursion cases.
Change-Id: I06f403603d4f86c3d319debfe74b5a52eec00990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6327
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This actually makes things full-circle, as this tree already had this
implementation once before all the other required components were in
place.
With this commit, the compiler can resolve recursive upvalues within
the same scope (though they will not yet work at runtime).
Change-Id: I6267e477d08f367257c3a6dde054b880d7b47211
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6326
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
In order to resolve recursive references correctly, these two can not
be initialised the same way as a potentially large number of (nested!)
locals can be declared without initialising their depth.
This would lead to issues with detecting things like shadowed
variables, so making both bits explicit is preferable.
Change-Id: I100cdf1724faa4a2b5a0748429841cf8ef206252
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6325
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The previous closure refactoring introduced a bug in which the same
closure object would get mutated constantly for each instance of a
closure, which is incorrect behaviour.
This commit instead introduces an explicit new Value variant for the
internal "blueprint" that the compiler generates (essentially just the
lambda) and uses this variant to construct the closure at runtime.
If the blueprint ever leaks out to a user somehow that is a critical
bug and tvix-eval will panic.
As a ~treat~ test for this, the fibonacci function is being used as it
is a self-recursive closure (i.e. different instantiations of the same
"blueprint") getting called with different values and it's good to
have it around.
Change-Id: I485de675e9bb0c599ed7d5dc0f001eb34ab4c15f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6323
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This puts together the puzzle pieces for threading dynamic
upvalues (that is, upvalues resolved from the `with`-stack) all the
way through.
Reading the test case enclosed in this commit and walking through it
is recommended to understand what problem is being tackled here.
In short, because the compiler can not statically know *which*
with-scope a dynamic argument is resolved from it needs to lay the
groundwork for resolving from *all* possible scopes.
There are multiple different approaches to doing this. The approach
chosen in this commit is that if a dynamic upvalue is detected, the
compiler will emit instructions to close over this dynamic value
in *all* enclosing lambda contexts.
It uses a new instruction for this that will leave around a sentinel
value in case an identifier could not be resolved, and wire the
location of this found value (or sentinel) up through the upvalues to
the next level of nesting.
In this tradeoff, tvix potentially closes over more upvalues than are
needed (but in practice, how often do people create *really* deep
`with`-stacks? and in *this* kind of code situation? maybe we should
even warn for this!) but avoids keeping the entire attribute sets
themselves around.
Looking at the test case, each surrounding closure will close
over *all* dynamic identifiers that are referenced later on visible to
it, but only the last one for each identifier will actually end up
being used.
This also covers our bases for an additional edge-case this creates,
in which an identifier potentially resolves to a dynamic upvalue *and*
to a dynamic value within the function's own scope (again, would
anyone really do this?) by introducing a resolution instruction for
that particular case.
There is likely some potential for cleaning up this code which is
quite ugly in some parts, but as this implementation is now carefully
calibrated to work I decided it is time to commit it and clean it up
in subsequent commits.
Change-Id: Ib701e3e6da39bd2c95938d1384036ff4f9fb3749
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6322
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
With this change, encountering a dynamic upvalue will thread through
all contexts starting from the lowest context that has a non-empty
`with`-stack.
The additional upvalues are not actually used yet, so the effective
behaviour remains mostly the same. This is done in preparation for an
upcoming change, which will implement proper dynamic resolution for
complex cases of nested dynamic upvalues.
Yes, this whole upvalue + dynamic values thing is a little bit
mind-bending, but we would like to not give up being able to resolve a
large chunk of the scoping behaviour statically.
Change-Id: Ia58cdd47d79212390a6503ef13cef46b6b3e19a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6321
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is a common idiom in both Nix and other languages when a local is
declared without actually being used.
Since Tvix warns for unused locals, having this available is useful
and can be included in the final error message as a suggestion if an
unused variable is intentional.
Change-Id: Ia85f704ba183499a3bae657c58166e2e29f9bde5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6320
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This does not yet correctly resolve them if they are more than one
scope up, however.
Change-Id: I6687073c60aee0282f2b6ffc98b34c1e96a60f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6319
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
With this change, it becomes possible for functions to call themselves
as they are being defined in local bindings.
Change-Id: Ib46a39ba17b1452b5673d96fa729d633d237241a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6314
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is required to efficiently construct the upvalue array at
runtime, as there are situations where during Closure construction
multiple things already have a reference to the closure (e.g. a
self-reference).
Change-Id: I35263b845fdc695dc873de489f5168d39b370f6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6312
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
`builtins.getFlake` doesn't interest us, of course, but some others may
be worth (or easy) to implement. They are pretty low priority, though,
since nixpkgs has compatiblity wrappers for the ones it uses.
The new debugging-related builtins (break and traceVerbose) are
interesting to note, but may not make sense to implement at all.
Change-Id: Icae547aa3bd9d6ee6b87897ba8210eb9b9b044c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6332
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
This avoids unnecessary rebuilds of //tvix/eval when working on it
locally using impure cargo.
Change-Id: I028033a6345a9655e2877a534448706b8f85a1a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6317
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
The `With` struct no longer contained any internals after the cleanup
logic for the stack had been moved into Compiler::compile_with,
leaving the `Vec<With>` to essentially act as a counter for the number
of things on the with stack.
That's inefficient of course, so with this commit it actually becomes
an integer (with an encapsulated API within scope::Scope).
Change-Id: I67a00987fc8b46b30d369a96d41e83c8af5b1998
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6311
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The compiler module is getting quite long and this will help keep some
order.
Right now the scope internals are not very well encapsulated; this
paves a way to reducing the API surface of the `scope` type to the
things that are actually used by the compiler instead of giving access
to its internals.
Change-Id: I8c16c26d263f018baa263f395c9cd80715199241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6310
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This has no effect yet, other than changing the way in which some
upvalue captures break (that are already not working correctly).
However, after this change the compiler correctly detects
self-recursion and can start emitting the instructions to deal with
this at runtime.
Change-Id: Id3b0ac206c0204739597a4325bcc66f9c806c242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6309
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Calculating the with_idx (i.e. the stack offset of the "phantom"
variable from which a `with` dynamically reads at runtime) needs to
account for unitialised variables the same way as the resolution of
normal locals does.
Change-Id: I9ffe404535bf1c3cb5dfe8d9e005798c857fff94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6308
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is required to correctly clean up the `with` values.
At the moment, the attrset from which identifiers are being taken is
always pushed on the stack. This means that it must also be removed
again, otherwise in an expression like
with { a = 15; }; a
The final stack is `[ { a = 15; } 15 ]` *after the last operation*,
which means that the attrset is still on there as garbage.
This has little practical impact right now because it is always
shadowed by the fact that the actual expression value is at the right
location, but becomes relevant when accounting for upvalue captures.
Change-Id: I69e9745bfaa4d6bbcb60ee71f4dc3f8d8695d16a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6303
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This extends the logic of `Scope::resolve_local` to detect cases where
self-recursion is occuring (i.e. an identifier is being accessed in
its own identifier).
These cases are not yet handled specially, and the logic of when
things are marked initialised (which was previously always at the same
spot as their declaration) has not changed, making this commit a
runtime no-op for now.
Change-Id: I3179642a7c55869ad4465fdd2678b0cd51a20f15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6302
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Nix does not allow things like `let a = 1; a = 2; in a`, but doing it
across depths is allowed.
Change-Id: I6a259f8b01a254b433b58c736e245c9c764641b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6301
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This does not yet change anything semantically, but will be useful for
resolving simple cases of self-recursion etc.
Change-Id: I139ecb7e4a8a81193774392a96e73e0ea6b9f85d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6300
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
These need to be handled specially by the runtime if the compiler
determines that a given local must be resolved via `with`.
Note that this implementation has a bug: It currently allows `with`
inside of nested lambdas to shadow statically known identifiers. This
will be cleaned up in the next commit.
Change-Id: If196b99cbd1a0f2dbb4a40a0e88cdb09a009c6b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6299
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The previous implementation of OpResolveWith manually controlled the
loop iteration, which skipped over the disassembler's tracing
instruction.
Instead, the resolution of dynamic variables has been delegated to a
new helper function. This has the additional benefit that the loop
labels are no longer required, making things a bit cleaner.
Change-Id: If22b74c3d49c74bf3a1ec4497cb761a9ee6cf2a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6298
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Instead of tying the popping of the with stack to scope depth, clean
up the stack immediately after processing a with body.
The previous behaviour was actually incorrect, as it would leave
things on the with-stack longer than they were supposed to be there.
This could lead to false positive resolutions in some situations
involving closures.
Change-Id: I7b0638557503f1f71eb602e3d5ff193cdfcb67cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6297
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Implements the final bit of logic remaining for wiring up closures,
which is the runtime construction of closure objects.
When encountering an OpClosure, the VM walks through the bytecode
collecting all the upvalue location operands (see commit introducing
the OpCode::Data* variants for details) and stores the runtime values
in the new closures upvalue vector.
After that, the handling of the closure itself becomes functionally
identical to that of lambdas.
With this initial implementation of closures there are several large
optimisation potentials available, the two most notable ones are:
- Distinguish the runtime representation of lambdas and closures
explicitly.
- Detect and handle multiple-arity functions directly in the compiler.
However, for both of these we should wait until we have appropriate
benchmarking infrastructure in place. This is because our test
implementations have shown that the complexity of either of these
changes is quite significant, and we do not yet know if they really
pay off.
Change-Id: I077e977810fd5cb2b1ecd7f1a119e728025dd786
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6295
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This resolves an upvalue at runtime by pushing it on the stack from
the closure's upvalue vector.
Change-Id: Ic3e7a7ecd9f7032f679114a1995e5bbf83062fcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6294
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
In preparation for implementing calling of closures, store a closure
directly in the VMs call frame.
Change-Id: Iad24cd8c49fee4ebd4d0c84ffaa4c2505ee3dfd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6293
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
... same as the others
Change-Id: I9c8868388c10b0b6484c5bdd3799d801296c6979
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6292
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Fully implements the instructions for compiling closure
objects (without runtime handling yet).
Closure (and thunk) objects are created at runtime by capturing all
known upvalues. To represent this, the instructions for creating them
need to have a variable number of arguments. Due to that, this commit
introduces new variants in OpCode that are not actually operations,
but data.
If the VM is implemented correctly, the instruction pointer should
never point at these. Due to this, the VM will panic if it sees a data
operand during an execution run.
Change-Id: Ic56b49b3a42736dc437751e76df0e89c8d0619c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6291
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This adds a new upvalue tracking structure in the compiler to resolve
upvalues and track their positions within a function when compiling a
closure.
The compiler will emit runtime upvalue access instructions after this
commit, but the creation of the runtime closure object etc. is not yet
wired up.
Change-Id: Ib0c2c25f686bfd45f797c528753068858e3a770d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6289
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This adds a transparent wrapper around `usize` used for jump offsets
in the opcodes. This is a step towards getting rid of ambiguous plain
`usize` usage in the opcode.
Change-Id: I21e35e67d94b32d68251908b96c7f62b6f56a8bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6282
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
grfn pointed out in cl/6174 that `Result` might cause developers to
believe that this behaves like std::Result, which it does not.
Change-Id: Ia30ab0dcb7e8da7bf842777ee3fe17bcf35cb0c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6281
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Refactors the update function to take the attribute sets by value
instead.
To facilitate this, we use an equivalent of the currently unstable
`Rc::clone_or_unwrap` in the VM when encountering attribute sets, so
that in cases where the only references to the attrs being updated are
the ones on the stack those clones are avoided completely.
This does make update() a little bit more tricky internally, as some
optimised branches can directly return the moved value, and others
need to destructure with ownership. For this reason there are now two
different match statements handling the different ownership cases.
Change-Id: Ia77d3ba5c86afb75b9f1f51758bda61729ba5aab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6279
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Add a quick couple of benchmarks for merging attribute sets, large and
small.
Change-Id: I26940a9cf4e0d30e3d9eb07a7b8c366ca4072ca3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6286
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
This is no longer needed for anything and the extra clone here is not
really more costly than constructing a blackhole value in a different
place.
Change-Id: I5c63085b1b4418b629ea58a42e3bfe9a9b586d76
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6275
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Without this escape, it is possible for Nix to produce escaped
representations which are not literal Nix values again.
This was fixed in upstream Nix in
https://github.com/NixOS/nix/pull/4012 (though only for eval, not in
the REPL) and the updated test is picked from upstream after that commit.
Because we run the C++ Nix tests against our test suite as well, this
also bumps our custom Nix 2.3 to a commit that includes the
cherry-picked fix from the PR above.
Change-Id: I478547ade65f655c606ec46f7143932064192283
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6271
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is a more sensible place for this function to live and makes
upvalue resolution easier down the line.
Change-Id: I48ee39bdcdb4f96a16a327f7015aff60db5b15fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6270
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This struct will carry the upvalue machinery in addition to the lambda
itself. For now, all lambdas are wrapped in closures (though
technically analysis of the environment can later remove innermost
Closure wrapper, but this optimisation may not be worth it).
Change-Id: If2b68549ec1ea4ab838fdc47a2181c694ac937f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6269
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Primarily to make sure we build benchmark targets, and avoid breaking
them
Change-Id: I0c43f4cf99ddfd38e7545ef2d8276ef6b240a1e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6285
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Interpret was updated to take an optional path arg in
6fe5e2d75 (feat(tvix/eval): resolve relative path literals, 2022-08-12),
but since benchmarks aren't building in CI the resulting breakage of
benchmarks was missed.
Change-Id: I8a93f1b25ae62e2d032fafc153d91977c6466712
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6284
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The blackhole allocation is not going to be cheaper than cloning this.
Change-Id: Id3ad44812decb4392830be06645e67bb0a982b96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6267
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Compilation of `let`-expressions is going to become a lot more
complicated due to attempts to avoid thunking when encountering
internal references, so this is just being moved out of the way.
Change-Id: Iecfa4b13d14532e21c2540e6561b4235ce29736a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6266
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This is just for dev comfort, it's not going to be useful for the
final version.
Change-Id: I05fdd590097a61085ed641810655d9ddaf8f3511
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6265
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
In conditions where no dynamic identifiers exist in a scope,
inheriting is usually a no-op - *unless* the identifier is not
statically known and the scope has a non-empty `with`-stack.
Change-Id: Iff4138d9cd4c56e844bc574203708dacc11c3f73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6264
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This avoids copying around the value more than needed.
Change-Id: I35949d16dad7fb8f76e0f641eaccf48322144777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6263
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
These do essentially the same, but return different error variants as
upstream Nix considers `throw` to be (sometimes) catchable.
Change-Id: I1a9ea84567d46fb37287dbf3f3f67052f9382cca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6259
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The set of things that can leak out of `builtins` into the global
scope is statically known (it is what Nix 2.3 leaks there,
essentially).
This is a mild change over the previous mechanism, where instead at
the point where the `builtins` set is constructed we "lift" the
globals out of there (if they exist).
This way users will still eventually be able to add additional
builtins, HOWEVER they will not be able to leak them into the global
scope.
Note that upstream Nix technically leaks _all_ builtins into the
global scope using the `__*` prefix, but we are trying to avoid this
in Tvix if it is not required in nixpkgs.
Change-Id: Ie9dec2ce33740134f3b2464eba3749f421dd5953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6258
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Adds a new builtins module in which builtins can be constructed. The
functions in this module should return a correctly structured value to
be passed to the compiler's `globals`.
This is wired up all the way to the compiler with an example
`toString` builtin, available as a global. Note that this does not yet
actually behave like the real toString, which has some differences
from `Display`.
Change-Id: Ibb5f6fbe6207782fdf2434435567fc1bd80039a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6254
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Previously, the tokens that could poison a scope (`true`, `false`,
`null`) had individual fields in the scope to track whether or not
they were poisoned.
This commit sets up new machinery that instead tracks scope poisoning
dynamically using a HashMap, and which makes it possible to introduce
additional tokens to the top-level ("global") scope that are directly
resolved by the compiler by passing a map of runtime values to be
used.
With this solution, the compiler now contains all machinery required
for wiring up builtins resolution.
The set of builtins to be exposed at runtime must, however, be
constructed *outside* of the compiler and passed in. Everything is
prepared for this, but it is not yet wired up (so the only existing
builtins are the ones we already had before).
Note that this technically opens up an optimisation potential when
compiling selection operations, where the attribute set being selected
from is `builtins`. The compiler could directly resolve the builtins
and place the right values on the stack.
Change-Id: Ia7dad3c2a98703e7ea0c6ace1a722d57cc70a65c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6253
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Builtins are represented as a Rust function pointer that accepts a
vector of arguments, which represents variable arity builtins.
Change-Id: Ibab7e662a646caf1172695d876d2f55e187c03dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6251
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Nix functions always have a single argument and we do not yet make
efforts to optimise this in Tvix for known multi-argument functions
being directly applied.
For this reason, the call instruction is fairly simple and just calls
out to construct a new call frame.
Note that the logic for terminating the run loop has moved to the top
of the dispatch; this is because the loop run needs to be skipped if
the call frame for the current lambda has just been dropped.
Change-Id: I259bc07e19c1e55cd0a65207fa8105b23052b967
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6249
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Compiles lambda definitions of the simple form (i.e. without formals
arguments) and emits them as constants like any other value.
This does not yet implement actually invoking these functions in the VM.
Change-Id: Ie1e0a13220b68c1728be229b875f0992e685c5ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6247
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This structure carries context about the lambda currently being
compiled (which may well be the top-level lambda of an input AST).
Using the indirection helpers in the compiler, things like the scope,
code and constants of the function being compiled are now taken from
the current lambda context instead.
Change-Id: If5f864d826c2e72855cee4b728ea1830e9b5ac06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6246
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This wires up most of the machinery for executing different call
frames inside of the VM and stuffs the top-level lambda which the
compiler outputs in there, as well.
Change-Id: Ib6201b3e3be1af96a4d195f6eb147f452860ffc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6242
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This is going to carry the data for a function invocation inside of
the VM.
Change-Id: I86664563a7e35697a64294acd37ffde037fbd32d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6241
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Changes the internal compiler plumbing to not just return a chunk of
code, but the same chunk wrapped inside of a lambda value.
This is one more step towards compiling runtime lambdas.
Change-Id: If0035f8e65a2970c5ae123fc068a2396e1d8fd72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6240
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
With these indirections in place it becomes easier to change internals
of the compiler when introducing functions, which need the compiler to
be able to target different code chunks.
Change-Id: I4eb11572a93c140b1d059ba0a5af905756745d65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6239
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Instead of exiting the compiler at the first sight of an error,
skip any erroneous nodes and continue compiling, collecting more
errors along the way.
This paves the way for nicer error reporting in which multiple errors
can be reported at once, avoiding situations in which users are
hunting a fault error-by-error and possibly getting distracted by
less useful output.
Change-Id: I80c9a87272e33a31297167ae2eb2706a46adf15a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6236
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This starts paving the way for nicer, source-code based error
reporting.
Right now the code paths in the VM do not emit annotated errors, as we
do not yet preserve that structure from the compiler. However, error
emitting code paths in the compiler have been amended to include known
nodes.
Change-Id: I1b74410ffd891c40cd913361bd73c4336ec8aa5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6235
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
In this commit, the string interpolation parsing is identical to
nixpkgs which makes some of the upstream Nix tests for
interpolation-related weirdness pass.
Change-Id: I3a295cfdc404c32228a54846e6efd3c0dcee5842
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6233
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
These were missing an additional level of escaping, silly oversight
caught by an upstream test.
Change-Id: I0312084475e4b88c83945614e9aa5b34c6bc3ec2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6232
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Since the latest published version of rnix-parser on crates.io, the
crate has undergone major changes which are only available in the git
repository at the moment. This commit updates the compiler to this
newer version of rnix.
Most notably, the entire AST provided by rnix is now wrapped in the
AST type system. As a result of this traversal is much nicer in many
places, especially for things like nested attribute selection.
There are a handful of smaller features missing for full feature
parity with the previous version, especially handling of path
literals, but PRs for these already exist in rnix-parser.
Change-Id: Icde6d393067976549492b7d89c4cc49e5e575fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6231
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This implements `assert`, which evaluates an expression and aborts
evaluation if the value is not `true`.
At this point we should introduce eval-failed-* tests; probably
asserting against some representation of the error enum?
Change-Id: If54c8f616d89b829c1860a4835dde60a2cd70d7a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6230
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This makes for much more readable output especially when long strings
are involved.
Change-Id: I43dd73a0480535d7181a760788c42883a9b083f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6229
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
`push_local`/`push_phantom` were worse names because they sound like
the value itself is being pushed, where in actuality it is just being
declared to the compiler.
Change-Id: Ibfda5c4c8e47d5d3262bfe005b0f1f84908a117e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6228
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
These tokens are optionally parsed as identifiers by Nix, which means
that within any scopes that resolve them the compiler needs to track
whether they have been overridden to know whether to emit the literal
instructions or resolve a variable.
This is implemented by a new concept of "scope poisoning", where the
compiler's scope structure tracks whether or not any builtin
identifiers have been overridden.
Change-Id: I3ab711146e229f843f6e1f0343385382ee0aecb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6227
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
While full recursion through thunking is not available, there are
actually incorrect behaviours introduced by declaring before
binding (example in the newly introduced test).
This commit simplifies the implementation to avoid this issue, and
also because I intend to explore a bit more how far we can get in non
left-to-right bindings *without* introducing thunks immediately.
Change-Id: I21fd3007ac3946570639772d7d624d70bd209958
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6226
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
If an unknown variable is encountered and the with stack is not empty,
emit instructions for resolving the variable at runtime.
Change-Id: I752f4bd0025335744e4747364abd1bd34130374e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6223
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Adds an additional structure to the compiler's scope to track the
runtime "with stack", i.e. the stack of values through which
identifiers should be dynamically resolved within a with-scope.
When encountering a `with` expression, the value from which the
bindings should be resolved is pushed onto the stack and tracked by
the compiler in the "with stack", as well as with a "phantom value"
which indicates that the stack contains an additional slot which is
not available to users via identifiers.
Runtime handling of this is not yet implemented.
Change-Id: I5e96fb55b6378e8e2a59c20c8518caa6df83da1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6217
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This name is much more sensible actually; its more than just a
collection of locals as it tracks additional scope information in the
case of Nix.
Change-Id: Ia2739bbd39aab222b1c4355e9248828973b0db43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6216
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Note that at this point recursive bindings do not yet work in either
attrsets or let, so inheriting from the same scope is generally not
possible yet.
Change-Id: I6ca820d04b8ded5c22fb7ea18e2ec203bcaa8e9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6215
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Straightforward implementation, evaluating the elements of an inherit
and preparing the stack so that `OpAttrs` sees all relevant values
when constructing the attribute set itself.
The emitted instructions for inheriting a lot of values from the same
attribute set are inefficient, but it's too early to say whether this
actually matters.
Change-Id: Icb55a20936d4ef77173f34433811c5fa5d2c9ecc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6214
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
If the directory in which REPL history is stored does not exist,
gently try to create it, but do not raise an error if it doesn't work.
We may want to warn about it, but in general this sort of
non-essential feature should not cause a hard failure.
Change-Id: If4fe8db0c7893c39627efe72c9cd9ebf7ed63f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6213
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Using `OpAttrSelect`, the ? operator will fail when encountering a
nested value that is not an attribute set.
This however breaks valid code, such as:
{ bs = 42; } ? bs.a.b
The fix is simply to use the same operator used in the `or` statement,
which leaves a sentinal on the stack if a field is not found or the
value is not an attribute set.
Change-Id: Ib28fc8a96e6d592b4cdbc3e65ba129ad8faecd66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6211
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
If a nested attrpath encounters a non-set value, the sentinel value
denoting a lack of next values should be emitted. This mirrors the
behaviour of Nix.
Change-Id: Ia80443d5a11243cc6d98dcab1249a3f5fdf77e27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6210
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Not sure how exactly this snuck in, but it caused some subtle
breakages in deeply nested attribute sets.
Change-Id: I8049ce912405d3750031f79cc8d86ff1c3c02c2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6208
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
We're confident that we're handling all branches that can reasonably
occur from valid AST, any other cases should be considered a critical
evaluator bug and panic rather than surfacing something that looks
like user error.
Change-Id: If96966eb32b8ff12fcaeb9ea3b0c8fc51b6abd11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6205
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This is actually *tested* behaviour in C++ Nix, so we need to
implement it here, too.
Change-Id: Ic4a4659a2f04cdd928cbe78a85dae90401515371
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6199
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This adds a `disassembler` feature to the crate configuration that
traces the operations executed and the state of the stack at runtime.
This can be enabled by compiling with `--feature disassembler`.
This will also gain a more sensible layout of code slices eventually.
Change-Id: I34c15e1cd346ecc4362b5afba6bf82dd49359d20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6193
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This exposes tvix.nix_cli as a proper attribute to readTree, so it's
actually built by CI.
Change-Id: I3cef085bd872b61c5944270c8926727bf1fa705d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6083
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Using `inherit` in a let-binding can not possibly have an effect, as
the given identifier is already bound exactly the same way in the
current scope.
This introduces a subtle bug that is fixed later on, as there
actually *is* a (single) condition where these inherits are
meaningful.
Change-Id: I8b24f0edcfe80db8153bb7e86cf478d36957d6f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6192
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This makes basic `let ... in ...` statements work correctly. It does
not yet account for the call frames pushed into the VM during function
application.
Change-Id: I67155171daf1a43011b96716dd9d1ab04b27db33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6190
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Nix does not allow dynamic identifiers in let expressions (only in
attribute sets), but there are several different kinds of things it
considers static identifiers.
The functions introduced here put the path components of a let
expression into normalised (string) form, and surface an error about
dynamic keys if one is encountered.
Change-Id: Ia3ebd95c6f3ed3cd33b94e156930d2e9c39b6cbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6189
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
These expressions now leave the binding values on the stack, and clean
up the scope after the body of the expression.
While variable access is not yet implemented (as the identifier node
remains unhandled), this already gives us the correct stack behaviour.
Change-Id: I138c20ace9c64502c94b2c0f99a6077cd912c00d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6188
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
These are going to be used during compilation of `let`-expressions to
determine stack offsets for local variables.
Change-Id: Ibb79f3f1ae86650303f88eacf623ae456458de87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6187
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Canonicalisation performs much more functionality than what C++ Nix
implements for paths, and causes some undesirable behaviour (e.g.
handling non-existant files becomes difficult, but should be possible
in literals).
Instead, the path_clean crate provides a pure normalisation method.
There is an intention to add this to Rust itself:
https://github.com/rust-lang/rfcs/issues/2208
Change-Id: I775d238136db0a52cf6b12a68985833c8fb32882
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6186
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Resolves relative paths (e.g. `./foo`) either relative to the location
of the Nix file, or relative to the working directory if none is
supplied.
Change-Id: I70ec574657b221b458015117a004b6e4a9c25a30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6185
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
There are multiple things that can theoretically fail while resolving
a path, as some of it includes I/O. A new error variant has been added
for this and appropriate errors have been introduced.
Change-Id: Ie222245425207dabbf203166eb5ed1eec0114483
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6184
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
These two paths are the easiest to handle, as they are simply built up
from the components supplied in the text node and then normalised.
Note that the normalisation of fs::canonicalize includes symlink
resolution, which Nix does not actually do. We will need to fix that
at some point.
Change-Id: I54158f0684f197dd2a2583f7d0982d54c7619993
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6183
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
grfn pointed out in cl/6082 that this is actually the desugaring of
the write! macro, so it doesn't make sense to write it out.
Change-Id: If7c055b042ad22b034722aec1eaadba92736d684
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6180
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
grfn pointed out in cl/6069 that naming them like this makes it clear
that things are being added to the end of the state.
Change-Id: I6a23215c4fef713869a3c85b0dde1ebbda7637e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6179
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
For representation wrappers that are used to control the visibility of
type internals, this ensures that the wrapper does not increase the
size of the type.
In practice, the optimiser likely does this anyways but it is good to
guarantee it.
Change-Id: Ic6df7d668fe6006dfbd5b6cfcfc2088afa95b810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6178
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This makes it possible to quickly detect code errors that might blow
up the size of the OpCode type.
Change-Id: I7662dd0aa30c4762c0f9e4fa346418c9ca8b9994
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6177
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
These can be used predominantly to emit warnings about things that the
compiler can infer, such as deprecated language features.
Change-Id: I3649c625459d7f3f95cdf42d5c651d23d66569ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6174
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
The parser creates this node type from literal URL values. Technically
these are deprecated and have been removed from nixpkgs.
Change-Id: I4d05034dd9b4d8348e4ed8a2bbb37c1b6ccef8bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6173
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is a substantially nicer experience, immediately granting us
history, proper exiting and so on.
Change-Id: Iba4cb1713b9ac53d0799722bdbe2cd0e94a2f527
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6171
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This crate provides sensible default locations for directories on
different platforms.
Change-Id: I0b61cc7f626dc6c8df903ba0f873be24e07d69b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6170
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
There's a few options for this, but this one seems fine in terms of
features and doesn't have an insane amount of dependencies.
Change-Id: Ief99e66bfee0ba0ba1cfdd09568b002121b7325b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6169
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This operator allows for accessing attribute sets (including nested
access) while also providing a default value.
This is one of the more complex operations to compile, as it needs to
keep track of a fairly large number of jumps that all need to be
patched correctly.
To make this easier to understand there's a small diagram included in
the comments.
Change-Id: Ia53bb20d8f779859bfd1692fa3f6d72af74c3a1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6167
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This sentinel value is going to be used for certain nested accesses
into attribute sets.
There is a new instruction similar to `OpAttrsSelect` which leaves the
sentinel on the stack if a key is not found, instead of raising an
error.
Additionally, a new jump instruction makes its jump operation
conditional on finding such a sentinel value.
Change-Id: I2642f0a0bcc85bbe0ead68ea09a7dd794dbedeac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6166
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The only uses of the static variant were for `"name"` and `"value"`,
which are both small enough to fit into a SmolStr. The size of
NixString accomodates `String` anyways, so we may as well inline them.
Additionally smol_str is already in the dependency graph because rnix
uses it, and using it for representations of identifiers is sensible.
Change-Id: I9969312256d1657d69128e54c47dc7294a18ce58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6165
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Having these visible more explicitly is useful while debugging.
Change-Id: I86b497883063d32792b635eb4514b7aeae484af4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6164
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Implements the nested presence check operator for attribuet sets by
traversing the chain of lookups through instructions that push/pop
sequentially deeper attribute sets onto the stack.
Note that this commit introduces a bug in case of nested attributes
not being found, which is fixed in a later commit.
Change-Id: Ic8b4c8648736f6cb048e3aa52592e4d075bf0544
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6163
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
This makes it possible to check things like `{} ? a` with a single
level of nesting.
Change-Id: I567c36fcfd2f9e2f60071acd3ebfe56dea59b26f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6161
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
With this change, attribute set access is working as intended.
Change-Id: Ic5dbbd68aa59156106069289e7375a696909f78b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6159
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Fairly straightforward, handling the optimised representations
manually and otherwise delegating to BTreeMap.
Note that parsing of raw identifiers is not yet implemented.
Encountering an identifier node usually means that there is locals
access going on, so we need a special case for compiling a node in
such a way that an identifier's literal value ends up on the stack.
Change-Id: I13fbab7ac657b17ef3f4c5859fe737c321890c8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6158
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
This operation is required because both sides of the logical operators
are strictly evaluated by Nix, even if the resulting value is not used
further.
For example, in our implementation of `&&`, if the left-hand side is
`true`, then the result of the expression is simply the right-hand
side value. This value must be asserted to be a boolean for the
semantics of the language to work correctly.
Change-Id: I34f5364f2a444753fa1d8b0a1a2b2d9cdf7c6700
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6157
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Bootstrap some (initially very simple, mostly proof-of-concept)
benchmarking infrastructure using Criterion, using the newly-exposed lib
from tvix-eval.
Change-Id: I4bb93c142ba8d018d7e67e58ac8907a0429398a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6156
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Add a new `lib.rs` to tvix/eval, which `pub use`s the `interpret`
function, and all types mentioned in its return type, and then uses
*this* instead of direct `mod` statements in the `main.rs` to implement
the entrypoints to the interpreter. This is in preparation for calling
these functions from integrated benchmarking infrastructure using
Criterion, though other things (like integration tests) might want to do
that as well.
Change-Id: I7b585134a96b1c56a2ac64d2036b0e51d321bd27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6155
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Similar to `||`, but inverting the left-hand side.
In other words, `a -> b` is essentially rewritten as `!a || b`.
Change-Id: I8a62da65ff070b389e46048d047a54279060a97b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6152
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
This logical operator is implemented in terms of jumping operations
and thus requires slightly different treatment than other binary
operators.
Change-Id: Ib3d768b70dd7e16014c9b47d770aa74eec60ae92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6150
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
These expressions use simple jumps to skip the correct expression
conditionally in the bytecode by advancing the instruction pointer.
Note that these expressions are already covered by a test behind the
`nix_tests` feature flag, but adding more is probably sensible.
Change-Id: Ibe0eba95d216321c883d3b6b5816e2ab6fe7eef1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6148
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Wraps the string representation in an additional newtype struct with a
private field in order to hide the representation from other modules.
This is done in order to avoid accidental leakage of the internals
outside of value::string.
In fact, this caught a mistake in the compiler module which was
directly constructing an internal variant.
Change-Id: If4b627d3cff7ab9cd50ca1a3ac73245d4dcf7aef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6147
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Wraps the attrs representation in an additional newtype struct with a
private field in order to hide the representation from other modules.
This is done in order to avoid accidental leakage of the internals
outside of value::attrs.
Change-Id: I68d1d02514aa0443df4c39801001a3f1f6cc5d5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6146
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Ensuring that the implementation is not leaking out of the module lets
us keep things open for optimisations (e.g. empty list or pairs
through tuples).
Change-Id: I18fd9b7740f28c55736471e16c6b4095a05dd6d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6145
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is accomplished by simply delegating to the Rust implementations
of (Partial)Ord and (Partial)Eq, which are implemented for Value and
underlying wrapper types to behave like they do in Nix.
To ease the implementation overhead, a new comparison operator macro
has been added to the VM module.
Incomparable types will raise a new error variant when a comparison is
attempted, containing both supplied types. This mimics the information
carried in the error thrown by C++ Nix.
Change-Id: Ia19634d69119d40722f3ca672387bc3a80096998
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6143
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This can now fully delegate to PartialEq of Value
Change-Id: Iaa9f4ec9b8830d516d72f83a93ab2df9a6e5697c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6142
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The underlying implementation does a few tricks based on which pair of
attrset representations is encountered.
Particularly the effect of short-circuiting the empty cases might be
relevant in nixpkgs/NixOS, due to the use of lib.optionalAttrs.
Change-Id: I22b978b1c69af12926489a71087c6a6219c012f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6140
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
I assumed that `Ord` is a marker trait like `Eq`, but it actually has
a member. Without this ordering was incoherent.
Change-Id: Id37cbdf333daf748d29b85243046c7e061b1ce29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6139
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
