Also expose both formats, then use it from
users/tazjin/presentations/tvix-eval-2023.
Change-Id: Id906e8aff5510a7a4f33336326472e86db18ea32
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9280
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Previously, compare_strict_eval_tests() was using Strictness::Lazy.
This appears to be a minor oversight from
0ab6494286.
This commit corrects that, by changing Strictness::Lazy to
Strictness::Strict.
Change-Id: Ia2389a5d30481cd322ed55230731340b795e5d87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9282
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
This target builds and runs the go tests in that directory, allowing us
to spot regressions as the one fixed in cl/9285.
Change-Id: Ia16c0622f29db343eae7c0386e715b292703bd4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9286
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
The underlying protos were changed to return `[]byte` instead of `string`.
Change-Id: I5d3e5d8de0ed7200325f7ab0d62e3c10d8eb1b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9285
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Included changes:
* tvix/eval: enable some lang tests on nix_latest
Nix 2.16 contains some breaking language changes which Tvix does
not yet implement, but the existing tests for them are now passed by
Nix 2.16 (but not yet by Tvix).
* tvix/eval: disable a lang test on nix_latest
In Nix 2.17, the identifier formatting test fails because some
behaviour changed. We have not investigated further yet.
* 3p/overlays: use version of ihp-hsx that works with GHC 9.4
Originally from the separate cl/9185.
* top-level: introduce a mechanism to exclude build targets from CI in
the top level. This fixes b/296.
* users/grfn: disable builds of xanthous (and dependents) until the
CLs fixing its build are submitted
* 3p/overlays: build nixos-option against Nix 2.15, the only version
with which it builds
* 3p/overlays: bump tdlib to 1.8.16
Change-Id: Ia377f39dbdb08ac45ff830a615e64babc091e5ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9125
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This makes the inside code a bit less verbose.
I wasn't able to describe the type of the async move closure itself,
which would allow us to remove the JoinHandle<_> type annotation
entirely.
Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This enables the tokio `signal` feature, and registers a ctrl_c signal
handler, which will use the unmount handle to unmount in case a ctrl-c
signal is received.
This avoids having disconnected mountpoints when Ctrl-C'ing a
`tvix-store mount` invocation.
In case the filesystem is unmounted externally (via `umount /path/to/
mountpoint`), the future is waiting for the signal is never resolved and
the task is stopped.
Change-Id: I149f705a6cb50188177f2a6c6a5fcd77218e2a3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9218
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This provides an additional configuration flag to the tvix-store mount
subcommand, and logic in the fuse module to request listing for the
root of the mountpoint.
Change-Id: I05a8bc11f7991b574696f27a30afe0f4e718a58c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: adisbladis <adisbladis@gmail.com>
Tested-by: BuildkiteCI
This provides an additional method in the PathInfoService trait, as
well as an RPC method on the gRPC layer to list all PathInfo objects in
a PathInfoService.
Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We don't need to explicitly describe the type of the task itself,
describing the return type of the async closure is sufficient.
Also, use io::Result<_> instead of Result<_, io::Error>.
Change-Id: I9ab3f990eb49929b0aea335b2bb07da392ab631f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9267
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
https://github.com/BurntSushi/walkdir/pull/170 got merged, meaning we
don't need to keep our own logic in here anymore.
Our test cases already cover this.
Change-Id: Ied3043ee651c8aafa10271c1e1ca5d460fb6c0b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9269
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This trait is eval-specific, there's no point in dealing with these
things in tvix-store.
This implements the EvalIO interface for a Tvix store.
The proper place for this glue code (for now) is tvix-cli, which knows
about both tvix-store and tvix-eval.
There's one annoyance with this move: The `tvix-store import` subcommand
previously also used the TvixStoreIO implementation (because it
conveniently did what we wanted).
Some of this code had to be duplicated, mostly logic to calculate the
NAR-based output path and create the PathInfo object.
Some, but potentially more of this can be extracted into helper
functions in a shared crate, and then be used from both TvixStoreIO in
tvix-cli as well as the tvix-store CLI entrypoint.
Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Don't repeat the name of the method in the description, don't repeat
things already described in request message comments.
Change-Id: I180e4792577419050947eea8fea7043861aba463
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9213
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This returns a node with a new name.
Change-Id: Iebcab537f8dd63d826b9841d4d0181fcb941afdd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Since the refactor to use URIs for all three services, this actually
does talk to a daemon by default.
Change-Id: Ied296772b77eef514bfcae0a9dfc50f848a1c2f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9210
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
replaceStrings would previously fail to replace the last character
in a string.
Change-Id: I43a7c960945350b2e7a5b731b7fdb617723eb38f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9151
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
We already have these hashes accessible in the Cargo.nix file created
by cargo2nix, so there's no need to also manually maintain them here.
It removes one potential footgun I ran into while updating wu-manber to
a different rev, without updating it here.
Change-Id: I93932ac8ba55f83746ee38571b7740af2d49bbdf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9090
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
There was a NixHash::new() before, which didn't perform any validation
of the digest length. We had some length validation when parsing nix
hashes or SRI hashes, but some places didn't perform validation and/or
constructed the struct directly.
Replace NixHash::new() with a
`impl TryFrom<(HashAlgo, Vec<u8>)> for NixHash`, which does do this
validation, and update constructing code to use that, rather than
populating structs directly. In some rare cases where we're sure the
digest length is correct we still populate the struct manually.
Fixes b/291.
Change-Id: I7a323c5b18d94de0ec15e391b3e7586df42f4229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9109
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This question popped up every once in a while. While already explained
quite well at
https://inbox.tvl.su/depot/20230316120039.j4fkp3puzrtbjcpi@tp/T/#t,
it's not easily accessible.
Lift it from there into tvix/docs for better visibility.
Change-Id: I5f2d4aff31ab4adc421e06a7d36c871f45e09100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9080
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This passes a unit value to the function.
Change-Id: I4df3ad8fb0f35c0f110cee3349971ae28ce2878c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9101
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We already have the parsed output_hash from above, no need to construct
it again.
Change-Id: Ie6d924ab446137c25c29fbeaf671aa7e5418262d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9110
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Previously, Output deserialization would silence validation errors and
provide `None` for `hash_with_mode` as soon as a validation error would
happen inside of the `NixHashWithMode` deserialization, e.g. invalid
hash length would not provide an validation error but a silent `None`
value.
This is problematic, we workaround a serde limitation here by writing
our own Deserializer.
As you can see, we write some boilerplate code unfortunately, as, for
example:
- `#[serde(fail_if_unparsed_as="Option::is_none")]` is not a thing,
otherwise, we could have been able to just bubble up errors in case of
"not fully parsed" (and not missing) values.
- `From<&serde_json::Value> for serde:🇩🇪:Unexpected` is not a thing,
otherwise, we could just map invalid type errors and reuse the
existing types instead of doing extremely bizarre things with
`serde:🇩🇪:Unexpected::Other`, note: this is a not problem for
expected, we know what we expect, we don't know what we received in
practice.
I decided to write a `NixHashWithMode::from_map` which will eat a map
deserialized via `serde_json`, so our serde magic is not totally "data
model" agnostic.
I wanted to go for data model agnosticity and enable maximal
performance, e.g. building the structure as the map values are streamed
in the Visitor, this is needlessly painful because `Output` and
`NixHashWithMode` are in different files and this really makes sense
only if we write the full implementation in one file, indeed, otherwise,
you end up duplicating the work or having strange coupling.
So, for now, we will allocate a full map of the fields inside the
`Output`, i.e. if any "unknown field" is in that map, we will
deserialize it for no reason.
Doing it properly, as I repeat it in the code and to flokli at C3Camp
2023, requires to patch serde upstream IMHO.
Change-Id: I46fe6ccb8c390c48d6934fd3e3f02a0dfe59557b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9107
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't need rust-analyzer to run cargo doc.
Change-Id: I5e2fd559e4045cadeab24b438c28d6df7f1d5d5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9092
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
cl/8306 fixed building all docs, but we forgot to update the comment.
Change-Id: I17829612f13e7357bd0efe8223cc28ed0f6cdea2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9091
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This `.into_iter()` call is equivalent to `.iter()` and will not consume
the `BTreeMap`.
Change-Id: Ie26637ebecb0bea5b09c447cc45ed207f8b50913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9088
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We only need bstr::ByteSlice to be able to use replace, it doesn't need
to return a BString.
Change-Id: I811948436fb89652e880970c2c05356183f3e439
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9084
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
builtins.div ought to truncate towards zero so that
-(builtins.div a b) == builtins.div (-a) b
-(builtins.div a b) == builtins.div a (-b)
Change-Id: I8b7c08cd7f4fa8a1363c786d42c8d484f6cd133d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9006
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.
I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.
Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Make it more obvious if these are bytes pointing to JSON or ATerm, so we
don't get confused.
Change-Id: I2402c687b7ba9c05aac20ed63b0df54e4e96a9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8998
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This is more concise than a io::copy of a Cursor to bytes, and we have
everything to be written in memory.
Change-Id: I81f34666aa61aef4e16b33423ce4a69c3781efc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8997
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
write_input_derivations shouldn't need to write a comma to separate it
from the previous output from write_outputs.
This is better placed in the function calling all of these helper
functions.
Change-Id: I9ccc440e4665b52369ef39e75151b9a29469ce48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8995
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We're happy with any &[S], as long as <S: AsRef<[u8]>.
This allows passing both strings and &[u8].
Change-Id: If2a80d9b1ee33ba328c9cdab4fa83ca7b98a71e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8994
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Let the escape function only take care of string escaping, not quoting.
Let write_array_elements always quote and escape strings it consumes.
Move the business of writing additional wrapping characters around it to
the caller.
Change-Id: Ib8dea69c409561b49862c531ba5a3fe6c2f061f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8993
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Derivations can have non-unicode strings in their env values, so the
ATerm representations are not necessarily String anymore, but Vec<u8>.
Change-Id: Ic23839471eb7f68d9c3c30667c878830946b6607
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8990
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This allows sorting Store Paths. We delegate the sorting business to the
PartialOrd, Ord impls for our digest fields only, as two StorePaths with
the same digest, but different names can't exist.
Change-Id: I5f81631e5f5063893b316c63a240c5266b7e5bad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8988
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Add the position in the string where the name is problematic.
Change-Id: If6fd8be6100b718f8d68568eafc77ebb3cfb82d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8979
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This will save us some copies, because a clone will simply create an
additional pointer to the same data.
Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Some paths might use names that are not valid UTF-8. We should be able
to represent them.
We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.
While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?
Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This "reverts" commit 9f600de226 (the
initial revert of f5e291cf83).
Now with BlobService returning a BlobReader that implements io::Seek, we
can actually just call blob_reader.seek(io::SeekFrom::Start(offset as
u64)).
This means, we currently will fail to seek backwards inside a file.
Change-Id: I9c19448df6831a3537252f99210374f2126ecfc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8886
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
For memory and sled, it's trivial, as we already have a Cursor<Vec<u8>>.
For gRPC, we simply reject going backwards, and skip n bytes for now.
Once the gRPC protocol gets support for offsets and verified streaming,
this can be improved.
Change-Id: I734066a514aed287ea3db64bfb1680911ac1eeb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8885
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The primary constructor for this is now from_bytes, from_string is
simply calling .as_bytes() on the string, passing it along.
The InvalidName error now contains a Vec<u8>, to encode the invalid name
(which might not be a string anymore).
from_absolute_path now accepts a &[u8] (even though we might want to
make this a OSString of some sort).
StorePath::validate_name has been degraded to a pub(crate) function.
It's still used in src/derivation, even though it probably shouldn't at
all - that cleanup is left for cl/8412 though.
Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This being a nested error makes things more complicated than necessary.
Also, this caused BuildStorePathError to only hold NameError,
so refactor these utility functions to either return Error, or
BuildStorePathError.
Change-Id: I046fb403780cc5135df8b8833a291fc2a90fd913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8972
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This explicitly documents behavior of C++ Nix that goes against the
intuition you'd gather from this document: that e.g. a simple select
from an attribute set causes a value to no longer be pointer equal to
its former self.
The point of documenting this is that we can show in a to be written
section on the use of pointer equality in nixpkgs that pointer equality
is only needed in a limited sense for evaluating it (C++ Nix's exterior
pointer equality). Tvix's pointer equality is far more powerful since
value identity preserving operations also preserve pointer equality,
generally speaking (this is because we implement interior pointer
equality in my made up terminology). This should eventually also be
documented.
Change-Id: I6ce7ef2d67b012f5ebc92f9e81bba33fb9dce7d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8856
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
This fixes a subtle issue which would occasionally lead to a crash (e.g.
when evaluating (pkgs.systemd.outPath with --trace-runtime): With each
character in the string that has a multi byte representation in UTF-8,
the actual byte position and what tvix thought it was would get out of
sync. This could either lead to
* Tvix swallowing characters or jumbling characters if multi byte
characters would cause the tracked index to become out of sync with
the byte position before the first character to be escaped, or
* Tvix crashing if (in the same situation) the out of sync index would
be within a UTF-8 byte sequence.
Luckily, std's `char_indices()` iterator implements exactly what
`nix_escape_char()`'s original author had in mind with
`.chars().enumerate()`. Using `i + 1` for continuing is safe, since all
characters that need (in fact, can) to be escaped in Nix are represented
as a single byte in UTF-8.
Change-Id: I1c836f70cde3d72db1c644e9112852f0d824715e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8952
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Improve change some little things I noticed while reading through it.
Change-Id: I033209eece395e5aad4e10825e8dd6c0cfe68191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8725
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
I want to expand on the C++ Nix behavior, since it seems relevant to
note that a lot of operations in C++ Nix (like select) don't preserve
pointer equality (see
<https://github.com/NixOS/nix/issues/3371#issuecomment-1596167957>).
It is especially so, as Tvix establishes pointer equality in a different
way and thus shows differing behavior. Therefore I want to additionally
document Tvix's current behavior and make it more explicit to what
extent nixpkgs needs pointer equality.
Change-Id: I9b4ba75dacb749c9fcbba4b9646c6b48bb57bbad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8852
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This reverts commit f5e291cf83.
The offsets are relative to the start of the file, and as long as we
don't have BlobReaders implement seek, this will be very annoying to
deal with.
Change-Id: I05968f7c5c0ec0000597da90f451d6bb650c3e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8882
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
buf contains everything written so far, whereas b is the slice passed in
the current write() call. If we copy from &buf, we end up with the wrong
hash, because we keep writing the wrong data to the hash function.
Change-Id: I768d4645934a6a7d75b9c8eeba35f8f3be5edd26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8880
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This moves from stateless I/O to actually dealing with file handles,
allowing the filesystem to keep reusing existing blobreaders, instead of
opening a new reader on every read() call.
Change-Id: I3fc35c071e4aee1021c8bbd58749d082b0abd188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8834
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
The change allows applications that use tvix_serde for parsing
nix-based configuration to extend the language with domain-specific
set of features.
Change-Id: Ia86612308a167c456ecf03e93fe0fbae55b876a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8848
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I've noticed this behavior when writing the admittedly cursed test case
included in this CL. Alternatively we could use some sort of machinery
using `builtins.trace`, but I don't think we capture stderr anywhere.
I've elected to put this into the eval cache itself while C++ Nix does
it in builtins.import already, namely via `realisePath`. We don't have
an equivalent for this yet, since we don't support any kind of IfD, but
we could revise that later. In any case, it seems good to encapsulate
`ImportCache` in this way, as it'll also allow using file hashes as
identifiers, for example.
C++ Nix also does our equivalent of canon_path in `builtins.import`
which we still don't, but I suspect it hardly makes a difference.
Change-Id: I05004737ca2458a4c67359d9e7d9a2f2154a0a0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8839
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is a first implementation of a FUSE filesystem, mounting tvix-store
to a given location.
This is mostly meant as one additional lens into a store, and could be
used for builds. It's not meant to be used as a general-purpose thing.
It still has some rough edges:
- It doesn't implement open/close, so it doesn't use file handles.
Which means, we need to open blobs for partial reads over and over
again.
- It doesn't implement seek, as BlobReader doesn't implement seek yet.
- It doesn't track "lifetimes" of inodes by listening on forget,
meaning it might hold more data in memory than necessary.
- As we don't have store composition (and a caching layer) yet,
operations might be slow.
Change-Id: Ib1812ed761dfaf6aeb548443ae939c87530b7be8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8667
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
When dealing with a formal argument in a function argument pattern that
has a default expression, there are two different things that can happen
at runtime: Either we select its value from the passed attribute
successfully or we need to use the default expression. Both of these may
be thunks and both of these may need finalisers. However, in the former
case this is taken care of elsewhere, the value will always be finalised
already if necessary. In the latter case we may need to finalise the
thunk resulting from the default expression. However, the thunk
corresponding to the expression may never end up in the local's stack
slot. Since finalisation goes by stack slot (and not constants), we need
to prevent a case where we don't fall back to the default expression,
but finalise anyways.
Previously, we worked around this by making `OpFinalise` ignore
non-thunks. Since finalisation of already evaluated thunks still
crashed, the faulty compilation of function pattern arguments could
still cause a crash.
As a new approach, we reinstate the old behavior of `OpFinalise` to
crash whenever encountering something that is either not a thunk or
doesn't need finalisation. This can also help catching (similar)
miscompilations in the future. To then prevent the crash, we need to
track whether we have fallen back or not at runtime. This is done using
an additional phantom on the stack that holds a new `FinaliseRequest`
value. When it comes to finalisation we check this value and
conditionally execute `OpFinalise` based on its value.
Resolves b/261 and b/265 (partially).
Change-Id: Ic04fb80ec671a2ba11fa645090769c335fb7f58b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8705
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
This adds a `from_str_with_config` function which takes a
user-supplied closure that sets additional settings on the
`tvix_eval::Evaluation`.
Note that users can not set `strict = false`, but other settings are
not restricted.
This solves b/262.
Change-Id: Ice184400b843cfbcaa5b6fe251ced12b6815e085
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8808
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Adds new tests for foldl', intersectAttrs as well as fills in missing
.exp files.
New test cases we don't pass:
- fromTOML timestamp capabilities
- path antiquotation
- replaceStrings is lazier on C++ Nix master
The C++ Nix revision used is 7066d21a0ddb421967980094222c4bc1f5a0f45a.
Change-Id: Ic619c96e2d41e6c5ea6fa93f9402b12e564af3c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8778
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
genericClosure has very limited support for pointer equality: It relies
on comparison (not equality!) in C++ Nix, so as soon as C++ Nix supports
comparing lists (langVersion >= 6) we can rely on pointer equality for
key.
Since Tvix uses equality, not comparison for the insert, our behavior is
currently different, as documented by the notyetpassing tests.
Change-Id: Ifcd741ed4fc3ccc3825f7038875d56a9918b786a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8720
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
In order for the test suite we have currently to be comparable to C++
Nix, we need to display values in the same way. This was largely the
case except in some weird cases.
* <CODE> for thunks and <CYCLE> for repeated thunks (?) are already in
use. <CODE> formatting is tested by the oracle test suite already.
* Instead of lambda, we need to use <LAMBDA>
* <<primop>> and <<primop-app>> (a formatting C++ Nix uses nowhere)
now are <PRIMOP> and <PRIMOP-APP>.
We'll probably want to have a fancier display of values (in a separate
trait) down the line. This could be used for interactive usage, e.g. the
REPL or a potential debugger.
There is a peculiarity with C++ Nix 2.3 formatting primops: import is
considered a <<PRIMOP-APP>>, since it is internally implemented by means
of scopedImport. This implementation detail no longer leaks in C++ Nix
2.13 nor in Tvix.
<CYCLE> display is untested at the moment, since we exhibit a
discrepancy to C++ Nix 2.3. Our current detection is more similar to C++
Nix 2.13—luckily it is also the more consistent of the two. See also
b/245.
Change-Id: I1d534434b02e470bf5475b3758920ea81e3420dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8760
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
A previous iteration of this code did actually connect (in the gRPC
client), which was why we had this function async.
However, as the connection there is now lazy too, we can drop the
asyncness in this function.
Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This allows constructing blob stores with a URL syntax at runtime,
by passing the --blob-service-addr arg.
We probably still want to have some builder pattern here, to allow
additional schemes to be registered.
Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
These were added by us in r/5276, so they should go into our test suite.
Change-Id: I6dc74fc242f33c22a17e0b4aee546ccae886ac85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
C++ Nix resolves home relative paths at [parse] time. This is not an
option for us, since it prevents being able to separate the compilation
and execution phase later (e.g. precompiled nix expressions). However, a
practical consequence of this is that paths expressions are always
literals (strict) and never thunks.
[parse]: 7066d21a0d/src/libexpr/parser.y (L518-L527)
Change-Id: Ie4b9dc68f62c86d6c7fd5f1c9460c850d97ed1ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7041
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
There's very little reason to instantiate a GRPCPathInfoService in a
context where we are not already in a tokio context.
Change-Id: Ib81d649387717cb98de8a8039f92472f727b10c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8755
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The only place where we did use new was also already where we've been in
a tokio context, so just using from_client is easier.
Change-Id: I39dbc18f6aaa3abc342409be623395647f968530
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8754
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
There's very little reason to instantiate a GRPCBlobService in a context
where we are not already in a tokio context.
Change-Id: Ic6e18809a9f2a76f1c098ed330118d8dcfba5137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8753
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This removes the use of generics, like previously done with Blob and
Directory services.
Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
We never returned Err here anyways, and we can still return an error
during the first (or subsequent) write(s).
Change-Id: I4b4cd3d35f6ea008e9ffe2f7b71bfc9187309e2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8750
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
From 64 bytes to 100 KBytes.
We need to provide a custom wrapper with a different Default instance.
Change-Id: Id7c6c437b8183b355a9e388f98cef1622b363f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This allows us to blob services without closing them before putting them
in a box.
We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers
require Sync.
Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Once we support configuring services at runtime, we don't know what
DirectoryService we're using at compile time.
This also means, we can't explicitly use the is_closed method from
GRPCPutter, without making it part of the DirectoryPutter itself.
Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Putting this in the PathInfoService trait makes much more sense, we can
have direct control over where/how to cache the results in the
implementation.
This now requires each PathInfoService to hold pointers to BlobService
and DirectoryService.
Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
There's only one way to calculate NAR files, by walking through them.
Things like caching such replies should be done closer to where we use
these, composing NARCalculationService doesn't actually give us much.
Instead, expose two functions, `nar::calculate_size_and_sha256` and
`nar::writer_nar`, the latter writing NAR to a writer, the former using
write_nar to only keeping the NAR size and digest.
Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
To construct various stores at runtime, we need to eliminate associated
types from the BlobService trait, and return Box<dyn …> instead of
specific types.
This also means we can't consume self in the close() method, so
everything we write to is put in an Option<>, and during the first close
we take from there.
Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Unfortunately, nixpkgs has at least one case[1] where the out environment
variable is shadowed -- though it doesn't cause a problem, since it's
shadowed with the correct value, odd as this may be!
[1]: c7c2984716/pkgs/development/python-modules/pybind11/default.nix (L19)
Change-Id: Ibf6790d2484dc9cce8e424feeb5886664d498dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8696
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
`tvix-store mount PATH` will mount the tvix-store to the given path.
Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This allows using a StorePath as a key in a hashmap.
Change-Id: Id3eed623da4e1fc44a970a3982c7caa21d2495c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8666
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This brings in fuse (via the `fuser` crate), and adds pkg-config and
libfuse to the dev shell, so `cargo build` can link against it.
Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This testcase tests a missing blob fails the rendering, the comment has
been copied from elsewhere.
Change-Id: I48fa3fa454e12506590fa14a3591d156bafa8b5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8722
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The CLs did bitrot a bit, they're based on an older version of the
protocol, and it's unclear if they'll be a separate Go Binary, or just
another HTTP handler inside tvix-store itself, considering we now have
way more NAR juggling code than before.
Change-Id: I3632035cda8d75a8ff23b3132312f0f086d9e02f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8732
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This has moved to a `daemon` subcommand.
Change-Id: Iae9778d8a59e6bf84555119fabfd62db3917bb62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8731
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
When comparing to C++ Nix, we notice that the thunking of default
expressions in function formals corresponds to their normal thunking,
e.g. literals are not thunked. This means that we can just invoke
compile() without much of a care and trust that it will sort it out
correctly.
If function formals blow up as a result of this, it likely indicates
that the expression is treated incorrectly by compile(), not
compile_param_pattern().
Change-Id: I64acbff2f251423eb72ce43e56a0603379305e1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8704
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
C++ Nix forces and typechecks the passed argument even if it is not
necessary in order to compute the return value of the function. I
discovered this when I thought our formals miscompilation might be that
we are too strict, but doesn't look like it in this case.
Change-Id: Ifb3c92592293052c489d1e3ae8c7c54e4b6b4dc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8701
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Before, the construction of a TwoByteWM would panic when no patterns
were provided, as in `tvix --expr 'builtins.toFile "snens" "soos"'`.
Change-Id: I25ed498c475523aec5baf8683b23059fadabb21c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8697
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This now creates different store client, depending on the cli
subcommand.
The `import` command will connect to the gRPC service, and the `daemon`
command will use the sled implementation.
It might make sense to define some URI syntax to make this configurable
by the user, via the CLI.
Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We don't need to be able to clone these services in here.
Change-Id: Ifb69450f7ebdc8364cbf9cdfb6464f8560440e4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8645
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
nix_oracle.rs now gives us the possibility to check this by stuffing the
expressions in a list. In fact, the incorrect behavior fixed in
- cl/8656
- cl/8655
- cl/8662
was discovered using this test suite.
Change-Id: Id0ab01ee6be0b875791214e0a72a2ac941c46c96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8658
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
