This commit fixes our handling of `throw` within an `assert`
condition.
Fixes: b/340
Change-Id: I40a383639ec266da50a853f16216b1b7868495da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10318
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This commit fixes out `?` operator so it correctly propagates
catchables.
Change-Id: Iebaa153a8492101ee3ddd29893c98730ff331547
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10317
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Previously, using a catchable as either argument of OpAttrsSelect
would result in an unrecoverable error. This commit matches cppnix
behavior by propagating the catchable.
Change-Id: I4877f4068ec2b823225f185290693c101d0b9c9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10303
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
When attempting to call a Value, if it is a Value::Catchable we must
not cause an uncatchable failure. This commit simply reuses the
Value::Catchable as the result of attempting to call it. This is
safe because nix is designed so that nix code cannot distinguish
between different catchable failures -- they all look the same to
the interpreted code.
This fixes b/351.
Change-Id: Ibf763a08753e541843626182ff59fdbf15ea2959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10300
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
error[E006]: expected value of type 'bool', but found a 'internal[catchable]'
--> src/tests/tvix_tests/notyetpassing/eval-okay-test-catchables-in-implications.nix:1:43
|
1 | (builtins.tryEval (({ foo ? throw "up" }: foo -> true) { })).success
| ^^^^^^^^^^^
Relates to b/345
Change-Id: Ic331c32ea59bf67ae775f485b444dc6804ca13d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10289
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
This commit adds Opcode::OpJumpIfCatchable, which can be inserted
ahead of most VM operations which expect a boolean on the stack, in
order to handle catchables in branching position properly.
Other than remembering to patch the jump, no other changes should be
required.
This commit also fixes b/343 by emitting this new opcode when
compiling if-then-else. There are probably other places where we
need to do the same thing.
Change-Id: I48de3010014c0bbeba15d34fc0d4800e0bb5a1ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10288
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
This is a test case for b/343, wherein tvix dies if you try to
branch on an argument whose defaulted value is a catchable.
Change-Id: I891ca825e39ad14dda9f220f06d9591874fcd45d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10287
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
After this commit, the only non-builtins uses of generators are:
- coerce_to_string() uses generators::request_enter_lambda()
- Thunk::force() uses generators::request_enter_lambda()
That's it! Once those two are taken care of, GenCo can become an
implementation detail of `builtins::BuiltinGen`. No more crazy
nonlocal flow control within the interpreter: if you've got a GenCo
floating around in your code it's because you're writing a builtin,
which isn't part of the core interpreter. The interpreter won't
need GenCos to talk to itself anymore.
Technically generators::request_path_import() is also used by
coerce_to_string(), but that's just because the io_handle happens to
be part of the VM. There's no recursion-depth issue there, so the
call doesn't need to go through the generator mechanism
(request_path_import() doesn't call back to the interpreter!)
Change-Id: I83ce5774d49b88fdafdd61160975b4937a435bb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10256
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This commit implements deep_force() nonrecursively, by maintaining
an explicit stack rather than using the call stack for recursion.
As an added bonus, we don't need to pass around the SharedThunkSet
anymore, and can in fact completely eliminate SharedThunkSet.
Change-Id: I7c4f59f37834d451a28bf6be317eb0a90eac4ee6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10252
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
The comment explaining ThunkSet makes it seem like it does the same
think as ThunkRepr::Blackhole. In fact neither one is a substitute
for the other. Let's explain the difference.
Change-Id: I89ceaaa9d3c499edbc7d48f70ca5d11f97666c43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10250
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
LightSpan::Delayed was introduced in commit
bf286a54bc which claimed that "This
reduces the eval time for `builtins.length (builtins.attrNames
(import <nixpkgs> {}))` by *one third*!"
I am unable to reproduce this result. In fact, dropping the
LightSpan::Delayed variant of the enum makes eval of the same
expression slightly faster! I also tried a large evaluation
(pkgsCross...hello) and got similar results: slightly faster,
slightly less memory. See git footers.
I suspect that there was some unrelated horrific inefficiency that
has since been fixed. The avoided computation in `get_span()` is
nothing more than a binary search! If this were in fact a major
performance issue we could simply precompute the mapping from
CodeIdx to Span when the Chunk becomes immutable (i.e. at the end of
the compilation process, when compiler backtracking is no longer a
concern). Since a Span is just 64 bits this is not a space issue,
and since binary search is much simpler than compiling Nix
expressions it isn't a performance issue either.
Technically there is no longer any reason to have LightSpan since it
is now a single-variant enum. However there is no rush to remove
it, since Rust will optimize its representation into the same thing
you'd get if you replaced LightSpan by Span.
Prev-Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
This-Benchmark: {"nixpkgs-attrnames":{"kbytes":"230192","system":"0.29","user":"2.00"}}
Prev-Benchmark: {"pkgsCross.aarch64-multiplatform.hello.outPath":{"kbytes":"458936","system":"0.73","user":"5.36"}}
This-Benchmark: {"pkgsCross.aarch64-multiplatform.hello.outPath":{"kbytes":"451808","system":"0.53","user":"5.10"}}
Change-Id: Ib9e04806850aa1fc4e66e2a042703986440a7b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10254
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
cl/7558 used this expression as a benchmark to justify the introduction
of LightSpan::Delayed:
builtins.length (builtins.attrNames (import ${pkgs.path} {}))
Let's add it as a benchmark case so it can be referenced easily.
Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
Change-Id: Idb6c69ddd284605dd3b5fd9ac5c79a69b9a470b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10253
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is just a alias for Result<_, io::Error>, but shorter.
Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This commit fixes b/338 by properly propagating catchables through
comparison operations.
Change-Id: I6b0283a40f228ecf9a6398d24c060bdacb1077cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10221
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Commit 05f42519b5 fixed b/281 by
establishing a hygiene regimen to partition *catchable* errors
(i.e. those which tryEval can detect) from all other errors, like
internal VM failures or I/O errors (which Nix must not be allowed to
detect, since these errors are fundamentally impure).
Unfotunately there are still cases where tvix assumes that anything
other than Value::Bool means it should panic!(). I found another
one, and added a test case for it in:
eval_okay_src_tests_tvix_tests_eval_okay_compare_ordering_catchable_nix
Not yet passing.
Change-Id: I69c62ed9ea5c8f81870e8de5c5fe12dcde849763
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10220
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit rewrites Value::nix_cmp_ordering() into an equivalent
nonrecursive form. Except for calls to Thunk::force(), the new form
no longer uses generators, and is async only because of the fact
that it calls Thunk::force().
I originally believed that this commit would make evaluation faster.
In fact it is slightly slower. I believe this is due to the added
vec![] allocation. I am investigating.
Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460224","system-seconds":"0.67","user-seconds":"5.84"}
Change-Id: Ic627bc220d9c5aa3c5e68b9b8bf199837cd55af5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10212
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This is part of a fix for b/338.
We should never use PartialOrd::partial_cmp().
All Nix types except floats are obviously totally-ordered. In
addition, it turns out that because Nix treats division by zero
rather than producing a NaN, and because it does not support
"negative zero", even floats are in fact totally ordered in Nix.
Therefore, every call to PartialOrd::partial_cmp() in tvix is an
error. We have to *implement* this function, but we should never
call it on built-in types.
Moreover, nix_cmp_ordering() currently returns an Option<Ordering>.
I'm not sure what was going on there, since it's impossible for it
to return None. This commit fixes it to return simply Ordering
rather than Option<Ordering>.
Change-Id: If5c084164cf19cfb38c5a15554c0422faa5f895d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10218
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit rewrites Value::nix_eq() into an equivalent. Except for
calls to Thunk::force(), the new form no longer uses generators, and
is async only because of the fact that it calls Thunk::force().
I believed that the nonrecursive form would be faster. It is, in
fact, slightly slower. I believe this is due to the vec![]
allocation; I am investigating.
Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"459068","system-seconds":"0.71","user-seconds":"5.39"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
Change-Id: I10f4868891e4b7475df13f0cbc41ec78dd985dd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10118
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This commit adds a simple MVP benchmark, built on our nix
infrastructure instead of cargo. It simply runs `tvix-eval` inside
of GNU time, and prints the three essential statistics in a short
JSON blob.
You can run the benchmark with a simple `nix run`, like:
nix run -f . tvix.cli.benchmark-hello
nix run -f . tvix.cli.benchmark-firefox
nix run -f . tvix.cli.benchmark-cross-firefox
Currently these blobs are stored only in the CI logs, which I'm sure
get garbage-collected at some point. We should be putting them in
the git trailers, but that can wait for a future CL.
I tried using `cargo bench` for this but found it incredibly
frustrating. Maybe I'm doing it wrong. It seems to be designed for
microbenchmarks only, and very hard to control. It kept building
all sorts of unnecessary stuff (like the tests), and unlike
crate2nix it was doing all the builds on only a single machine
instead of using more than one machine. Worse, for that single
machine it kept picking my laptop instead of my fast servers! It
seems excessively cargo-flavored for such a straightforward task.
Benchmark: {"hello.outPath":{"kbytes":"244736","system":"0.36","user":"2.76"}}
Benchmark: {"firefox.outPath":{"kbytes":"1506736","system":"2.38","user":"32.01"}}
Benchmark: {"pkgsCross.aarch64-multiplatform.firefox.outPath":{"kbytes":"11334548","system":"10.70","user":"107.07"}}
Change-Id: I85bc046ec551360284d7ecfc81a03914f0085909
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10216
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Add support for running a command inside a `mg shell`, specified as an
extra argument after the target to the shell command
Change-Id: Icbbd9cf4e1f099fcd7e6b13655b8447775a236d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10247
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds a criterion.rs-based testbench into tvix-glue.
It can be invoked by running `cargo bench` from inside the `tvix-glue`
crate.
`target/criterion/report/index.html` contains nice graphs.
It's able to diff against the previous run, so you can invoke `cargo
bench` before and after a certain change to reason about the impact in
evaluation performance.
Currently, we need to create a bunch of Evaluator resources inside the
benchmark loop itself, which is a bit annoying, as it leaks into the
things we benchmark.
This should become better with b/262.
Fixes b/322.
Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
cl/9364 did introduce a warning here, which is visible when building in
release mode - or invoking `cargo bench` in tvix-glue.
Change-Id: Ia82082a58543f0fdd32866fdfcd37d0a5fdfda9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10261
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We don't really require the Path to be a PathBuf, we don't even require
it to be a Path, we only need it to be AsRef<Path>>.
This removes some conversion in the from_addr cases, which can just
reuse `url.path()` (a `&str`).
Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
It's been a while since the last sled release, and that one binds to a
pretty old version of zstd, requiring workarounds like cl/10090.
Upstream sled main branch currently has zstd halfway patched out (it's
a no-op, but the feature flag and options are still there), and it's in
that state for a year.
Rather than maintaining our own fork of sled, let's just stop using the
compression feature in sled, dropping the version pin to zstd that way,
removing the need for cl/10090.
This doesn't mean we won't reintroduce per-blob compression - but we
probably just won't let sled take care of the compression, but do it
ourselves - which is necessary for more chunked blob storage anyways.
Even though we do drop the feature flag, we still need to explicitly use
use_compression(false).
Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This only contained the (unused) evaluator.proto file.
Considering we're less likely to have the CLI talk to a long-running
evaluator, but instead embed the evaluator inside the CLI, remove this.
If we add a RPC to speak to an evaluator, we can resurrect this from
git history.
Change-Id: I2196aade55221660330dfd32dc3e52c39ec6ed43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10241
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Only pass in the proto files that are actually needed to build that
crate. They are already constructed in depot.tvix.$crate.protos.protos.
Change-Id: If4381e6c3350e420ee4ddce1e0513bfe970678a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10240
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Break up the go-bindings derivation. Keep "protos" containing all proto
files (well, and the buf config), and use it for a check phase running
linter and formatter, as well as the existing "go-bindings" attribute
Change-Id: I52cb9d08570bb76452acb831eb711c5b6c0eacfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10239
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This function converts from a nix_compat::derivation::Derivation to
a BuildRequest.
In addition to the Derivation itself, it needs two lookup functions to
map input paths to their castore nodes.
Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
The command is called `regenerate`, not `generate`.
Change-Id: I18075042ebd461e4dd0718a936e6bbe738a144d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10259
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This adds the tvix-build crate, currently only containing a
`tvix_build::proto` module, exposing the data structures defined in
tvix/build/protos.
Change-Id: I75f5d9196969ed0877b1fe640cacfecba0fb2e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10235
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This change is done by josh on push back to Tvix, as the formatting of
our workspace files differs from its canonical formatting.
Change-Id: I97bf9d87f82b6acbe3350c1a3c99bd38b5af98e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10251
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This got renamed recently, but we forgot to update it here.
Change-Id: I7d713c8a0e6ccca57fe67985d9cb4e7f1eeef3b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10243
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Add some hyperfine benchmarks to Tvix's windtunnel benchmark script for
evaluating the outPath of hello and a cross-compiled hello.
Change-Id: I9d76e5ce0a3fd7d9c125c36c5fced675b660a8a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10248
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
The Nix AWS integration is a complete dumpster-fire, the library it
relies on (aws-sdk-cpp) accounts for more build time than cppnix
itself, and its tests fail nondeterministically. We recently
disabled it for nixpkgs' CI:
https://github.com/NixOS/nixpkgs/pull/266443
Since it appears TVL does not use the AWS integration (correct me if
I'm wrong) let's disable it here too. Hopefully someday cppnix will
give up on it and just delete it.
Change-Id: I71668c03379275b2083e59ff7c3a48b3bc7f5e48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10202
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Currently we produce wrong drvPaths for a large number of packages
that use fetchurl (but not fetchurlBoot, which is what stdenv uses).
A simple reproducer is `pkgs.perl538`.
I debugged this down to the root cause, which is the fact that tvix
doesn't realize that the mapping from FOD-paths to outputHash is
*NOT* a 1:1 mapping. It is a many-to-one mapping. You can have
lots of different FODs with the same outputHash or even the same
outPath. For example, perl538.src and perldevel.src use the same
source tarball but a different `version`.
Anyways, I have found the root cause but have run out of time for a
while, so I've added a panic!() to in the spot where we have a logic
bug in order to call it out.
Change-Id: I9766b39cfe2fe7eafec84945b2ad6cc28f9c4b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9364
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
