Commit graph

19536 commits

Author SHA1 Message Date
Adam Joseph
29878259b3 feat(tvix/cli): add benchmark for bf286a54bc
cl/7558 used this expression as a benchmark to justify the introduction
of LightSpan::Delayed:

  builtins.length (builtins.attrNames (import ${pkgs.path} {}))

Let's add it as a benchmark case so it can be referenced easily.

Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
Change-Id: Idb6c69ddd284605dd3b5fd9ac5c79a69b9a470b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10253
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:28:53 +00:00
Florian Klink
27c07b72c6 refactor(tvix): use io::Result for EvalIO
This is just a alias for Result<_, io::Error>, but shorter.

Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 14:28:50 +00:00
Adam Joseph
ad566999ca fix(tvix/eval): preserve catchables in nix_cmp_ordering(), fix b/338
This commit fixes b/338 by properly propagating catchables through
comparison operations.

Change-Id: I6b0283a40f228ecf9a6398d24c060bdacb1077cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10221
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
ae28dc3ca6 test(tvix/eval): test for b/338 catchable hygiene problem
Commit 05f42519b5 fixed b/281 by
establishing a hygiene regimen to partition *catchable* errors
(i.e. those which tryEval can detect) from all other errors, like
internal VM failures or I/O errors (which Nix must not be allowed to
detect, since these errors are fundamentally impure).

Unfotunately there are still cases where tvix assumes that anything
other than Value::Bool means it should panic!().  I found another
one, and added a test case for it in:

  eval_okay_src_tests_tvix_tests_eval_okay_compare_ordering_catchable_nix

Not yet passing.

Change-Id: I69c62ed9ea5c8f81870e8de5c5fe12dcde849763
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10220
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
edbd5055a1 feat(tvix/eval): nonrecursive nix_cmp_ordering(), fixes b/339
This commit rewrites Value::nix_cmp_ordering() into an equivalent
nonrecursive form.  Except for calls to Thunk::force(), the new form
no longer uses generators, and is async only because of the fact
that it calls Thunk::force().

I originally believed that this commit would make evaluation faster.
In fact it is slightly slower.  I believe this is due to the added
vec![] allocation.  I am investigating.

Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460224","system-seconds":"0.67","user-seconds":"5.84"}
Change-Id: Ic627bc220d9c5aa3c5e68b9b8bf199837cd55af5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10212
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
8a40f75c2d fix(tvix/eval): never use partial_cmp() (partial fix b/338)
This is part of a fix for b/338.

We should never use PartialOrd::partial_cmp().

All Nix types except floats are obviously totally-ordered.  In
addition, it turns out that because Nix treats division by zero
rather than producing a NaN, and because it does not support
"negative zero", even floats are in fact totally ordered in Nix.

Therefore, every call to PartialOrd::partial_cmp() in tvix is an
error.  We have to *implement* this function, but we should never
call it on built-in types.

Moreover, nix_cmp_ordering() currently returns an Option<Ordering>.
I'm not sure what was going on there, since it's impossible for it
to return None.  This commit fixes it to return simply Ordering
rather than Option<Ordering>.

Change-Id: If5c084164cf19cfb38c5a15554c0422faa5f895d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10218
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
19d13eb070 test(tvix/eval): add test case for b/339
Not yet passing.

Change-Id: I1de3f72d8b3f46567fdba010fc3ab4bace3f1699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10219
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
72ece2e518 feat(tvix/eval): nonrecursive nix_eq()
This commit rewrites Value::nix_eq() into an equivalent.  Except for
calls to Thunk::force(), the new form no longer uses generators, and
is async only because of the fact that it calls Thunk::force().

I believed that the nonrecursive form would be faster.  It is, in
fact, slightly slower.  I believe this is due to the vec![]
allocation; I am investigating.

Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"459068","system-seconds":"0.71","user-seconds":"5.39"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
Change-Id: I10f4868891e4b7475df13f0cbc41ec78dd985dd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10118
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
0c15a09b15 feat(tvix/cli): add macrobenchmark
This commit adds a simple MVP benchmark, built on our nix
infrastructure instead of cargo.  It simply runs `tvix-eval` inside
of GNU time, and prints the three essential statistics in a short
JSON blob.

You can run the benchmark with a simple `nix run`, like:

  nix run -f . tvix.cli.benchmark-hello
  nix run -f . tvix.cli.benchmark-firefox
  nix run -f . tvix.cli.benchmark-cross-firefox

Currently these blobs are stored only in the CI logs, which I'm sure
get garbage-collected at some point.  We should be putting them in
the git trailers, but that can wait for a future CL.

I tried using `cargo bench` for this but found it incredibly
frustrating.  Maybe I'm doing it wrong.  It seems to be designed for
microbenchmarks only, and very hard to control.  It kept building
all sorts of unnecessary stuff (like the tests), and unlike
crate2nix it was doing all the builds on only a single machine
instead of using more than one machine.  Worse, for that single
machine it kept picking my laptop instead of my fast servers!  It
seems excessively cargo-flavored for such a straightforward task.

Benchmark: {"hello.outPath":{"kbytes":"244736","system":"0.36","user":"2.76"}}
Benchmark: {"firefox.outPath":{"kbytes":"1506736","system":"2.38","user":"32.01"}}
Benchmark: {"pkgsCross.aarch64-multiplatform.firefox.outPath":{"kbytes":"11334548","system":"10.70","user":"107.07"}}

Change-Id: I85bc046ec551360284d7ecfc81a03914f0085909
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10216
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 11:56:58 +00:00
Aspen Smith
02c6621813 feat(tools/magrathea): Allow running commands in a shell
Add support for running a command inside a `mg shell`, specified as an
extra argument after the target to the shell command

Change-Id: Icbbd9cf4e1f099fcd7e6b13655b8447775a236d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10247
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 11:44:46 +00:00
Florian Klink
8018313b68 feat(tvix/glue): add nixpkgs eval benchmark
This adds a criterion.rs-based testbench into tvix-glue.

It can be invoked by running `cargo bench` from inside the `tvix-glue`
crate.
`target/criterion/report/index.html` contains nice graphs.
It's able to diff against the previous run, so you can invoke `cargo
bench` before and after a certain change to reason about the impact in
evaluation performance.

Currently, we need to create a bunch of Evaluator resources inside the
benchmark loop itself, which is a bit annoying, as it leaks into the
things we benchmark.

This should become better with b/262.

Fixes b/322.

Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 10:27:51 +00:00
Florian Klink
6c87ab960d chore(tvix/glue): allow unused_variables
cl/9364 did introduce a warning here, which is visible when building in
release mode - or invoking `cargo bench` in tvix-glue.

Change-Id: Ia82082a58543f0fdd32866fdfcd37d0a5fdfda9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10261
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 09:01:51 +00:00
Florian Klink
ccaf10b4a6 refactor(tvix/*store/sled): make ::new() more generic
We don't really require the Path to be a PathBuf, we don't even require
it to be a Path, we only need it to be AsRef<Path>>.

This removes some conversion in the from_addr cases, which can just
reuse `url.path()` (a `&str`).

Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 22:46:50 +00:00
Florian Klink
9748543f1c chore(tvix): drop sled compression support
It's been a while since the last sled release, and that one binds to a
pretty old version of zstd, requiring workarounds like cl/10090.

Upstream sled main branch currently has zstd halfway patched out (it's
a no-op, but the feature flag and options are still there), and it's in
that state for a year.

Rather than maintaining our own fork of sled, let's just stop using the
compression feature in sled, dropping the version pin to zstd that way,
removing the need for cl/10090.

This doesn't mean we won't reintroduce per-blob compression - but we
probably just won't let sled take care of the compression, but do it
ourselves - which is necessary for more chunked blob storage anyways.

Even though we do drop the feature flag, we still need to explicitly use
use_compression(false).

Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 22:46:50 +00:00
Florian Klink
6025be423a chore(tvix/proto): drop evaluator.proto
This only contained the (unused) evaluator.proto file.

Considering we're less likely to have the CLI talk to a long-running
evaluator, but instead embed the evaluator inside the CLI, remove this.

If we add a RPC to speak to an evaluator, we can resurrect this from
git history.

Change-Id: I2196aade55221660330dfd32dc3e52c39ec6ed43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10241
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
1cc3e9c1a8 refactor(tvix): use granular proto files
Only pass in the proto files that are actually needed to build that
crate. They are already constructed in depot.tvix.$crate.protos.protos.

Change-Id: If4381e6c3350e420ee4ddce1e0513bfe970678a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10240
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
43943e2623 refactor(tvix/*/protos): separate lint target
Break up the go-bindings derivation. Keep "protos" containing all proto
files (well, and the buf config), and use it for a check phase running
linter and formatter, as well as the existing "go-bindings" attribute

Change-Id: I52cb9d08570bb76452acb831eb711c5b6c0eacfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10239
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
8486f87e3c feat(tvix/build): add derivation_to_build_request
This function converts from a nix_compat::derivation::Derivation to
a BuildRequest.

In addition to the Derivation itself, it needs two lookup functions to
map input paths to their castore nodes.

Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 22:35:39 +00:00
Florian Klink
92bd69aef2 docs(tvix/*store-go): fix README invocation
The command is called `regenerate`, not `generate`.

Change-Id: I18075042ebd461e4dd0718a936e6bbe738a144d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10259
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-11 21:47:43 +00:00
Florian Klink
a1b2dc8aaf feat(tvix/build-go): init
This adds the generated golang bindings for tvix-build.

Change-Id: I2eb0d1cc38bc2fa34afd7c904eea05c5ee192cce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10242
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 21:43:38 +00:00
Florian Klink
fd27d8ddc3 feat(tvix/build): init
This adds the tvix-build crate, currently only containing a
`tvix_build::proto` module, exposing the data structures defined in
tvix/build/protos.

Change-Id: I75f5d9196969ed0877b1fe640cacfecba0fb2e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10235
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 21:43:38 +00:00
Florian Klink
3c0a9a949a chore(tvix/store/protof): buf format
Change-Id: I29b0fb62aa849ee01860dc7fe3dbba9b941ca172
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10238
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 10:29:50 +00:00
Florian Klink
459d9e106f chore(tvix/castore/protof): buf format
Change-Id: Idf11de78b0d6eca69fda34a89f2c57a00ed89ad5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10237
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 10:24:44 +00:00
Vincent Ambo
db3ef5255f fix(views/tvix): Canonicalise Tvix workspace formatting
This change is done by josh on push back to Tvix, as the formatting of
our workspace files differs from its canonical formatting.

Change-Id: I97bf9d87f82b6acbe3350c1a3c99bd38b5af98e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10251
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-12-11 10:22:12 +00:00
Aspen Smith
d9ac36d307 fix(tvix): Fix cwd in benchmark script
Change-Id: I6d9435c6710ce086aaedb2c1dc634b73a19f476c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10249
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-11 04:13:41 +00:00
Florian Klink
01a3c87a3a chore(tvix): bump criterion to 0.5
Change-Id: I28904ca23437b4bb745c0eb1f4eb9ae33e09eb5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10244
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 03:59:30 +00:00
Florian Klink
b14053ae03 docs(tvix): fix crate2nix-generate invocation
This got renamed recently, but we forgot to update it here.

Change-Id: I7d713c8a0e6ccca57fe67985d9cb4e7f1eeef3b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10243
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-11 03:59:28 +00:00
Aspen Smith
b2c8c1ef94 feat(tvix): Benchmark nixpkgs eval with hyperfine
Add some hyperfine benchmarks to Tvix's windtunnel benchmark script for
evaluating the outPath of hello and a cross-compiled hello.

Change-Id: I9d76e5ce0a3fd7d9c125c36c5fced675b660a8a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10248
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
2023-12-11 03:59:27 +00:00
Adam Joseph
19fd6ffae5 chore(third_party/overlays): disable nix aws integration
The Nix AWS integration is a complete dumpster-fire, the library it
relies on (aws-sdk-cpp) accounts for more build time than cppnix
itself, and its tests fail nondeterministically.  We recently
disabled it for nixpkgs' CI:

  https://github.com/NixOS/nixpkgs/pull/266443

Since it appears TVL does not use the AWS integration (correct me if
I'm wrong) let's disable it here too.  Hopefully someday cppnix will
give up on it and just delete it.

Change-Id: I71668c03379275b2083e59ff7c3a48b3bc7f5e48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10202
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-10 14:17:53 +00:00
Adam Joseph
2b4920c355 fix(tvix/cli): panic on root cause of the fetchurl(non-boot) bug
Currently we produce wrong drvPaths for a large number of packages
that use fetchurl (but not fetchurlBoot, which is what stdenv uses).
A simple reproducer is `pkgs.perl538`.

I debugged this down to the root cause, which is the fact that tvix
doesn't realize that the mapping from FOD-paths to outputHash is
*NOT* a 1:1 mapping.  It is a many-to-one mapping.  You can have
lots of different FODs with the same outputHash or even the same
outPath.  For example, perl538.src and perldevel.src use the same
source tarball but a different `version`.

Anyways, I have found the root cause but have run out of time for a
while, so I've added a panic!() to in the spot where we have a logic
bug in order to call it out.

Change-Id: I9766b39cfe2fe7eafec84945b2ad6cc28f9c4b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9364
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-10 14:16:21 +00:00
Florian Klink
efaff79584 feat(tvix): make clippy stricter
Also run clippy on tests, and enable all features.

Change-Id: Ide9f1bc9f565333072afb918c391c7930b658f41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10234
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
22a669d27e refactor(tvix/castore): address clippy
We match to destructure a single pattern.

Change-Id: I564a3510b4860e90b3315a9639effc48ee88b483
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10233
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
Florian Klink
340242174d refactor(nix-compat/aterm): address clippy
This pushes to a Vec immediately after creation.

Change-Id: I2360b45810475d98ededc1d097fb4cbdeabc576b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10232
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-09 12:58:39 +00:00
Florian Klink
f9ff4fe969 refactor(store/fs): address clippy
Change-Id: I28d60263ef672942940656caa8e0f9d20cc6b7d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10231
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
f4c4e32040 refactor(nix-compat/nixbase32): address clippy
Change-Id: I07b53ead10b0efb31e88c6ae93c124c5f35261d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10230
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
06350445e9 refactor(nix-compat/narinfo): address clippy lints
Change-Id: I00f28624ec0469525e52f408f00caa20d3b701b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10229
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
1bdb90e56d refactor(nix-compat/derivation): address clippy
Change-Id: I5d7057347838f733d7dc1331c7b7ef8d27244412
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10228
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-12-09 12:58:39 +00:00
Florian Klink
d3ecef1a5b refactor(tvix/eval): address clippy lints
Change-Id: Ic2bd4e8291b30ceac9fa0e88a4f56e61ae99b603
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10227
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
Florian Klink
f11c6d610c chore(tvix/nix_cli): drop crate
The only thing this was doing was invoking nix-store --add, which is not
gonna help us populate the tvix-store at all (and we now have
`tvix-store import`).

This is also (rightfully) causing clippy warnings, because of some
fields being unused.

It's more of a skeleton, and rather than shelling out to Nix for some
usecases, we might introduce a "compatible" Nix CLI frontend for a
subset of commands.

Drop this for now, to decrease the noise and confusion.

Change-Id: I2fd399e9320260f08893b685561755af9c7c961c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10226
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
0907420600 refactor(nix-compat/store_path): from_absolute_path to StorePathRef
The only non-test usage was only checking for the error case, and we can
still convert this to an owned StorePath by calling to_owned() on
StorePathRef.

Change-Id: I9f67a759e580c9c429c96896bcdd295392aa5a2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10225
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:35 +00:00
Florian Klink
5855959d87 refactor(nix-compat/derivation): rename input derivation components
Match the naming in parse_input_derivations, call the keys
"input_derivation", and the values "output_names".

Change-Id: I7d1974819028f8ea543dc3ad78afb803ff9db865
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10224
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:20:51 +00:00
Adam Joseph
5fd42a9e70 feat(tvix/eval): impl DoubleEndedIter for OwnedAttrsIterator
Change-Id: I4bd85dbe9c27047f4abbdeff4e2b796e9bcab3a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-09 11:35:19 +00:00
Adam Joseph
36c3e646e1 chore(store/src/tests): make clippy shut up
Change-Id: I6dfceaa32a8c01f8395b7889ae19847fabf95ed3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10214
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 10:51:10 +00:00
Adam Joseph
dfca879535 fix(tvix/glue): do not panic if PathInfoService returns Err
Nixpkgs tries to `import` the value returned by
`builtins.unsafeGetAttrPos`, which in our case is the file
`/deep/thought`.  Since that doesn't exist, tvix-glue panics, but
there's no interpreter backtrace to follow.

Let's return an Err instead of panicking.

------------------------------------------------------------------------------
Before:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at src/tvix_store_io.rs:276:58:
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(41580), ...)
Command exited with non-zero status 101

------------------------------------------------------------------------------
After:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: while evaluating this Nix code
   --> [code]:1:1
    |
1   | (import /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src {}).pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:6:78
    |
6   |   runCommand "rocm-llvm-binutils-${llvm.version}" { preferLocalBuild = true; } ''
    |  ______________________________________________________________________________^
7   | |   mkdir -p $out/bin
8   | |
9   | |   for prog in ${lld}/bin/*; do
...   |
27  | |   ln -s ${lld}/bin/lld $out/bin/ld
28  | | ''
    | |__^
note: while evaluating this as native code (coerce_to_string)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:27:9
    |
27  |   ln -s ${lld}/bin/lld $out/bin/ld
    |         ^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:23
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:30
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                              ^^^^^^^^^
note: while evaluating this as native code (throw)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:262:8
    |
262 |     in handler msg;
    |        ^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |________________________________________________________^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |__________^
error[E029]: I/O error: /deep/thought:42: task panicked
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:255:41
    |
255 |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
    |                                         ^^^^^^^^^^^^^^^

Command exited with non-zero status 1
Benchmark: {"pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath":{"kbytes":"26613180","system":"22.35","user":"140.62"}}

Change-Id: I587b57e9e49d1f3ecdc0fc9cf996d179a3548f34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10223
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 10:47:39 +00:00
Adam Joseph
f4460bd612 chore(nix_cli): make clippy shut up
Change-Id: I0c243baf4deecfd81bd939da86d685e2cbc59d69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10215
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-09 09:25:24 +00:00
Adam Joseph
2132bbcb47 fix(tvix/nar-bridge): fix FOD hash
The golang mothership seems to be monkeying with hashes again.

Change-Id: I7430b4cde84fa51be2b572fba02e3567864bb87a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10209
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 09:24:53 +00:00
Adam Joseph
0e2633048e feat(tvix): run crate2nix generate in CI
This runs `crate2nix generate` in CI and then runs `depotfmt` on the
result to ensure that our machine-generated code is really, really
readable and pretty.  Then it checks that the result of all that
is identical to the committed Cargo.nix.

A self-hashing FOD is used to allow network access.

No magic hashes are involved.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I68ec5003dbc6a40894a5a4d6e902f138c99f6719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10194
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-07 18:43:46 +00:00
Profpatsch
23dae8ebc5 feat(users/Profpatsch/aliases): bell.
bell.

Change-Id: If9385d0dfbdd3eda9a2cad6b1cabb3d938987195
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10206
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-12-06 19:15:15 +00:00
Aspen Smith
0b524890e9 refactor(tvix): Make benchmark script directory-agnostic
Allow running the benchmark script from any directory - primarily so
Windtunnel can point to the /tvix josh workspace rather than the depot
overall

Change-Id: Ie5fc3ef995bf8114277298ae5c5010e6a0bf13ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10205
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2023-12-06 16:19:46 +00:00
Vincent Ambo
e59d263b44 chore(3p/sources): bump channels & overlays (2023-12-06)
* 3p/rust-crates: fix names of crate URLs to accommodate recent
  crates.io changes:

  https://blog.rust-lang.org/2023/10/27/crates-io-non-canonical-downloads.html

* 3p/rust-crates: fix comment describing what this is

* 3p/overlays: discard custom overrides of Nix 2.3: it's now
  maintained properly upstream

* users/wpcarro/emacs: disable doom themes package

Change-Id: Ic5def77319a0a55e78c8ffe05b9309d59784cfd9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10204
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-12-06 12:39:38 +00:00