Commit graph

164 commits

Author SHA1 Message Date
Florian Klink
3cdde8ad5a feat(tvix/glue): add some import tests
This creates a directory with a .keep file inside, and uses
"${path/to/there}" to coerce it to a string (and import it into the
store), ensuring it calculates the same store paths as Nix does.

Change-Id: Ie14ae075104ce278bc4f2cce93aab5762a2734d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10343
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-13 13:41:44 +00:00
Florian Klink
307675f5e7 fix(tvix/glue/tvix_store_io): don't unwrap ingest_path errors
Fixes b/344.

Change-Id: I1446726e3be3a8fc20801d466a964c4d6b8cbc02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10331
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:43 +00:00
Adam Joseph
07ca556a94 feat(tvix/glue): make unimplemented-structuredAttrs catchable
This commit adjusts the error produced by STRUCTURED_ATTRS so that
it is catchable.  This way we are able to enumerate the release
packageset, and the enumeration process will simply treat the few
derivations using structured attributes as being broken, rather than
killing the whole eval session.

Change-Id: I2e17638b8e3227f88543c3718aaf505deaec22ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10306
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 17:21:53 +00:00
Florian Klink
27c07b72c6 refactor(tvix): use io::Result for EvalIO
This is just a alias for Result<_, io::Error>, but shorter.

Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 14:28:50 +00:00
Florian Klink
8018313b68 feat(tvix/glue): add nixpkgs eval benchmark
This adds a criterion.rs-based testbench into tvix-glue.

It can be invoked by running `cargo bench` from inside the `tvix-glue`
crate.
`target/criterion/report/index.html` contains nice graphs.
It's able to diff against the previous run, so you can invoke `cargo
bench` before and after a certain change to reason about the impact in
evaluation performance.

Currently, we need to create a bunch of Evaluator resources inside the
benchmark loop itself, which is a bit annoying, as it leaks into the
things we benchmark.

This should become better with b/262.

Fixes b/322.

Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 10:27:51 +00:00
Florian Klink
6c87ab960d chore(tvix/glue): allow unused_variables
cl/9364 did introduce a warning here, which is visible when building in
release mode - or invoking `cargo bench` in tvix-glue.

Change-Id: Ia82082a58543f0fdd32866fdfcd37d0a5fdfda9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10261
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 09:01:51 +00:00
Florian Klink
8486f87e3c feat(tvix/build): add derivation_to_build_request
This function converts from a nix_compat::derivation::Derivation to
a BuildRequest.

In addition to the Derivation itself, it needs two lookup functions to
map input paths to their castore nodes.

Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 22:35:39 +00:00
Adam Joseph
2b4920c355 fix(tvix/cli): panic on root cause of the fetchurl(non-boot) bug
Currently we produce wrong drvPaths for a large number of packages
that use fetchurl (but not fetchurlBoot, which is what stdenv uses).
A simple reproducer is `pkgs.perl538`.

I debugged this down to the root cause, which is the fact that tvix
doesn't realize that the mapping from FOD-paths to outputHash is
*NOT* a 1:1 mapping.  It is a many-to-one mapping.  You can have
lots of different FODs with the same outputHash or even the same
outPath.  For example, perl538.src and perldevel.src use the same
source tarball but a different `version`.

Anyways, I have found the root cause but have run out of time for a
while, so I've added a panic!() to in the spot where we have a logic
bug in order to call it out.

Change-Id: I9766b39cfe2fe7eafec84945b2ad6cc28f9c4b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9364
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-10 14:16:21 +00:00
Adam Joseph
dfca879535 fix(tvix/glue): do not panic if PathInfoService returns Err
Nixpkgs tries to `import` the value returned by
`builtins.unsafeGetAttrPos`, which in our case is the file
`/deep/thought`.  Since that doesn't exist, tvix-glue panics, but
there's no interpreter backtrace to follow.

Let's return an Err instead of panicking.

------------------------------------------------------------------------------
Before:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at src/tvix_store_io.rs:276:58:
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(41580), ...)
Command exited with non-zero status 101

------------------------------------------------------------------------------
After:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: while evaluating this Nix code
   --> [code]:1:1
    |
1   | (import /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src {}).pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:6:78
    |
6   |   runCommand "rocm-llvm-binutils-${llvm.version}" { preferLocalBuild = true; } ''
    |  ______________________________________________________________________________^
7   | |   mkdir -p $out/bin
8   | |
9   | |   for prog in ${lld}/bin/*; do
...   |
27  | |   ln -s ${lld}/bin/lld $out/bin/ld
28  | | ''
    | |__^
note: while evaluating this as native code (coerce_to_string)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:27:9
    |
27  |   ln -s ${lld}/bin/lld $out/bin/ld
    |         ^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:23
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:30
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                              ^^^^^^^^^
note: while evaluating this as native code (throw)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:262:8
    |
262 |     in handler msg;
    |        ^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |________________________________________________________^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |__________^
error[E029]: I/O error: /deep/thought:42: task panicked
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:255:41
    |
255 |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
    |                                         ^^^^^^^^^^^^^^^

Command exited with non-zero status 1
Benchmark: {"pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath":{"kbytes":"26613180","system":"22.35","user":"140.62"}}

Change-Id: I587b57e9e49d1f3ecdc0fc9cf996d179a3548f34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10223
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 10:47:39 +00:00
Florian Klink
d4d1387409 docs(tvix/glue): fix doc-comment reference
This has been renamed to descend_to in cl/9373.

Change-Id: Ia6201fb81c7d4fa953d311451cfff95373549a50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10045
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-16 21:26:54 +00:00
Florian Klink
c8cc31e079 refactor(tvix/glue): move builtins into separate directory
Change-Id: I25b7197458dbfbde8623545dc0a0286eb2744f10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9911
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 15:20:46 +00:00
Florian Klink
a72a1044c2 refactor(tvix/glue): move nix_path handling to helper in tvix-glue
Change-Id: I2327560c4cf0d3f90e253e3c2f47cb29c762461e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9910
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-04 15:18:45 +00:00
Florian Klink
619f1fe88d refactor(tvix/glue): use add_derivation_builtins helper
Change-Id: I284c82612f1c8c81a8b7711fe63c19778ff2fdf0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9909
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-04 15:18:44 +00:00
Florian Klink
3196fe0143 refactor(tvix): move tvix glue code into glue crate
There's various bits and pieces in tvix-cli that use both the store and
evaluator, as well as nix-compat. For example, builtins.derivation, as
well as the reference scanning implementation.

This "glue code" currently isn't accessible from anywhere else, but it'd
be very useful if it were.

Move it out into a `glue` crate, and make `tvix-cli` a consumer of it.

All the KnownPaths setup and passing around, as well as NIX_PATH
handling is also something that should probably be moved into the glue
crate as well, but that's something left for a future CL.

Change-Id: I080ed3d1825ab23790666486840f301f00856277
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 15:18:44 +00:00