Commit graph

1835 commits

Author SHA1 Message Date
Florian Klink
efaff79584 feat(tvix): make clippy stricter
Also run clippy on tests, and enable all features.

Change-Id: Ide9f1bc9f565333072afb918c391c7930b658f41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10234
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
22a669d27e refactor(tvix/castore): address clippy
We match to destructure a single pattern.

Change-Id: I564a3510b4860e90b3315a9639effc48ee88b483
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10233
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
Florian Klink
340242174d refactor(nix-compat/aterm): address clippy
This pushes to a Vec immediately after creation.

Change-Id: I2360b45810475d98ededc1d097fb4cbdeabc576b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10232
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-09 12:58:39 +00:00
Florian Klink
f9ff4fe969 refactor(store/fs): address clippy
Change-Id: I28d60263ef672942940656caa8e0f9d20cc6b7d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10231
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
f4c4e32040 refactor(nix-compat/nixbase32): address clippy
Change-Id: I07b53ead10b0efb31e88c6ae93c124c5f35261d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10230
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
06350445e9 refactor(nix-compat/narinfo): address clippy lints
Change-Id: I00f28624ec0469525e52f408f00caa20d3b701b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10229
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
1bdb90e56d refactor(nix-compat/derivation): address clippy
Change-Id: I5d7057347838f733d7dc1331c7b7ef8d27244412
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10228
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-12-09 12:58:39 +00:00
Florian Klink
d3ecef1a5b refactor(tvix/eval): address clippy lints
Change-Id: Ic2bd4e8291b30ceac9fa0e88a4f56e61ae99b603
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10227
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
Florian Klink
f11c6d610c chore(tvix/nix_cli): drop crate
The only thing this was doing was invoking nix-store --add, which is not
gonna help us populate the tvix-store at all (and we now have
`tvix-store import`).

This is also (rightfully) causing clippy warnings, because of some
fields being unused.

It's more of a skeleton, and rather than shelling out to Nix for some
usecases, we might introduce a "compatible" Nix CLI frontend for a
subset of commands.

Drop this for now, to decrease the noise and confusion.

Change-Id: I2fd399e9320260f08893b685561755af9c7c961c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10226
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
0907420600 refactor(nix-compat/store_path): from_absolute_path to StorePathRef
The only non-test usage was only checking for the error case, and we can
still convert this to an owned StorePath by calling to_owned() on
StorePathRef.

Change-Id: I9f67a759e580c9c429c96896bcdd295392aa5a2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10225
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:35 +00:00
Florian Klink
5855959d87 refactor(nix-compat/derivation): rename input derivation components
Match the naming in parse_input_derivations, call the keys
"input_derivation", and the values "output_names".

Change-Id: I7d1974819028f8ea543dc3ad78afb803ff9db865
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10224
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:20:51 +00:00
Adam Joseph
5fd42a9e70 feat(tvix/eval): impl DoubleEndedIter for OwnedAttrsIterator
Change-Id: I4bd85dbe9c27047f4abbdeff4e2b796e9bcab3a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-09 11:35:19 +00:00
Adam Joseph
36c3e646e1 chore(store/src/tests): make clippy shut up
Change-Id: I6dfceaa32a8c01f8395b7889ae19847fabf95ed3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10214
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 10:51:10 +00:00
Adam Joseph
dfca879535 fix(tvix/glue): do not panic if PathInfoService returns Err
Nixpkgs tries to `import` the value returned by
`builtins.unsafeGetAttrPos`, which in our case is the file
`/deep/thought`.  Since that doesn't exist, tvix-glue panics, but
there's no interpreter backtrace to follow.

Let's return an Err instead of panicking.

------------------------------------------------------------------------------
Before:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at src/tvix_store_io.rs:276:58:
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(41580), ...)
Command exited with non-zero status 101

------------------------------------------------------------------------------
After:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: while evaluating this Nix code
   --> [code]:1:1
    |
1   | (import /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src {}).pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:6:78
    |
6   |   runCommand "rocm-llvm-binutils-${llvm.version}" { preferLocalBuild = true; } ''
    |  ______________________________________________________________________________^
7   | |   mkdir -p $out/bin
8   | |
9   | |   for prog in ${lld}/bin/*; do
...   |
27  | |   ln -s ${lld}/bin/lld $out/bin/ld
28  | | ''
    | |__^
note: while evaluating this as native code (coerce_to_string)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:27:9
    |
27  |   ln -s ${lld}/bin/lld $out/bin/ld
    |         ^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:23
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:30
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                              ^^^^^^^^^
note: while evaluating this as native code (throw)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:262:8
    |
262 |     in handler msg;
    |        ^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |________________________________________________________^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |__________^
error[E029]: I/O error: /deep/thought:42: task panicked
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:255:41
    |
255 |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
    |                                         ^^^^^^^^^^^^^^^

Command exited with non-zero status 1
Benchmark: {"pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath":{"kbytes":"26613180","system":"22.35","user":"140.62"}}

Change-Id: I587b57e9e49d1f3ecdc0fc9cf996d179a3548f34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10223
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 10:47:39 +00:00
Adam Joseph
f4460bd612 chore(nix_cli): make clippy shut up
Change-Id: I0c243baf4deecfd81bd939da86d685e2cbc59d69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10215
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-09 09:25:24 +00:00
Adam Joseph
2132bbcb47 fix(tvix/nar-bridge): fix FOD hash
The golang mothership seems to be monkeying with hashes again.

Change-Id: I7430b4cde84fa51be2b572fba02e3567864bb87a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10209
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 09:24:53 +00:00
Adam Joseph
0e2633048e feat(tvix): run crate2nix generate in CI
This runs `crate2nix generate` in CI and then runs `depotfmt` on the
result to ensure that our machine-generated code is really, really
readable and pretty.  Then it checks that the result of all that
is identical to the committed Cargo.nix.

A self-hashing FOD is used to allow network access.

No magic hashes are involved.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I68ec5003dbc6a40894a5a4d6e902f138c99f6719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10194
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-07 18:43:46 +00:00
Aspen Smith
0b524890e9 refactor(tvix): Make benchmark script directory-agnostic
Allow running the benchmark script from any directory - primarily so
Windtunnel can point to the /tvix josh workspace rather than the depot
overall

Change-Id: Ie5fc3ef995bf8114277298ae5c5010e6a0bf13ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10205
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2023-12-06 16:19:46 +00:00
Adam Joseph
49b34183e3 feat(tvix/eval): rewrite Thunk::force() in nonrecursive form
This commit rewrites Thunk::force() so that it is not (directly)
self-recursive.  It maintains a Vec of all the
previously-encountered thunks which point to the one it is currently
forcing, rather than recursively calling itself.

Benefits:

- Short term:

  This commit saves the cost of a round-trip through the generator
  machinery for the generators::request_force() which is removed by
  this commit.

- Medium term:

  Once a similar transformation has been applied to nix_cmp(),
  nix_add(), nix_eq(), and coerce_to_string(), those four functions,
  along with Thunk::force(), will make non-tail calls only to each
  other.  They can then be merged into a single tail-recursive
  function which does not use the generator machinery at all:

    enum Task { Cmp, Add, Eq, CoerceToString, Force};

    fn Value::walk(task:Task, v1:Value, v2:Value) {
      // ...

- Long term:

  The long-term goal here is to use generators **only for builtins**
  and [Marionette]-style remote control of the VM.  In other words:
  use `async` for things that actually involve concurrency.  Calls
  from the VM to builtins can then be blocking calls, because even
  cppnix will overflow the stack if you make a MAX_STACK_DEPTH-deep
  recursive call which passes through a builtin at every stack frame
  (e.g. `{ func = builtins.sort (a: b: ... func ...) ...}`).

  This way the inner "tight loop" of the interpreter doesn't pay the
  costs of `async` and generators.  These costs manifest in terms
  of: performance, complex nonlocal control flow, and language
  impediments (async Rust is a restricted subset of real Rust, and
  is missing things like traits).

[Marionette]: https://firefox-source-docs.mozilla.org/testing/marionette/Intro.html

Change-Id: I6179b8abb2ea0492180fcb347f37595a14665777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10039
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-06 06:53:01 +00:00
Aspen Smith
8135a8d38c fix(tvix/eval): Return error rather than panicking on bad substring
If builtins.substring is invoked with (byte!!) offsets that aren't at
codepoint boundaries, return an error rather than panicking. This is
still incorrect (see b/337) but pushes the incorrectness forward a step.

Change-Id: I5a4261f2ff250874cd36489ef598dcf886669d04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10199
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-05 23:12:23 +00:00
Florian Klink
41235bf908 refactor(tvix): move src into let binding
Change-Id: Ida2a3ac722fb2445745759323975884dfeef3e87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10193
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-05 21:55:09 +00:00
Aspen Smith
6edcb03cb6 feat(tvix): Add script for running benchmarks in Windtunnel
Currently this just uses a Docker container, which is gross but works
fine for now since we don't have the ability to build benchmarks in nix
as of cl/7538

Change-Id: I48e317f44bc2c73533d7663403786a3a37c7952f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10189
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
2023-12-05 17:46:26 +00:00
Florian Klink
dfb48dcade fix(tvix/nar-bridge): drop pathinfoservice
This now exists in tvix-store directly, as NixHTTPPathInfoService, and
contrary to this version, also validates signatures.

Change-Id: Ib6ca161e40d627b7d9741839fc849f2392f422da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10155
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-28 16:08:59 +00:00
Florian Klink
563886c3de feat(tvix/store/pathinfosvc/nix_http): allow configuring pubkeys
This allows setitng the trusted-public-keys URL parameter to a
(whitespace-separated) list of public keys.

NARInfo files retrieved need to contain a valid signature.

Change-Id: Ifd6580b723cbae3182e9cadfa54f1ca2b41d6599
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10153
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
4297e33d94 feat(tvix/store/pathinfosvc): add signature verification
Introduce an Option<Vec<narinfo::PubKey>>, configurable with a
`set_public_keys` method.

If set, this configures NixHTTPPathInfoService to validate signatures.

Change-Id: I157c5e13c41fc9bfd40b0655381fb4cf33900868
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10152
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-28 16:08:59 +00:00
Florian Klink
5fc737b02e feat(nix-compat/narinfo): add PubKey::verify(fp, sig)
This makes it easy for each PubKey to check if a given Signature is
correct for a given fingerprint.

Change-Id: I56e6211d133f74f390fd1ae3ae799eef12221904
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10151
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
6af67af76e feat(nix-compat/narinfo): add PubKey
This represents a ed25519 public key and "name".
These are normally passed in the `trusted-public-keys` Nix config option,
and consist of a name and base64-encoded ed25519 pubkey, separated by a `:`.

Change-Id: I9ab4b3e0e5821805ea6faf2499626630fc5a3f0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10150
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
e7a86273b1 refactor(nix-compat/narinfo/signature): rename Error type
Just call this Error, we can infer from the package what error this is.

Change-Id: I5df25d2873ec739c49c08804f35562c84c222e06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10149
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
7e8719be91 feat(tvix/store/pathinfosvc/nix_http): check Nar{Size,Hash} matches
Ensure the initially communicated NarHash/NarSize from the NarInfo
matches what we read, and don't return a PathInfo message if there's
a mismatch.

Also move the buffering layer around a bit.

Change-Id: I68c60ecfaf0f9cd5edacea648437ecb0c9729251
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10148
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
fce9c0e99e fix(tvix/store/fs): don't panic on PathInfoService error
An error in the PathInfoService request can appear in case the
underlying request returns an error.

We shouldn't panic and bork the fuse mount, but instead return an IO
error.

Change-Id: I2daeae629e1627d06adcd7b82ddb76c50c602212
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10154
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-28 14:54:56 +00:00
Vincent Ambo
360c98ce38 chore(3p/sources): bump channels & overlays (2023-11-27)
* picked avrdude from stable channel
* removed override for texlive, as the upstream fix is merged
* picked awscli2 from stable channel
* bump tdlib to 1.8.21 (new minimum for telega.el)
* tvix/turbofetch: switch to nixpkgs-native mechanism for
  CARGO_MANIFEST_LINKS (whatever that is)

Change-Id: Ic695721b5ca750b89d21cab7a257e1db682b23c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10083
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-27 21:16:46 +00:00
Florian Klink
0415bc6fd2 fix(nix-compat/narinfo/signature): validate name field
We should restrict this to alphanumeric mostly, and we definitely don't
want newlines.

Not entirely sure about the exact additionally allowed characters
outside of alphanumeric, but this can always be extended further.

Change-Id: I1357e79e553f2df2fa97792889f63f0f35d50ed5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10147
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-27 15:12:13 +00:00
Florian Klink
b7de931cc6 fix(nix-compat/narinfo): don't panic trying to parse signatures
BASE64.decode_mut panics if we're passing data that has the wrong size.
Do the size check first and error out there.

Also update the error, and talk about b64-encoded sizes.

Change-Id: I290f80a37d48526a30bf1df9d1d9fe34865008eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10146
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-11-27 15:12:13 +00:00
Florian Klink
dfaaf41cef refactor(nix-compat): use ed25519_dalek::SIGNATURE_LENGTH
No need to hardcode magic numbers here, we have a constant for that.

Change-Id: I67b671c0c4bb7c3bfb001e9c36499f31873ee717
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10145
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-27 15:11:41 +00:00
Vincent Ambo
e65fa82d74 refactor(tvix/eval): use or_default helper in entry API
This fixes a future clippy lint.

Change-Id: Ic830e94ef23595580c1037f10878c76bbb546dd9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10110
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-11-25 15:48:46 +00:00
Vincent Ambo
5ffb997864 fix(tvix): ensure PartialOrd/Ord agree for StorePath & NixString
This fixes a *future* clippy lint:
https://rust-lang.github.io/rust-clippy/master/index.html#/incorrect_partial_ord_impl_on_ord_type

In essence, because the implementation of *both* Ord and PartialOrd
implies that ordering is not partial, all results of PartialOrd should
simply be those of Ord. This is to avoid subtle bugs in future
refactorings.

Change-Id: I8fc6694010208752dd47746a2aaaeca0c788d574
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10109
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-25 15:48:46 +00:00
Adam Joseph
512346ba0b refactor(tvix/eval): add ThunkRepr::is_forced()
Change-Id: I4eab5c81fb82337da06327248845cd2f3a4490d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10038
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-11-25 02:55:57 +00:00
Adam Joseph
b5a15989cd feat(tvix/eval): add Thunk::unwrap_or_clone()
This commit adds Thunk::unwrap_or_clone(), which uses
Rc::try_unwrap() to avoid cloning the Value out of a an Rc which has
only one strong reference.

Change-Id: Icacefe0c823dcddf046d90c0c5cd5ed59fe976d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10037
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-11-25 02:55:56 +00:00
sterni
875bb26fc3 fix(tvix/castore): correctly flag unreachable code
Change-Id: Id09afa4b77c3c70fb5695f253f6df4aa88b61e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10113
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-24 23:36:15 +00:00
sterni
c5961e7774 docs(tvix/eval): optimization potential for inherit (from) exprs
Change-Id: Ibddaa111a5b7a86c42dbe153ae8e53f9a5601a54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10112
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-11-24 23:36:15 +00:00
Florian Klink
6b685ec4a5 feat(tvix/store): add as_narinfo() for PathInfo
This allows seeing a PathInfo as a nix_compat::narinfo::NarInfo<'_>.

It doesn't allocate any new data, but the NarInfo<'_> view allows us to
access things like signature verification, or rendering out
(alternations of this) as strings.

Change-Id: Id0d8d7feeb626ee02c3d8a4932f24ace77022619
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10108
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
639cca3e22 feat(nix-compat/narinfo/signature): add new() constructor
This is useful when creating a new Signature struct where the individual
elements are already parsed.

Change-Id: Ie33c66287641951e7a030aaa1e7ff0a86b2628ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10111
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
c44cbc852a refactor(tvix/store): impl From<&nar_info::Ca> for nixhash::CAHash
Change-Id: I637a4cff5a5ca29c4d86e0b76a2f20f8741f5628
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10107
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
1e04f60800 refactor(tvix/store): impl From<nixhash::CAHash> for nar_info::Ca
Change-Id: Iaa68044d3b469f15a932aa3b59548505eaa6b8bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10106
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:11:49 +00:00
Florian Klink
671206a63f refactor(tvix/nix-compat): move from_name_and_digest to StorePathRef
We can simply use .to_owned() on that thing afterwards if we want to
construct an owned StorePath.

Change-Id: I0f3e2e4434b99ee522f2a7dbfa391e13a987479c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10105
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:11:49 +00:00
Florian Klink
eb84898c17 feat(nix-compat/narinfo): drop .drv from Narinfo.deriver field
We always know this needs to end with a .drv, and fail parsing if it
doesn't, so there's no need to hang onto these 4 bytes.

This will make it much easier to synthesize a NarInfo<'_> later on from
a PathInfo proto, because we don't have to make this ".drv" appear out
of thin air.

Change-Id: Id95e7fd937d7c9a420a39b5a4bab73985640ca3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10084
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-11-22 18:24:04 +00:00
Florian Klink
ef8a8af0bf refactor(tvix/nix-compat): cleanup parse_{ca,hash} and fmt structs
These were used to format to and parse from strings.

Move this to the CAHash and NixHash structs directly, and be explicit in
the name about which encoding for digests is used.

For output path calculation, nix encodes the nixpaths in hex, but for
writing out NARInfos, it's using nixbase32.

Change-Id: Ia585a76a3811b2609e7ce259fda66a29403b7e07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10079
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-22 17:54:59 +00:00
Florian Klink
a834966efd feat(tvix/nix-compat/narinfo): add fingerprint
This adds support to compute the fingerprint string, which is what's
ed25519-signed in binary caches.

Change-Id: I8947239c609896acfd7261f110450014bedf465a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10080
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 22:34:59 +00:00
Florian Klink
bb18556bf3 feat(tvix/nix-compat): add narinfo::Signature::verify
This adds support for verifying signatures found in NARInfo files.

This still needs to be hooked together with the nix+http[s] backend.

Change-Id: Ic1c8ddbdecfb05cefca2492808388b0f7f3f2637
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10081
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-19 22:06:07 +00:00
Florian Klink
a5749fada5 refactor(nix-compat/narinfo): move signature into separate file
Change-Id: Ic257475e2afebf059c5317c1cc5b04ba63d5d318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10078
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 21:58:02 +00:00
Florian Klink
6039b97b55 refactor(tvix/nix-compat): move narinfo into separate mod
Change-Id: Id85f979e46946da0345483cbbc6de3dd29c94c63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10077
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 21:58:02 +00:00
Florian Klink
be48ba75ab feat(tvix/store/pathinfoservice): implement NixHTTPPathInfoService
NixHTTPPathInfoService acts as a bridge in between the Nix HTTP Binary
cache protocol provided by Nix binary caches such as cache.nixos.org,
and the Tvix Store Model.
It implements the [PathInfoService] trait in an interesting way: Every
[PathInfoService::get] fetches the .narinfo and referred NAR file,
inserting components into a [BlobService] and [DirectoryService], then
returning a [PathInfo] struct with the root.
Due to this being quite a costly operation, clients are expected to
layer this service with store composition, so they're only ingested
once.
The client is expected to be (indirectly) using the same [BlobService]
and [DirectoryService], so able to fetch referred Directories and Blobs.
[PathInfoService::put] and [PathInfoService::nar] are not implemented
and return an error if called.

This behaves very similar to the nar-bridge-pathinfo code in nar-bridge,
except it's now in Rust.

Change-Id: Ia03d4fed9d0657965d100299af97cd917a03f2f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10069
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 13:19:24 +00:00
Florian Klink
4e9e4b19ef refactor(tvix/castore/blobservice): rm AsyncBufRead from BlobReader
There's no need to already require this to be buffered here.

Change-Id: Ib9a11b194e0754d87ab8d2ef0b8cb0f4edc01229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10074
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 12:54:19 +00:00
edef
a11abc02e2 feat(nix-compat/nar/reader): provide passthrough buffered I/O
Allow taking advantage of the buffer of the underlying reader to avoid
unnecessary copies of file data.

We can't easily implement the methods of BufRead directly, since we
have some extra I/O to perform in the final consume() invocation.

That could be resolved at the cost of additional bookkeeping, but this
will suffice for now.

Change-Id: I8100cf0abd79e7469670b8596bd989be5db44a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10089
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 09:53:10 +00:00
edef
785ff80c8b fix(nix-compat/nar/reader): require BufRead
We rely on being able to make small reads cheaply, so this was already
an implicit practical requirement. Requiring it explicitly removes a
performance footgun, and makes further optimisations possible.

Change-Id: I7f65880a41b1d6b5e6bf2e52dfe47d4c49b34bcd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10088
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-19 09:53:10 +00:00
edef
2eaee1d48e fix(nix-compat/store_path): valid names ⊊ UTF-8
We don't need to validate UTF-8 separately, since valid names are
a strict subset of ASCII, and therefore a strict subset of UTF-8.

Change-Id: I3261bf0efe3480b5b315074efafcf5e47a6c5a65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10087
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-19 09:53:10 +00:00
edef
9ad7cc629e fix(tvix): patch futures::AsyncBufReadExt::fill_buf
This fixes EOF handling for buffered readers.

Link: https://github.com/rust-lang/futures-rs/pull/2801
Change-Id: Ie98ca6a3e1de38500b0195e9b62511501acb1d2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10086
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 08:35:38 +00:00
edef
479f26f46c chore(tvix): upgrade futures to 0.3.29
Change-Id: I8fd63be3cbec8766fd6d72cd9271989a19774816
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10085
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 08:35:38 +00:00
Florian Klink
8beb38b50c refactor(tvix/store/fs): simpllify read
We can just use take(size) to restrict reading to that as a max.

Change-Id: I0fbda74e4fb98ffeababae86a325233416029acf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10072
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-18 23:07:24 +00:00
Florian Klink
eda5d4da37 feat(tvix/store): From<&nix_compat::...::NarInfo<'_>> for PathInfo
This allows converting from the NarInfo falling out of the NarInfo
parser (which is a bit annoying to handle due to lifetimes) to the
PathInfo proto struct.

The narinfo field, containing most of the data from the original
NARInfo file, as well as the references (bytes) are populated.
The node field is not populated, because it requires ingesting the NAR
itself to describe the root node.

Change-Id: I9c04dd6ad4cae556b455188a4255e34b4f6443c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10067
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-18 19:38:15 +00:00
Florian Klink
e32c2070e4 refactor(tvix/nix-compat): no impl <StorePathRef<'_>> for StorePath
This suggests it's cheap to convert around, but name actually does
allocate.

Move to a `to_owned(&self) -> StorePath`, to better signal that this
does allocate.

Change-Id: Ifaf7c21599e2a467d06e2b4ae1364228370275db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10066
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-18 19:37:04 +00:00
Ryan Lahfa
df63b719ac feat(tvix/castore): fix tracing instrument in MemoryBlobService
Change-Id: Iedba57e8b3e1a44f14f5baa1e981275d4b02eb56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10070
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-18 19:23:55 +00:00
Florian Klink
68e473ed6b feat(tvix/castore): impl From<std::io::Error> for Error
Make it less annoying to convert from io::Error to this. We already have
one direction, doesn't hurt to have the other too.

Change-Id: I9fe2c6da608c9d54910ee8c397572aadb1d90d99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10068
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-18 19:23:47 +00:00
Florian Klink
ebfe456251 refactor(tvix/castore/tonic): use match in channel_from_url
Having random if blocks and returning from them is error-prone.

Also, turns out we only need the unprefixed scheme in the fallback case,
so move it down to there.

Change-Id: Ifcb09279c963f8a39e0dbabe145990263f3d7cf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10041
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-17 14:01:26 +00:00
Florian Klink
d4d1387409 docs(tvix/glue): fix doc-comment reference
This has been renamed to descend_to in cl/9373.

Change-Id: Ia6201fb81c7d4fa953d311451cfff95373549a50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10045
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-16 21:26:54 +00:00
Florian Klink
4e5165d0a1 refactor(tvix/castore/utils): drop unused DuplexStreamWrapper
This wasn't used at all, let's remove it.

Change-Id: I426e3d93c32ebe65247ae5cf8d05b5bf686be2d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10044
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-11-15 22:04:04 +00:00
Florian Klink
30e0c32066 refactor(tvix/castore/tonic): make async, support wait-connect=?
This moves the sync `channel::from_url` to a async
`tonic::channel_from_url`. It now allows connecting non-lazily if `wait-
connect=1` is set in the URL params.

Also, make the pingpong tests for blobsvc and directorysvc use the wait-
connect=1 codepath.

Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
c83841d3a1 refactor(tvix/castore): remove DirectoryService::from_url
Make directoryservice::from_addr use the more specific constructors.

Change-Id: I9fee2afed77692505988d631d9fe246d9843d25a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10029
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
840bee1e97 refactor(tvix/castore/blobsvc): remove BlobService::from_url
Make blobservice::from_addr use the more specific constructors.

Change-Id: Id9637e279d6910ce6d92ff0086a984be5c65a8c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10028
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
25eca8af32 refactor(tvix/store/pathinfosvc/from_addr): use test_case
All we do is constructing some strings, and checking if from_addr
succeeds or not.

This can be written in a much more concise way using test_case.

Use lazy_static to provide temporary directories.

Also add some more grpc-related test cases.

Change-Id: Ia310dd01f617f7628f1e7e21304ac70da2ab3534
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10027
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-11-15 06:43:31 +00:00
Florian Klink
756290a4c0 refactor(tvix/store/pathinfosvc): inline SledPathInfoSvc::from_url
Change-Id: I0d905228df086a422bb30322add7236ca41e807b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10026
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
f57fd16d6e refactor(tvix/store/pathinfosvc): inline GRPCPathInfoSvc::from_url
Change-Id: Ib53b5525ae13c276e61b7f564673b7c6144ffc0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10025
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
362117fbf5 feat(tvix/castore/src/channel): move from_url tests
These gRPC PathInfoService tests were actually not too useful in here,
what we're mostly testing is the channel construction, so move it to
there.

Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-11-15 06:43:31 +00:00
Florian Klink
9aef3cfc89 refactor(tvix/store/pathinfosvc): inline MemoryPathInfoSvc::from_url
Change-Id: If27eb518d372f4004b7b38fc765a42957f2a6b50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10023
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
Florian Klink
8111caebc2 refactor(tvix/store): remove from_url from PathInfoService trait
We don't gain much from making this part of the trait, it's still up to
`tvix_store::pathinfoservice::from_addr` to do most of the construction.

Move it out of the trait and into the specific *Service impls directly.

This allows further refactorings in followup CLs.

Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 06:43:31 +00:00
edef
8d02928b14 feat(tvix/tools/narinfo2parquet): init
Convert turbofetch output to queryable Parquet.

Change-Id: I076f5a431f8aab8cfe7d973bdc9fe019cebde111
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9989
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:35:04 +00:00
edef
45de341794 chore(nix-compat/store_path): use hex_literal
Change-Id: Id093a0131aa7e3ac532daffbf5a883ca213c83ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9996
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 19:35:04 +00:00
edef
2997c1c304 feat(nix-compat/narinfo): turn flags into bitfields
Change-Id: I8b95723444013e97bc6ec8d282c7135b1aede114
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9987
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
1df53a5fcd feat(nix-compat/narinfo): permit out-of-order references
This appears in the cache.nixos.org dataset.

Change-Id: I2eadafe8441e0132a448828026553da2dc7c12aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9994
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 19:05:36 +00:00
edef
24187cfeec feat(nix-compat/narinfo): accept hex-encoded NarHash
This appears in the cache.nixos.org dataset.

Change-Id: I35921f7ef148f6681081a4e371abb8c9cc98854d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9993
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
4218e4dc0e feat(nix-compat/nixbase32): use data_encoding::DecodeError
Rather than having our own error type, just make decoding errors use
the same common error type.

Change-Id: Ie2c86972f3745c695253adc3214444ac0ab8db6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
7e317cfded feat(nix-compat/narinfo): default compression to bzip2
Change-Id: Ibdebc0a69672c00026c8748b93fcd2a83a0e69a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9992
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 19:05:36 +00:00
edef
407e5a53ed feat(nix-compat/narinfo): track presence of unknown fields
Change-Id: Ia3f8a86209a0045ff98322b56a21ae20220fbe99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9991
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
8694694b74 feat(nix-compat/narinfo): permit non-SHA256 CAHash::Nar
This appears in the cache.nixos.org dataset.

Change-Id: I055b60b9950a1a6a36c1b0576b957e11e1d4264b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9990
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
bdda10a2f5 feat(tvix/tools/turbofetch): init
Change-Id: I2efa6f94f57e812c52371256a4e62d1d54ff5057
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9925
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
Florian Klink
9cd2e92065 refactor(tvix/store/fs): mv sparse -> populated directories
Do this upgrade whenever someone is actually interested in the children
of a directory, but that directory doesn't contain a more detailed
listing. This is much more predictable, and removes a bunch of confusing
code from the inode tracker itself.

Change-Id: Ib3a13694d6d5d22887d2d04ae429592137f39cb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9982
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 20:19:38 +00:00
Florian Klink
a778b855d2 refactor(tvix/store/fs): simplify name_in_root_to_ino_and_data
Have it return libc::ENOENT errors rather than an Option<…>.

Also avoid having to traverse inode_data multiple times, by synthesizing
the Arc<…> on our own in the insert case. In that case, the data is
quite small, so cloning it is faster than traversing a second time.

Change-Id: I7ab14bac8bb23859ed8d166a12070d4f4749b6d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9981
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 14:52:10 +00:00
Florian Klink
3ae48465fa refactor(tvix/store/fs): move code to get_directory_children helper
As already established in the two previous CLs, these two pieces of code
where doing the same.

Move to a get_directory_children helper.

Change-Id: Id6876f0c34f3f40a31a22d59a2cdbfef39e2d8de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9980
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-11-07 14:52:10 +00:00
Florian Klink
9e6d89983a refactor(tvix/store/fs): reduce write lock, return children
Very similar to the previous CL

Change-Id: I0df07ddca742b7b9485d48771c8d295dc3aa7136
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9979
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 14:52:10 +00:00
Florian Klink
2405399580 refactor(tvix/src/fs): reduce write lock, avoid inode_tracker lookup
Code after this big match block only cares about parent_digest and
children, so there's no need to do another inode_tracker.get in there.

This also allows removing another if let block, right after, as we don't
need to destructure parent_data anymore.

Change-Id: I68fbbe3304194670caee5a453722369afa4e77ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9978
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 14:52:10 +00:00
Florian Klink
f3a240bf86 refactor(tvix/store/fs): move inode for store_path lookup to helper
This makes it much harder to keep the read lock around for too long, and
the code a bit easier to understand.

Change-Id: I7d99c85cadd433cad444b8edd34e2c43d7eaf5a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9977
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 14:52:10 +00:00
edef
a8e7f4eadb fix(tvix/castore): B3Digest::{to_vec -> as_slice}
Not a single call site actually makes use of the Vec.

Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-07 12:35:41 +00:00
Florian Klink
9f5b1213f9 feat(tvix/nix-compat): add drvfmt
This small tool formats A-Term in a more readable format. It's a lossy
conversion for non-valid UTF-8 environment values.

Change-Id: I65a51054d7faf528321bc2d9fc4425180a7813f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9970
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-07 11:27:08 +00:00
Vincent Ambo
7c32d85455 docs(tvix/eval): document where EvalIO methods are used
Change-Id: I335f2ba4420973861c2a22125995b45a34d3608c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9969
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-06 09:06:26 +00:00
Florian Klink
c86de1b2c4 chore(tvix/nar-bridge): bump castore-go
Change-Id: I81814ae326fb90ab50201cd06e3d9c0a3b15aec5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9958
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-06 08:34:14 +00:00
Florian Klink
72760b7716 chore(tvix/store-go): bump castore-go
Change-Id: I740251785f4ffb6c9305f430e0f2c3418dfaa15d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9957
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 20:32:34 +00:00
Florian Klink
a9329ff811 feat(tvix): add clippy target in CI
This will invoke `cargo clippy` on the codebase, ensuring we spot things
early on.

Relates to b/321.

Change-Id: Ifba13e5bef995d33b24346c6cee134e84d6566c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9106
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 20:28:53 +00:00
Vincent Ambo
f325456cba chore(tvix): bump proptest dependency
This *might* contain a fix for a clippy lint thrown by that crate.

Relates to b/321.

Change-Id: Ia7ebd3e26e0feb8bcc7a6c811b1e583f9016fd9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9966
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 20:28:38 +00:00
Vincent Ambo
87d3fac676 chore(tvix): add missing clippy attributes & config
For cases where clippy lints don't apply to us, or something is
misfiring, add appropriate configuration.

Relates to b/321.

Change-Id: I0af453910b4a4112bf685b2a8e9a73de10ec87ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9965
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-05 20:28:38 +00:00
Vincent Ambo
2e1399698b refactor(tvix/eval): use IntoIterator trait for owned NixAttrs iter
Uses the standard library IntoIterator trait for the construction of
our iterators. Clippy complains about duplicating this.

While doing this, I opted to rename the `IntoIter` type into something
that is more useful to users, in case somebody ends up working with
these manually.

This fixes a clippy lint, and is related to b/321.

Change-Id: I851fde0d7b8b38d182343a0fd6d9f8dd2a33ee11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9963
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 20:28:38 +00:00