Commit graph

150 commits

Author SHA1 Message Date
Simon Hauser
825d498908 feat(tvix/tracing): introduce common tvix-tracing crate
Introduce a new common crate that contains tracing boilerplate which then
can be used in the cli, tvix-store and tvix-build crates.
It has otlp as an optional feature, which is currently only used by
tvix-store.

Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-10 16:35:08 +00:00
Aspen Smith
d0ab3c8d15 feat(tvix): Switch to jemalloc
Switch tvix to using jemalloc as the default global allocator on
supported (eg, non-msvc) platforms. This gives a pretty reasonable
performance boost basically for free:

int                     time:   [24.361 µs 24.386 µs 24.418 µs]
                        change: [-19.355% -18.859% -18.527%] (p = 0.00 < 0.05)
                        Performance has improved.

merge small attrs       time:   [37.201 µs 37.328 µs 37.442 µs]
                        change: [-24.609% -24.266% -23.982%] (p = 0.00 < 0.05)
                        Performance has improved.

merge large attrs with small attrs
                        time:   [20.030 ms 20.135 ms 20.251 ms]
                        change: [-4.2811% -3.2549% -2.3807%] (p = 0.00 < 0.05)
                        Performance has improved.

hello outpath           time:   [967.91 ms 974.07 ms 983.33 ms]
                        change: [-5.5008% -4.4552% -3.4099%] (p = 0.00 < 0.05)
                        Performance has improved.

Change-Id: I6c6e6e3295ec2fca01ea28dc37bcb201cd811767
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10851
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-06 10:29:33 +00:00
Aspen Smith
c2f649f62e feat(tvix/repl): Add a command to recursively print
Add a command, :p, to evaluate an expression and recursively print the
result, as if `--strict` had been passed on the command line.

Demonstration of this working:

    ❯ cargo r --bin tvix
        Finished dev [unoptimized + debuginfo] target(s) in 0.27s
        Running `target/debug/tvix`
    tvix-repl> { x = (x: x) 1; }
    => { x = <CODE>; } :: set
    tvix-repl> :p { x = (x: x) 1; }
    => { x = 1; } :: set

Change-Id: I1a81d7481160c30d2a4483c6308e25fa45f2dfdf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11738
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-01 09:04:59 +00:00
Aspen Smith
4a489d930c feat(tvix/repl): Add a help command
Add a command to display help for the REPL, which can be either :? or
:h.

Change-Id: Ifdfd8c31130ca5afcde05a4c4276b768eb54c06f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11737
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-01 09:04:59 +00:00
Aspen Smith
55ec06b26b feat(tvix/repl): Implement :q, to quit
Change-Id: Ib3b314b21f4d8c30bfd674e79771179d51d4f2e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11736
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-01 09:04:59 +00:00
Aspen Smith
70c4b512c2 refactor(tvix/repl): Abstract out REPL command handling
Prepare for introducing additional REPL commands by splitting out
the *parsing* of the repl commands, which returns a new ReplCommand
type, from the actual *handling* of the commands.

Change-Id: If81a53c1e2d90204d26ce3bb2ea9eebf7bb3fd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11734
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-01 09:04:59 +00:00
Aspen Smith
23589a1db3 refactor(tvix): Break out REPL into its own module
In preparation for adding some new functionality to the tvix REPL, break
it out into its own module.

Change-Id: I4fb78320e92562e3474a3724536cb22c1d893e57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11733
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-01 09:04:59 +00:00
Aspen Smith
96ce9aea04 feat(tvix/repl): Support multiline input
Transparently support multiline input in the Tvix REPL, by handling the
UnexpectedEOF error returned by the parser and using it to progressively
build up an input expr over multiple iterations of the REPL's outer
loop.

This works quite nicely:

❯ cargo r --bin tvix
   Compiling tvix-cli v0.1.0 (/home/aspen/code/depot/tvix/cli)
     Finished dev [unoptimized + debuginfo] target(s) in 1.72s
     Running `target/debug/tvix`
tvix-repl> { foo
         >   =
         >   1;
         > }
=> { foo = 1; } :: set
<press up arrow>
tvix-repl> { foo
  =
  1;
}
=> { foo = 1; } :: set

Change-Id: Ib0ed4766b13e8231d696cdc27281ac158e20a777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11732
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-01 09:04:59 +00:00
Florian Klink
9a704acda5 chore(tvix): drop json feature for tracing-subscriber
Drop the (unused) feature flags here, and get rid of some crate
dependencies.

Change-Id: Id64852b498725467d56abb12b548301cfba6a760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11678
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-16 21:05:51 +00:00
Florian Klink
14766cfe1d refactor(tvix/store): drop calculate_nar from PathInfoService
This shouldn't be part of the PathInfoService trait.

Pretty much none of the PathInfoServices do implement it, and requiring
them to implement it means they also cannot make use of this calculation
already being done by other PathInfoServices.

Move it out into its own NarCalculationService trait, defined somewhere
at tvix_store::nar, and have everyone who wants to trigger nar
calculation use nar_calculation_service directly, which now is an
additional field in TvixStoreIO for example.

It being moved outside the PathInfoService trait doesn't prohibit
specific implementations to implement it (like the GRPC client for the
`PathInfoService` does.

This is currently wired together in a bit of a hacky fashion - as of
now, everything uses the naive implementation that traverses blob and
directoryservice, rather than composing it properly. I want to leave
that up to a later CL, dealing with other parts of store composition
too.

Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 13:33:59 +00:00
Aspen Smith
cff903caff feat(tvix/cli): Use the same TvixStoreIO for the full runtime
This specifically allows subsequent fetch calls in the REPL to avoid
fetching the same path twice (because they share the same
PathInfoService).

Change-Id: Ieda089080174913a8c014f13d6852cac599a7e17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11484
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-05-09 13:57:02 +00:00
Florian Klink
9401cccf84 feat(tvix/cli): change log format to compact, add RUST_LOG support
This brings feature parity with the tvix-store CLI command, at least as
far as logging (not tracing) is concerned.

Alternative to cl/11482, RUST_LOG seems to be the more canonical way to
influence this, and is consistent with how it's done in tvix-store.

Change-Id: I923a0b0ae55dc49af7efdacdcf5b1f24e561b3c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11527
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 15:10:58 +00:00
Florian Klink
4124f0e679 refactor(tvix/cli/default.nix): make eval tests and benches derivations
This makes the eval tests and benchmarks standalone Nix derivations and
readTree targets:

```
nix-build -A tvix.cli.eval-nixpkgs-cross-hello-outpath
nix-build -A tvix.cli.benchmark-nixpkgs-attrnames
```

Even without doing any fetches, We need to set `SSL_CERT_FILE`, so
reqwest is able to load its CA roots.

Change-Id: Ib45282d01044165c7816391adbeeb26334f8e924
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11493
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-22 13:15:22 +00:00
Florian Klink
8f255d3e64 chore(tvix/cli): remove unused dependency
Change-Id: I6a6cabde22d28bb9afc391b9bcd896a999090366
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11475
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-19 19:22:10 +00:00
Florian Klink
283c3f0de4 fix(tvix): allow trace level log messages in debug builds
We had max_level_debug, not max_level_trace, so it was not possible to
have `RUST_LOG` emit trace-level values (at least for our crates).

release_max_level_info still controls the level for release builds, so
that's not affected.

Change-Id: I1f127ab63f13b1622d6616e06759a9414d9bb201
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11208
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-03-20 12:17:42 +00:00
aspen
d74c68025b revert(tvix): don't use Rc::clone explicitly
This reverts commit d9565a4d0a.

Reason for revert: this was intentional - putting Rc::clone instead of
.clone is a common Rust idiom, and makes it explicit that we're cloning
a shared reference, not an underlying resource

Change-Id: I41a5f323ee35d7025dc7bb02f7d5d05d0051798d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10995
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 18:36:11 +00:00
Ryan Lahfa
7388078630 feat(tvix/eval): implement builtins.filterSource
We add a new set of builtins called `import_builtins`, which
will contain import-related builtins, such as `builtins.path` and
`builtins.filterSource`. Both can import paths into the store, with
various knobs to alter the result, e.g. filtering, renaming, expected
hashes.

We introduce `filtered_ingest` which will drive the filtered ingestion
via the Nix function via the generator machinery, and then we register
the root node to the path info service inside the store.

`builtins.filterSource` is very simple, `builtins.path` is a more
complicated model requiring the same logic albeit more sophisticated
with name customization, file ingestion method and expected SHA-256.

Change-Id: I1083f37808b35f7b37818c8ffb9543d9682b2de2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10654
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 14:16:36 +00:00
Florian Klink
d9565a4d0a refactor(tvix): don't use Rc::clone explicitly
All these functions have an Rc of Rc<TvixStoreIO> or Rc<dyn StoreIO>, so
we can call io.clone() directly.

Change-Id: I96a5bcb9af4aca93892cb72bcfaf14540da52381
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10989
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-20 12:24:13 +00:00
Vincent Ambo
94f582341e refactor(tvix/eval): use internal SourceCode field in error printers
Makes use of the SourceCode field now being stored directly in
errors (see parent CL). With this change, the default `Display`
implementation can now format errors correctly, and there is no need
to keep a `SourceCode` around just for error formatting.

Updates dependent crates (CLI, serde, tvixbolt) to use this correctly.

Change-Id: Iddc5d7a6b4bab391f30a999e4c68aca34304c059
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10987
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 09:18:08 +00:00
Aspen Smith
0db46dacea feat(tvix/glue): Init fetcher builtins
Initialize a new empty builtins module `fetcher_builtins`, which will
contain the builtins which fetch URLs from the internet:

* fetchurl
* fetchGit
* fetchTarball
* fetchTree (maybe? this is experimental)

These builtins are all implemented in CPP nix at:
https://github.com/NixOS/nix/blob/2.20.2/src/libexpr/primops/fetchTree.cc

These builtins are added to the evaluation context using a similar
mechanism to the derivation builtins, and have been added everywhere
derivation builtins were previously being added.

Change-Id: I133b91cc9560f23028621414537f712e7bd8a825
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10974
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-19 16:51:10 +00:00
Florian Klink
33e68496f2 fix(tvix/cli): set max level to info on release mode
Log levels can be statically disabled at compile time via Cargo
features.

Do this for everything more verbose than info in release mode.

Change-Id: I57d4282b4e19a8d44f68da2f15c7534dd3e08977
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10955
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-18 07:12:27 +00:00
Florian Klink
d293304667 feat(tvix/cli): set up tracing[-subscriber]
No otlp yet, this simply gives us structured log output and a cli
argument for the log level.

Change-Id: Ifaa60bae419640e92baebb6ee59eedd775c769c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10853
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-02-17 08:06:42 +00:00
Florian Klink
e7815945df feat(tvix/cli): support configuring BuildService
Allow using a BUILD_SERVICE_ADDR env var, or cmdline args to configure
it. Still default to the dummy implementation.

Change-Id: I68f34f7b09eabef2b0491103857bbc798398ebfc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10846
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-02-17 04:49:03 +00:00
Aspen Smith
201173afac fix(tvix): Represent strings as byte arrays
C++ nix uses C-style zero-terminated char pointers to represent strings
internally - however, up to this point, tvix has used Rust `String` and
`str` for string values. Since those are required to be valid utf-8, we
haven't been able to properly represent all the string values that Nix
supports.

To fix that, this change converts the internal representation of the
NixString struct from `Box<str>` to `BString`, from the `bstr` crate -
this is a wrapper around a `Vec<u8>` with extra functions for treating
that byte vector as a "morally string-like" value, which is basically
exactly what we need.

Since this changes a pretty fundamental assumption about a pretty core
type, there are a *lot* of changes in a lot of places to make this work,
but I've tried to keep the general philosophy and intent of most of the
code in most places intact. Most notably, there's nothing that's been
done to make the derivation stuff in //tvix/glue work with non-utf8
strings everywhere, instead opting to just convert to String/str when
passing things into that - there *might* be something to be done there,
but I don't know what the rules should be and I don't want to figure
them out in this change.

To deal with OS-native paths in a way that also works in WASM for
tvixbolt, this also adds a dependency on the "os_str_bytes" crate.

Fixes: b/189
Fixes: b/337
Change-Id: I5e6eb29c62f47dd91af954f5e12bfc3d186f5526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10200
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-01-31 14:51:49 +00:00
Aspen Smith
6f9e25943f feat(tvix/eval/observer): Allow capturing timing of events
Add a new --trace-runtime-timing flag (probably a better bikeshed for
this) that enables capturing the time, relative to the last event, of
each event recorded with the tracing observer.

This probably isn't *super* useful yet, but I'd like to start here in
adding new profiling tools to the VM, specifically based on the runtime
observer

Change-Id: Id7f12077291c39bf3eef42ab6744bfba53687a65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10713
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-01-31 04:41:37 +00:00
Florian Klink
501827db59 refactor(tvix/glue): add BuildService to TvixStoreIO
TvixStoreIO triggers builds whenever IO into a not-yet-built store path
is requested, if it knows how to build that path.

Change-Id: If30e9db6be2f2a30cbc9d0576f357f3ecfa0d35a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10645
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-18 14:30:23 +00:00
Ryan Lahfa
12ae96cff2 feat(tvix/glue): use TvixStoreIO as derivation builtin state
We propagate a `TvixStoreIO` as the `state` of our derivation-specific
builtins in the glue crate.

The evaluators `io_handle` itself is using a Rc<dyn EvalIO>.

An earlier version of TvixStoreIO was also introducing generics over the
different internal services themselves, but we opted for instead
hardcoding this to Arc<dyn …> for the sake of less macro voodoo.

Change-Id: I535c476f06b840858fa3070c4a237ece47f7a15b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10636
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-18 14:28:00 +00:00
sterni
99615002a7 fix(tvix/cli/ci): prevent unnecessary dep on compared out/drv path
Change-Id: Ieab16fd5fad67a4ed0124f118aef8abbefd51cdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10646
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-16 16:50:59 +00:00
Florian Klink
5f0360c566 refactor(tvix/glue): simplify TvixStoreIO further
We don't need to spawn in all these places, we can just block_on
directly, this is all IO bound.

This also means, we don't need to clone any of the service handles
(except preserving clone-ability of the BlobService).

Change-Id: I7d90f4d6a263a98491caa071ada538a5197a5472
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10540
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-05 16:49:10 +00:00
Florian Klink
5a82736122 chore(tvix): bump test-case dep to 3.3.1
Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-01-05 16:43:34 +00:00
Ryan Lahfa
54ad5a19e7 feat(tvix/cli): test firefox (drv|out)Path
With context strings, we are now able to reproduce up to firefox.

Until a more problematic example can be found, we should at least
lean on this particular example.

Change-Id: Ibeaf799d26e1f160b6c93f8ccd978702a2a7e3e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10460
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:48:03 +00:00
Ryan Lahfa
de6c8b7546 feat(tvix/cli): context-aware raw printing
Raw printing will transform the result into a string
and print it.

In case of a contextful string, this will fail by default, as expected.

Change-Id: I5e564329e7b001adc57a77a9153b4425cb332bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10457
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:48:03 +00:00
Ryan Lahfa
099ca6b7c0 feat(tvix/glue): contextful derivation
We calculate the input context by performing the union of context
over all input of the derivation.

Then, we just pass the rest of it to the remaining machinery.

Finally, we re-emit an `outPath` and a `drvPath` containing the expected
contexts.

Change-Id: I74905fb258b5bee8b08d1208c9eb87f51b92a890
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10436
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:23:33 +00:00
Florian Klink
f6d1a56c8c refactor(tvix/store): move construct_services helper here
This takes three URLs, and constructs Arc'ed
{Blob,Directory,PathInfo}Service, allowing to remove some of the
boilerplate.

Change-Id: I40e7c2b551442ef2acdc543dfc87ab97e7c742bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10484
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-31 22:23:49 +00:00
Florian Klink
41935fab70 refactor(tvix/castore/directorysvc): return Box, not Arc
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.

Change-Id: Ia05bb370340792a048e2036be30e285ef1e63870
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10483
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:18:14 +00:00
Florian Klink
9ca1353122 refactor(tvix/castore/blobsvc): return Box, not Arc
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.

Change-Id: I7bd337cd2e4c2d4154b385461eefa62c9b78345d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10482
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:18:14 +00:00
Florian Klink
e2b6c77bfc feat(tvix/cli): allow configuring different services
At some point, tvix-cli needs to talk to the outside world to persist
things into a real store.

Introduce the same CLI options to configure
{Blob,Directory,PathInfo}Service URLs.

We need to be a bit careful with how we set up stores, and make
this separate from setting up TvixStoreIO, as it's holding a
Rc<RefCell<KnonPath>> which not Send.

At some point, we might make this a Arc<RwLock<_>> later anyways, and
then this can be simplified a bit, but for now, this is sufficient.

Change-Id: I87d84ca3a10ce947e194ff985073791469773f35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10474
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 21:55:18 +00:00
Florian Klink
4fba57c2c9 refactor(tvix/eval): remove code and location from struct
Instead, it's passed in the evaluate/compile_only functions, which feels
more naturally. It lets us set up the Evaluation struct long before
we actually feed it with data to evaluate.

Now that Evaluation::new() would be accepting an empty list of
arguments, we can simply implement Default, making things a bit more
idiomatic.

Change-Id: I4369658634909a0c504fdffa18242a130daa0239
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10475
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-12-31 13:15:24 +00:00
Adam Joseph
29878259b3 feat(tvix/cli): add benchmark for bf286a54bc
cl/7558 used this expression as a benchmark to justify the introduction
of LightSpan::Delayed:

  builtins.length (builtins.attrNames (import ${pkgs.path} {}))

Let's add it as a benchmark case so it can be referenced easily.

Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
Change-Id: Idb6c69ddd284605dd3b5fd9ac5c79a69b9a470b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10253
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:28:53 +00:00
Adam Joseph
0c15a09b15 feat(tvix/cli): add macrobenchmark
This commit adds a simple MVP benchmark, built on our nix
infrastructure instead of cargo.  It simply runs `tvix-eval` inside
of GNU time, and prints the three essential statistics in a short
JSON blob.

You can run the benchmark with a simple `nix run`, like:

  nix run -f . tvix.cli.benchmark-hello
  nix run -f . tvix.cli.benchmark-firefox
  nix run -f . tvix.cli.benchmark-cross-firefox

Currently these blobs are stored only in the CI logs, which I'm sure
get garbage-collected at some point.  We should be putting them in
the git trailers, but that can wait for a future CL.

I tried using `cargo bench` for this but found it incredibly
frustrating.  Maybe I'm doing it wrong.  It seems to be designed for
microbenchmarks only, and very hard to control.  It kept building
all sorts of unnecessary stuff (like the tests), and unlike
crate2nix it was doing all the builds on only a single machine
instead of using more than one machine.  Worse, for that single
machine it kept picking my laptop instead of my fast servers!  It
seems excessively cargo-flavored for such a straightforward task.

Benchmark: {"hello.outPath":{"kbytes":"244736","system":"0.36","user":"2.76"}}
Benchmark: {"firefox.outPath":{"kbytes":"1506736","system":"2.38","user":"32.01"}}
Benchmark: {"pkgsCross.aarch64-multiplatform.firefox.outPath":{"kbytes":"11334548","system":"10.70","user":"107.07"}}

Change-Id: I85bc046ec551360284d7ecfc81a03914f0085909
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10216
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 11:56:58 +00:00
Florian Klink
c8cc31e079 refactor(tvix/glue): move builtins into separate directory
Change-Id: I25b7197458dbfbde8623545dc0a0286eb2744f10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9911
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 15:20:46 +00:00
Florian Klink
a72a1044c2 refactor(tvix/glue): move nix_path handling to helper in tvix-glue
Change-Id: I2327560c4cf0d3f90e253e3c2f47cb29c762461e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9910
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-04 15:18:45 +00:00
Florian Klink
3196fe0143 refactor(tvix): move tvix glue code into glue crate
There's various bits and pieces in tvix-cli that use both the store and
evaluator, as well as nix-compat. For example, builtins.derivation, as
well as the reference scanning implementation.

This "glue code" currently isn't accessible from anywhere else, but it'd
be very useful if it were.

Move it out into a `glue` crate, and make `tvix-cli` a consumer of it.

All the KnownPaths setup and passing around, as well as NIX_PATH
handling is also something that should probably be moved into the glue
crate as well, but that's something left for a future CL.

Change-Id: I080ed3d1825ab23790666486840f301f00856277
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 15:18:44 +00:00
edef
7f7c1ae7be refactor(nix-compat/store_path): make digest and name private
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
Florian Klink
6a04ba6bc5 fix(tvix/cli/derivation): fix populate_output_configuration
The `if let` wasn't matching `outputHashAlgo` being unset, and didn't
populate it in that case.

Port the remaining commented-out testcases over to nix-lang based tests.

Change-Id: I140b5643b9ed9d29f9522ec65d98d0b12262d728
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9825
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-23 14:57:32 +00:00
Florian Klink
2d99bfc7b7 refactor(tvix/cli): rename a to algo_str
Change-Id: I65d307126647841b50db008c21c155441a6b6cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9818
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-23 14:57:32 +00:00
Florian Klink
6800f1cdb9 test(tvix/cli/derivation): add test cases for output path calc
Get some test coverage with some calls to builtins.derivation.

The expected output paths were calculated with Nix.

Change-Id: I2ce205ea5244e8ef939d9cacb033283fc6f15d17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9817
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
077bf06c1d refactor(tvix/cli): move evaluator instantiation to helper
Have a private `eval` function in the test module that returns an
EvaluationResult, and migrate the existing tests over to use it, rather
than repeating itself.

Change-Id: I879987700c8507248c644ef03b62a8cb8e308139
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9816
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
34fc4637eb refactor(tvix/nix-compat): rename NixHashWithMode -> CAHash
This specific struct is only used to represent content-addressed paths
(in case a Derivation has a fixed-output hash, for example).
Rename `Output`'s `hash_with_mode` to `ca_hash`.

We now also include `CAHash::Text`, and update the `validate` function
of the `Output` struct to reject text hashes there.

This allows cleaning up the various output path calculation functions
inside nix-compat/src/store_path/utils.rs, as they can now match on
the type.

`make_type` is renamed to `make_references_string`,
`build_regular_ca_path` is renamed to `build_ca_path`, and
`build_text_path` has a disclaimer added, because you might not actually
want to use it.

Change-Id: I674d065f2ed5c804012ddfed56e161ac49d23931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9814
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
833957b374 feat(tvix/store/protos): add CA field
This adds support to represent the `CA` field found in some .narinfo
files. As `deriver`, it's also a subfield of the `narinfo` field.

Extending nix-compat with a more accessible data structure that can
take care of formatting, as well as writing validation functions in Rust
+ Golang, and integrating it into nar-bridge is something for a followup
CL.

Change-Id: I71e9c30957bcd03051a491aa54d7baac25b6dd2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9795
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-22 17:04:37 +00:00