Commit graph

2585 commits

Author SHA1 Message Date
Florian Klink
898830b332 fix(tvix/boot/tests): disable otlp
We don't have a OTLP collector inside the Nix build, this removes some
noise from the log about not being able to reach it.

Change-Id: I058839c8f214821b536306c8c30e1a8e7192153b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11189
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
Florian Klink
997b59e452 feat(tvix/boot): support seeding closures via nar-bridge
This updates the parameters mkBootTest can be called with. It now
accepts a `path`, and then either importPathName needs to be set, or
isClosure needs to be set to true.

The former activates the existing functionality, tvix-store import is
used to import contents  as a NAR-addressed store path.
The latter uploads the path as a closure (so including its references,
and keeping the store paths intact) to tvix-store.
We use nar-bridge, and the HTTP interface it provides to do this.
As `nix copy` can't be used inside a Nix build, we use
`pkgs.mkBinaryCache` to come up with the .narinfo and .nar files that
would be in a binary cache, and then use a bit of GNU Parallel and bash
to upload store paths ourselves.

Change-Id: Icfa5c0af0c22ab5418686947aa2c060f5987b873
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11188
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-18 16:10:05 +00:00
Florian Klink
8fb1d0ad4b docs(tvix/boot): document we could use a smaller kernel here.
We currently use a pretty big kernel to boot the tvix VMs, with a lot
of drivers.

It can probably be slimmed down significantly, at least on the hardware
driver and filesystem front.

Change-Id: I4c7f73fb2dafbf2bcdec8057d2b14a9b0e9b3275
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11187
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-03-18 16:10:05 +00:00
Florian Klink
fca3926a62 feat(tvix/boot/tests): make urls and paths configurable
Fork off a long-living tvix-store daemon in the background, and have
both the import, as well as the virtiofs invocation connect to that one,
rather than use the underlying storage directly.

This also opens up the potential to have other things seed the store
(like nar-bridge).

Change-Id: Ie13aedaf7ab31e5ebe78a7be42f144af3920fa9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11186
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
Florian Klink
05bdb68523 feat(tvix/blobservice/object_store) more logging
Have derive_{blob,chunk}_path emit trace-level events for both the
values they're called with, as well as the return value.

With RUST_LOG in place, it doesn't get lost in other unrelated noise.

Change-Id: Id2451e3657324eff482841eb26a22d19e22bde30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11136
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-18 16:10:05 +00:00
Florian Klink
c0e16059e6 feat(tvix/store): support RUST_LOG env var
This allows selectively increasing the log level for only parts of the
stack.

For example, the following RUST_LOG env var enables "tracing" level
logging for `tvix_store` and `tvix_castore`, while keeping it at "info"
for the rest of the stack:

export RUST_LOG='info,tvix_store=trace,tvix_castore=trace'

It only affects logs, not traces (if enabled).

Change-Id: Ib936bd132a405f216e75c843db83fbd71d20a18a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11182
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
Florian Klink
82f8ce8b7d feat(tvix/castore/blobsvc/grpc): read data in chunks
Whenever this encounters an open_read(), it'll first check for more
granular chunking. If there's more granular chunking data available, a
ChunkedReader is constructed (which supports seeking backwards).

This currently is still a bit stupid, and doesn't compose, as
`ChunkedReader` uses `self` as the `BlobService` to ask for the
individual chunks.

In store composition future, we might want to compose this differently,
essentially constructing `ChunkedReader` with another `BlobService`
representing the entire hierarchy, so there's a chance to locally cache
things, and do less requests.

Change-Id: I22e0df4d6245f666d083b4f0b7114d3ac41d1dce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11185
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
Florian Klink
50c81d7838 feat(tvix/nar-bridge): support listening on unix sockets
This simply checks for the address to contain slashes, and if so, opens
a unix socket, rather than a tcp one. We'll use this in //tvix/boot
tests to simplify waiting for nar-bridge to be up.

Change-Id: I7184f548d57142b1c5f698a1f0c30343489373a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11184
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 14:57:30 +00:00
Florian Klink
70bbf23767 refactor(tvix/castore/src/blobservice): remove useless else case
Change-Id: I09000371a1d8ff212ab46050d1a480509c6ffe70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11183
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-18 14:25:37 +00:00
Florian Klink
3b1c9172f6 feat(tvix/castore): impl Debug for B3Digest
Use the same format as Display, b3: followed by the base64
representation. This makes the debug implementation of everything
containing a b3 digest much nicer to read.

Change-Id: I3ca3154d0b6fb07781c8f9c83ece3ff1a6957902
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11181
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-17 22:18:20 +00:00
sterni
e220d80727 chore(3p/sources): Bump channels & overlays
- agenix has not been updated (https://github.com/ryantm/agenix/pull/241).

- //tvix: regenerate protobuf files

- //tvix:clippy: work around https://github.com/rust-lang/rust-clippy/issues/12281
  which exclusively causes false positives in our code at the moment.

Change-Id: I38d2f4c0e6d1abc92be360b06f58e6d40e7732a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11127
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-17 10:56:07 +00:00
Florian Klink
dbf87f3057 chore(tvix): bump tonic to 0.11.0
This bumps tonic and surrounding crates to 0.11.x.

We added support for tonic 0.11.x into tokio-listener
(https://github.com/vi/tokio-listener/pull/4), so that's bumped as well.

Change-Id: Icfade5894403228299836fefb21b2f9ae59dbebb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11156
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-16 17:04:12 +00:00
sterni
212f5afdcf fix(tvix/eval): builtins.unsafeGetAttrPos returns path as a string
Change-Id: Ia1058a5ea687f413fbb62365da199032bdbd3fd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11155
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2024-03-16 12:52:30 +00:00
Florian Klink
fdf9657654 fix(tvix): don't emit rerun-if-changed
`build.rs` emits rerun-if-changed statements for all proto files, as
well as all include paths we pass it.

Unfortunately, due to protobufs include path rules, we need to specify
the path to the depot root itself as an include path, at least when
building impurely with `cargo`. This causes cargo to essentially always
rebuild, as it also puts its own temporary files in there.

Unfortunately, tonic-build does not chase down to individual .proto
files that are included.

Disable emitting these `rerun-if-changed` statements for now.

This could cause cargo to not rebuild protos every time, causing stale
data until the next local `cargo clean`, but considering the protos
change not that frequently, and it'll immediately surface if trying to
build via Nix (either locally or in CI), it's a good-enough compromise.

Change-Id: Ifd279a2216222ef3fc0e70c5a2fe6f87997f562e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11157
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-03-16 09:34:10 +00:00
Florian Klink
56dc4f735b docs(tvix): fix some docstrings
Change-Id: Ife599387d0472cd746b992bd6755a2fb6a0e0dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11158
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-15 23:16:01 +00:00
Vincent Ambo
d3534ee051 fix(depotfmt): only exclude *imported* tvix tests
When we added the Nix language test suite in cl/6126, we excluded the
whole tvix tests folder from Nix formatting. This is unintentional, as
we probably want *our* tests to be formatted correctly.

Change-Id: I2b66d79e30fae17e75d5a1f8c44e279886091c5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11154
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-15 14:22:49 +00:00
Florian Klink
5fccbe5939 feat(nix-compat/wire): add read_bytes[_unchecked]
This introduces a version reading sized byte packets. Both read_bytes,
accepting a range of allowed sizes, as well as read_bytes_unchecked,
which doesn't care, are added, including tests.

Co-Authored-By: picnoir <picnoir@alternativebit.fr>
Change-Id: I9fc1c61eb561105e649eecca832af28badfdaaa8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11150
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
2024-03-15 10:35:24 +00:00
Florian Klink
c364c0b4de docs(nix-compat/wire): update docstrings
These are not streams, but AsyncRead and AsyncWrite.

Change-Id: I7d988fa0490800b72862f4f0fcac3dceac70ec26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11149
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-03-15 10:23:12 +00:00
Florian Klink
907ecff999 feat(nix-compat/wire): add low-level wire format primitives code
This brings some initial Nix wire format parsing code, used in the nix
daemon protocol, remote store/builder protocol, as well as the NAR
format itself (note we already have more specialized code for the last
one).

Thanks to embr, this code already exists, in
https://codeberg.org/gorgon/gorgon/src/branch/main/nix-daemon/src/wire.rs,
and we can vendor it into here, as EUPL is compatible with GPL (in that
direction).

The code uses the tokio::io Reader and Writer traits, not the ones from
the `futures` crate, as they provide some more convenient `read_u64_le`
functions.

More application-specific parsing code, as well as code to read strings,
or bytes are left out for now, as we want to be be more restrictive
w.r.t allowed max sizes, and need to parse bytes, not strings.

The code slightly diverges, as we have clippy looped into CI.
`Ok(…?)` can be turned into just the inner expression, and
some .and_then can be expressed in a simpler fashion.

Change-Id: Ie3adcb485e9d66786673b1962a08d4e5df3781d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11148
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-03-15 10:23:12 +00:00
Florian Klink
905a79308e refactor(nix-compat/derivation): emphasize aterm_bytes
derivation_or_fod_hash constructs ATerm bytes and feeds them to sha256.

input_derivations being slightly modified is an implementation detail,
so move the BTreeMap construction inline, and have aterm_bytes in a
let binding (and feed it to the hash function directly while
constructing it).

This makes it a bit more understandable what's going on.

Change-Id: I2f5cfbd1c964fd39ac731ca39e76cfc168f4c7d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11147
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
2024-03-14 23:10:54 +00:00
Florian Klink
142c72e070 refactor(nix-compat/store_path): add from_name_and_digest_fixed
Allow constructing a StorePath with a fixed-size digest.

Change-Id: Id7d0b0152f6c55660a8973a02c84afa9188ce3ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11144
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Tested-by: BuildkiteCI
2024-03-14 23:10:54 +00:00
Florian Klink
98e6936301 refactor(nix-compat/store_path/utils): move helper function in
This is only used inside this function, in 2 of the match cases.

Change-Id: Ib361f5ee0e3b203802f7d05b9a7f332d14bbcf80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11143
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Tested-by: BuildkiteCI
2024-03-14 17:38:47 +00:00
Florian Klink
43c851bc84 refactor(nix-compat/store_path): take [u8;32] for outer fingerprint
The outer fingerprint used for store path calculation is always a sha256
digest. This includes both input and output-addressed store paths.

We used a NixHash here, which can also represent other hash types, and
that had a bunch of annoyances:

 - Whenever we had the bytes, we had to wrap them in a NixHash::Sha256().
 - Things like AtermWriteable had to be implemented on NixHash,
   even though we then had an assertion it was only called in the
   NixHash::Sha256 case.

Change-Id: Ic895503d9b071800d2e52ae057666f44bd0ab9d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11142
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-03-14 16:52:42 +00:00
sterni
dc9c2c9b0c docs(tvix/eval): C++ Nix now has deduplicated inherit (from)
Change-Id: Icf74c699130c2ab774973a7e98bc44ffba8c5ec2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11139
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
2024-03-12 21:23:32 +00:00
Florian Klink
514edc2ea5 feat(tvix/boot/tests): add simple objectstore test
This makes BLOB_SERVICE_ADDR configurable, and creates a flavor setting
it to objectstore+file://$PWD/blobs.

Change-Id: I68c21367f83f68b4dee701c5678f438c7d8fbe43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11137
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-11 22:42:26 +00:00
Florian Klink
f22d5b3d11 feat(tvix/castore/blobsvc/from_addr): support object_store
The object_store crate supports a ton of different stores, with different schemes.

For now, use a objectstore+ scheme prefix to enable these.

Change-Id: I946f76e32a0fb0867ef59060217894cda5b959b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11080
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-03-11 22:42:01 +00:00
Florian Klink
1c2db676a0 feat(tvix/castore/blobsvc): add object storage implementation
This uses the `object_store` crate to expose a tvix-castore BlobService
backed by object storage.

It's using FastCDC to chunk blobs into smaller chunks when writing to
it.

These are exposed at the .chunks() method.

Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-03-11 22:42:01 +00:00
Florian Klink
d327bf775d feat(tvix/store/bin): allow disabling OTLP at runtime
This was only possible by disabling without the otlp feature flag so
far.

Introduce the same --otlp=false mechanism that nar-bridge also supports
to be able to turn it off at runtime.

Change-Id: Ib22a364c35056ca9d8e327c0e2a79970a4cf4b2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11135
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-11 17:02:26 +00:00
Aspen Smith
54609e8c17 feat(tvix/glue): Add AsyncRead wrapper to decompress streams
Add a new AsyncRead wrapper, DecompressedReader, that wraps an
underlying AsyncRead, but sniffs the magic bytes at the start of the
stream to determine which compression format is being used out of the
three that are supported by builtins.fetchTarball, and switches to the
correct decompression algorithm adapter dynamically.

This will be used in the implementation of builtins.fetchTarball

Change-Id: I892a4683d5c93e67d4c173f3d21199bdc6605922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11019
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-11 11:35:18 +00:00
Aspen Smith
de727bccf9 feat(tvix/glue): Implement builtins.fetchurl
Implement the fetchurl builtin, and lay the groundwork for implementing
the fetchTarball builtin (which works very similarly, and is implemented
using almost the same code in C++ nix).

An overview of how this works:

1. First, we check if the store path that *would* result from the
   download already exists in the store - if it does, we just return
   that
2. If we need to download the URL, TvixStoreIO has an `http_client:
   reqwest::Client` field now which we use to make the request
3. As we're downloading the blob, we hash the data incrementally into a
   SHA256 hasher
4. We compare the hash against the expected hash (if any) and bail out
   if it doesn't match
5. Finally, we put the blob in the store and return the store path

Since the logic is very similar, this commit also implements a *chunk*
of `fetchTarball` (though the actual implementation will likely include
a refactor to some of the code reuse here).

The main thing that's missing here is caching of downloaded blobs when
fetchurl is called without a hash - I've opened b/381 to track the TODO
there.

Adding the `SSL_CERT_FILE` here is necessary to teach reqwest how to
load it during tests - see 1c16dee20 (feat(tvix/store): use reqwests'
rustls-native-roots feature, 2024-03-03) for  more info.

Change-Id: I83c4abbc7c0c3bfe92461917e23d6d3430fbf137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-03-11 02:21:54 +00:00
Florian Klink
37703c9e44 docs(tvix/README): update IRC channels
`#tvl` is the general TVL community channel, `#tvix-dev` is dedicated to
Tvix development discussion.

Change-Id: If899d9ae70dffb4acf6a41ded54f80a1f0551c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11110
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-10 17:14:41 +00:00
Florian Klink
bd1def3ec4 fix(tvix/store/grpc/pathinfo): skip_all fields, handle errors
request only contains the outer metadata wrapping, and that's not too
interesting:

> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }

Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.

Also, ensure we handle all error cases properly, and log them. We
don't use `err` from instrument, as that'd also log an error on
`Status::not_found`.

Change-Id: Id1b983cb8b059c148c8a376f8802a1d28c59ba97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11103
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-09 05:48:46 +00:00
Florian Klink
4e78de7393 fix(tvix/castore/grpc/directory): skip_all fields in instrument
This only contains the outer metadata wrapping, and that's not too interesting:

> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }

Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.

Clean up the error logging a bit.

Change-Id: Ife1090ed411766a61e1fa60fd4c9570f38de1e98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11102
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-09 05:47:41 +00:00
Florian Klink
05deb37f44 fix(tvix/castore/grpc/blob): skip_all fields in instrument
This only contains the outer metadata wrapping, and that's not too interesting:

> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }

Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.

Log errors in some places where we didn't so far.

Change-Id: Ia68d6c526987d3716be62a0809195401cf28512b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11101
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-09 05:47:36 +00:00
Florian Klink
8f804f5d9c chore(tvix/nix-compat): bump zstd dependency
Otherwise this causes conflicts when adding a more recent zstd version
to another crate.

Change-Id: I02d1b1a37f18711c0969cb04e761e967fc4655dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11100
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-09 05:47:03 +00:00
Florian Klink
4954a39de4 fix(tvix/castore): also set SSL_CERT_FILE for tests there
For everything using reqwest here during test cases, we also need to
set SSL_CERT_FILE.

Change-Id: If8aeda65f3d75cb9ac5c9bc64e37a0cb7dffc17c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11092
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 17:43:42 +00:00
Florian Klink
0b396553d6 refactor(tvix/store/protos): more granular nix-compat conversions
implement From<&nix_compat::nixhash::CAHash> for nar_info::ca::Hash
individually, and make
From<&nix_compat::nixhash::CAHash> for nar_info::Ca a small wrapper that
uses it, as well as the .hash().digest_as_bytes() for the digest.

Change-Id: I7e9b6edd1e3f149eb270faf2928cd846d74e77ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11071
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 17:12:51 +00:00
Florian Klink
ef3f8936cb refactor(tvix/*/from_addr): improve test debuggability
If there's an unexpected test failure, print it out, rather than just
saying something is false even though it should be true.

Use .expect() for this, which displays the error if it failed.
We can't use expect_err(), as our stores are not display'able, so use an
assertion with a message there.

Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 16:54:19 +00:00
Florian Klink
b0e5a932ec docs(tvix): remove differences-drv-paths.md
This was true while we didn't do string contexts, but drvPath should not
differ anymore.

Change-Id: I58e6ed439519bd4d68259d1577946f3c65110dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11074
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 16:53:41 +00:00
Florian Klink
c9514b7c0c feat(tvix/docs): ship custom highlight.js with Nix support
Selected languages:
 - bash
 - c
 - cpp
 - diff
 - go
 - ini
 - json
 - markdown
 - nix
 - protobuf
 - rust
 - shell
 - xml
 - yaml

Created with https://highlightjs.org/download

Change-Id: I76a87d2c1e9fa582d1474f8ffb9f3129e9a858bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11073
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-03-03 16:53:16 +00:00
Florian Klink
65a810fc0b feat(tvix/docs): switch to mdbook
Some of the docs are still outdated (like architecture and drv path
inconsistencies).

Change-Id: I7a6afceb008ef4cd19a764dd6c637b39fa842a2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11072
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-03-03 16:52:38 +00:00
edef
6bdaebcb55 fix(tvix/tools/weave): handle sliced arrays correctly
The start/end offsets are not necessarily coterminous with the underlying
values array, so even if the stride is fixed, we still we need to slice
the chunks down to match the start/end offsets.

This bug shouldn't affect the correctness of any existing code, since
we're always working with unsliced arrays read directly from Parquet.

Change-Id: I2f7ddc4e66d4d3b2317a44bd436a35bff36bac79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11081
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 16:45:19 +00:00
Florian Klink
1c16dee207 feat(tvix/store): use reqwests' rustls-native-roots feature
This makes reqwest honor `SSL_CERT_FILE` - previously it was using the
chain bundled in webpki-roots.

`object_store` pulls in `reqwest` with this feature, and the cargo
solver will enable that feature globally as soon as we pull it in, as it
assumes features are additive.

This requires setting `SSL_CERT_FILE` when running tests, otherwise
they'll fail with the unhelpful "NotFound" error.

This was quite some fun to debug, why adding `object_store` to
tvix-castore suddenly made tvix-store tests fail!

Change-Id: I64fc82b4d994715480efdb1ffecb279716456ab9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11090
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:32:38 +00:00
Florian Klink
4b4443240e feat(tvix/castore): add HashingReader, B3HashingReader
HashingReader wraps an existing AsyncRead, and allows querying for the
digest of all data read "through" it.
The hash function is configurable by type parameter, and we define
B3HashingReader.

Change-Id: Ic08142077566fc08836662218f5ec8c3aff80be5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11087
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:31:31 +00:00
Florian Klink
8383e9e02e feat(tvix/castore/digests): impl From digest::Output<_> for B3Digest
This allows calling .into() to get a B3Digest.

Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:18:19 +00:00
Florian Klink
442eabe9d0 feat(tvix/nix-compat): add HashMode and CAHash::mode()
This allows peeking at the mode without matching on all enum kinds of
CAHash directly.

Change-Id: Icca147a876f7f4cff0bbaa067ac4dae40d05bee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11069
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 13:17:50 +00:00
Brian Olsen
eff2cc4f68 fix(tvix/nix-compat): Make CAHash deserialize more formats
Currently CAHash only deserializes the hash in hex code while
the serializer outputs a nixbase32 hash. This means that you can't currently
deserialize what has been serialized.

This change makes deserialize support any digest format (so hex, nixbase32
and base64) as well as flattens the deserialize code and error handling.

It also implements serde methods of HashAlgo directly using Display and TryFrom
implementations because otherwise these would get serialized as eg. Sha256 instead
of sha256 which also broke CAHash serialize/deserialize.

Change-Id: I1941a72eaec741e4956292adaaf0115b97f260ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11082
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 13:07:18 +00:00
Florian Klink
260c2938d4 refactor(tvix/nix-compat): rename NixHash::{digest,hash}
This doesn't return the digest, but the internal NixHash.

To get the digest, you use `ca_hash.hash().digest_as_bytes()`
(as done in tvix/nix-compat/src/derivation/write.rs).

Change-Id: Ib07918dec63ae42ddf8c8d3f1d10510f9c4df255
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11070
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 11:56:18 +00:00
Florian Klink
7bebf492ec refactor(tvix/castore/blobsvc/chunked_reader): refactor, document
The public-consumable thing here is ChunkedReader, not ChunkedBlob.

ChunkedBlob is a helper that can be used to get a new AsyncRead, but
not AsyncSeek. It is used internally by ChunkedReader whenever the
client seeks.

Make this more obvious, by extending the documentation, and putting
ChunkedReader at the top of this file.

Also make ChunkedBlob and its methods private, and give ChunkedReader a
more useful constructor (from_chunks, instead of from_chunked_blob).

Change-Id: I2399867591df923faa73927b924e7c116ad98dc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11079
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-03 11:22:56 +00:00
Florian Klink
53fb9ff4c6 feat(tvix/castore/blobsvc): BlobReader for more trivial types
Change-Id: I80e4f26c41a504fa4c6a013c2a1e76de613ba294
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11078
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-02 17:05:23 +00:00
Florian Klink
982459d343 fix(tvix/castore/blobwriter): don't require Sync + 'static
There's no reason for these two.

Change-Id: Ie6f238bbb0b17971c9877b11b61ea7ebca573c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11075
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-02 05:56:21 +00:00
Padraic-O-Mhuiris
e0ffb4d47a feat(tvix/eval): implement builtins.hashFile
The primary addition is the `builtin_hashFile` function which attempts
value-path coercion and on success reads the file to a nix_string.
Similar to the `builtins_hashString` implementation, the hash is then
derived using the `hash_nix_string` function in the introduced hash
module which has extracted hashing functionality initially specified in
cl/11005.

Change-Id: I5e3ef0317223af99488ebb79efc2fb49b4cbc1b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11007
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-29 10:58:21 +00:00
edef
e3860689ba feat(tvix/tools/weave): init
Scalable tracing GC for the cache.nixos.org dataset.

Change-Id: I6c7852796f28e1a1c7607384ffb55f44407e1185
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10765
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-27 11:50:21 +00:00
Florian Klink
d5ec9a0c6f chore: update links to tvixbolt
Use bolt.tvix.dev everywhere.

Change-Id: Ifebbea60058418c12cde20a2a3879a8a7f5f830c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11022
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-24 10:17:56 +00:00
Padraic-O-Mhuiris
5c3065b43a feat(tvix/eval): implement builtins.hashString
Implements md5, sha1, sha256 and sha512 using the related crates from
the RustCrypto hashes project (https://github.com/RustCrypto/hashes)

Change-Id: I00730dea44ec9ef85309edc27addab0ae88814b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11005
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-02-23 16:04:35 +00:00
Aspen Smith
ffb134398d refactor(tvix/glue): Make a single errors.rs module
To pave the way for adding a new error type for builtins in this crate,
move DerivationError to a new builtins::errors module.

Change-Id: I65fcad63e43ed40ad39c2c6540a2ab80fdd90fd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11016
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-02-23 15:47:46 +00:00
Florian Klink
91d5745c3d refactor(nix-compat/store_path): simplify build_ca_path
Move the the `fixed:out:[r:]{}:` generation to a helper function,
use matches! for more clarity.

Change-Id: I4e930c42aacbf5c7451d1f8c8c80ccb4c45389f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11006
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2024-02-22 17:57:55 +00:00
Aspen Smith
5e31096154 feat(tvix/eval): Store string context alongside data
Previously, Nix strings were represented as a Box (within Value)
pointing to a tuple of an optional context, and another Box pointing to
the actual string allocation itself. This is pretty inefficient, both in
terms of memory usage (we use 48 whole bytes for a None context!) and in
terms of the extra indirection required to get at the actual data. It
was necessary, however, because with native Rust DSTs if we had
something like `struct NixString(Option<NixContext>, BStr)` we could
only pass around *fat* pointers to that value (with the length in the
pointer) and that'd make Value need to be bigger (which is a waste of
both memory and cache space, since that memory would be unused for all
other Values).

Instead, this commit implements *manual* allocation of a packed string
representation, with the length *in the allocation* as a field past the
context. This requires a big old pile of unsafe Rust, but the payoff is
clear:

    hello outpath  time:   [882.18 ms 897.16 ms 911.23 ms]
                   change: [-15.143% -13.819% -12.500%] (p = 0.00 < 0.05)
                   Performance has improved.

Fortunately this change can be localized entirely within
value/string.rs, since we were abstracting things out nicely.

Change-Id: Ibf56dd16c9c503884f64facbb7f0ac596463efb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10852
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: aspen <root@gws.fyi>
2024-02-21 20:53:04 +00:00
aspen
d74c68025b revert(tvix): don't use Rc::clone explicitly
This reverts commit d9565a4d0a.

Reason for revert: this was intentional - putting Rc::clone instead of
.clone is a common Rust idiom, and makes it explicit that we're cloning
a shared reference, not an underlying resource

Change-Id: I41a5f323ee35d7025dc7bb02f7d5d05d0051798d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10995
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 18:36:11 +00:00
Florian Klink
771200df7c fix(tvix/eval): allow reading non-UTF8 files
With our values using bstr now, we're not restricted to only reading
files that contain valid UTF-8.

Update our `read_to_string` function to `read_to_end`
(named like `std::io::Read::read_to_end`), and have it return a Vec<u8>.

Change-Id: I87f0291dc855a132689576559c891d66c30ddf2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11003
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Pádraic Ó Mhuiris <patrick.morris.310@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 13:55:41 +00:00
sterni
3e93efdc8c docs(tvix/eval): notes on the implementation of catchable errors
Change-Id: I3b2f039d2c2d40ee5d13d071915ced6f3deb18f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11000
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-21 13:32:38 +00:00
Peter Kolloch
fde488ec6d feat(tvix/nix-compat): Use StorePath in Output
https: //b.tvl.fyi/issues/264
Change-Id: Icb09be9643245cc68d09f01d7723af2d44d6bd1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11001
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-21 11:38:03 +00:00
Peter Kolloch
035f617b7f feat(tvix/nix-compat): input_sources as StorePath
https: //b.tvl.fyi/issues/264
Change-Id: I7a235734dc1f8e93e387a04ba369f3b702c6d5b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10992
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-21 11:34:24 +00:00
Peter Kolloch
c06fb01b3b feat(tvix/nix-compat): input_derivations with StorePaths
...in `Derivation`.

This is more type-safe and should consume less memory.

This also removes some allocations in the potentially hot path of output hash calculation.

https: //b.tvl.fyi/issues/264
Change-Id: I6ad7d3cb868dc9f750894d449a6065608ef06e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-21 11:34:24 +00:00
Peter Kolloch
a44a8985cc feat(tvix/nix-compat): generalize aterm writing for derivation
...so that we can also use `StorePath`s in
derivation.input_derivations.

Towards https://b.tvl.fyi/issues/264

Change-Id: I71d296ca273979c70f277a7f4f88a5f76de3d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10973
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 11:34:24 +00:00
Aspen Smith
5ced8e7292 fix(tvix/eval): Propagate catchables in builtins.groupBy
One last place where we needed to wrap the forcing of the element of a
list in `try_value!`. This fixes a previously `notyetpassing` test

Change-Id: I8827a3e39630e6959013b70bdfa9cbcb93f4e91c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10789
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-02-20 23:53:58 +00:00
Ryan Lahfa
7388078630 feat(tvix/eval): implement builtins.filterSource
We add a new set of builtins called `import_builtins`, which
will contain import-related builtins, such as `builtins.path` and
`builtins.filterSource`. Both can import paths into the store, with
various knobs to alter the result, e.g. filtering, renaming, expected
hashes.

We introduce `filtered_ingest` which will drive the filtered ingestion
via the Nix function via the generator machinery, and then we register
the root node to the path info service inside the store.

`builtins.filterSource` is very simple, `builtins.path` is a more
complicated model requiring the same logic albeit more sophisticated
with name customization, file ingestion method and expected SHA-256.

Change-Id: I1083f37808b35f7b37818c8ffb9543d9682b2de2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10654
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 14:16:36 +00:00
Vincent Ambo
20833656ae fix(tvix/eval): propagate catchable errors at the top of an eval
(Re-)Adds an error variant that wraps a catchable error kind, which is
used for returning the result of an evaluation.

Previously this would return the internal catchable value, which would
lead to panics if users tried to use these. Somehow this was missed; I
think we need error output tests.

Change-Id: Id6e24aa2ce4ea4358a29b2e1cf4a6749986baf8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10991
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 12:33:57 +00:00
Florian Klink
d9565a4d0a refactor(tvix): don't use Rc::clone explicitly
All these functions have an Rc of Rc<TvixStoreIO> or Rc<dyn StoreIO>, so
we can call io.clone() directly.

Change-Id: I96a5bcb9af4aca93892cb72bcfaf14540da52381
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10989
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-20 12:24:13 +00:00
Vincent Ambo
d0d1027a85 feat(tvix/serde): add an example using nixpkgs/lib
This displays how users can configure an impure evaluation for
tvix-serde, which makes it possible to use e.g. `nixpkgs/lib`.

We might want to add an example showing how the full Nix-glue
compatibility stuff can be added here, too.

Change-Id: I2224a3fc66e739969d4c723c3d9d8127a046b6fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10994
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2024-02-20 12:13:45 +00:00
Vincent Ambo
e839116e5c fix(tvix/eval): make path resolution work by default in impure mode
The previous behaviour (enabling `import`, but not allowing e.g.
`<nixpkgs/lib>` to resolve) was very confusing.

Now imports from NIX_PATH become enabled by default, unless the user
already overrode that behaviour with something else by setting
`Evaluation::nix_path` manually.

Change-Id: Iad970beb633d9887be4b185b01e6f5858d81bea3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10993
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 12:13:45 +00:00
Vincent Ambo
f40ff1737e fix(tvix/eval): fix accidental recursion in printing errors
There's some code path where fancy_format_str()->Error::Display, which
recurses forever and overflows the stack. This was introduced in a
previous commit today.

Change-Id: I87a59492099f6c138c752478901b9aa614bb57cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10990
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-20 11:54:03 +00:00
Vincent Ambo
94f582341e refactor(tvix/eval): use internal SourceCode field in error printers
Makes use of the SourceCode field now being stored directly in
errors (see parent CL). With this change, the default `Display`
implementation can now format errors correctly, and there is no need
to keep a `SourceCode` around just for error formatting.

Updates dependent crates (CLI, serde, tvixbolt) to use this correctly.

Change-Id: Iddc5d7a6b4bab391f30a999e4c68aca34304c059
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10987
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 09:18:08 +00:00
Vincent Ambo
3c87687798 refactor(tvix/eval): add SourceCode directly into error types
With this change it's no longer necessary to track the SourceCode
struct separately from the evaluation for error reporting: It's just
stored directly in the errors.

This also ends up resolving an issue in compiler::bindings, where we
cloned the Arc containing file references way too often. In fact those
clones probably compensate for all additional SourceCode clones during
error construction now.

Change-Id: Ice93bf161e61f8ea3d48103435e20c53e6aa8c3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10986
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-20 09:18:08 +00:00
Florian Klink
b38badf206 docs(tvix/castore/directorysvc): K/V is not necessarily flat
Some implementations of DirectoryService might not allow retrieval of
intermediate Directory nodes, that are not at the "root".

Think about an object store implementation. The client is doing a
get_recursive anyways to reduce the number of roundtrips.

By documenting the fact we don't need to support looking up intermediate
Directory messages, we can just batch all directories into the same
object, keyed by the root.

Change-Id: I019d720186d03c4125cec9191e93d20586a20963
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10988
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 09:17:38 +00:00
Vincent Ambo
2277e62c2d refactor(tvix/eval): add API for enabling impure evaluation features
This makes some of the work of configuring an arbitrary I/O handler
easier.

Change-Id: I158db3235fe83df6e709578ed515e0e028c20086
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10959
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 08:46:51 +00:00
Florian Klink
43570bae11 refactor(tvix/store/proto): use bstr::ByteSlice
String::from_utf8_lossy simply discards invalid bytes, while bstr
replaces them with their replacement character.

Change-Id: Ib78ff36ca5faacc1ad60bc4ddde7b62773848c07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10981
Tested-by: BuildkiteCI
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 05:29:14 +00:00
Vincent Ambo
98a17dbdf9 chore(tvix/glue): remove unnecessary assignment
Change-Id: I747307317c45085f9f7762d659870c5bd75b908b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10958
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 04:58:31 +00:00
Aspen Smith
0631b870b0 fix(tvix/store): Improve error message for InvalidNodeName
Rather than debug-formatting a Vec<u8>, which gives just a sequence of
numbers (not very helpful!) use String::from_utf8_lossy to give a much
more human-readable error message

Change-Id: I6cae795ebe2ab9a656c087b82a143411e6c7c81a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10980
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 03:48:40 +00:00
Peter Kolloch
46d89f899f feat(tvix/nix-compat): Extract to_plain_hex_string
Towards https://b.tvl.fyi/issues/264

Change-Id: Ibde971bfb6baa97b5c678d84ce1941189bc59f6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10969
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-19 17:32:10 +00:00
Peter Kolloch
df73d5f242 refactor(tvix/nix-compat): Restrict visibility
Towards https://b.tvl.fyi/issues/264

Change-Id: I323907cb842ed10ddd05b72c285c598b60d213e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10972
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-19 17:15:31 +00:00
Peter Kolloch
5777050821 feat(tvix/castore): Compile fix for Darwin
Towards https://b.tvl.fyi/issues/264

Change-Id: If8fa912ae3fb2987b761f649ab738529ebf3b2e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10970
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-19 17:14:24 +00:00
Aspen Smith
0db46dacea feat(tvix/glue): Init fetcher builtins
Initialize a new empty builtins module `fetcher_builtins`, which will
contain the builtins which fetch URLs from the internet:

* fetchurl
* fetchGit
* fetchTarball
* fetchTree (maybe? this is experimental)

These builtins are all implemented in CPP nix at:
https://github.com/NixOS/nix/blob/2.20.2/src/libexpr/primops/fetchTree.cc

These builtins are added to the evaluation context using a similar
mechanism to the derivation builtins, and have been added everywhere
derivation builtins were previously being added.

Change-Id: I133b91cc9560f23028621414537f712e7bd8a825
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10974
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-19 16:51:10 +00:00
Peter Kolloch
b147452948 feat(tvix/nix-compat): Extend parse errors
...so that we can parse (and at the same time
validate) store paths directly to `StorePath`s.

https: //b.tvl.fyi/issues/264
Change-Id: Ide4cb5403f8e3400f7d2a5838c652ac35dbd1a4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10968
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-19 16:36:31 +00:00
edef
f4963cdfd8 chore(tvix/tools/narinfo2parquet): polars 0.34.2 -> 0.36.2
Change-Id: Ib17a7bbe560000c3fef3ba6182819f710d707909
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10967
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-19 16:03:14 +00:00
Florian Klink
d991548ea2 fix(tvix/docs): update builder magic string
It's `builtin:fetchurl`, not `builtins:fetchurl`.

Change-Id: I2e90fce5feecc75257a5156aee12e0737a525175
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-02-19 15:52:14 +00:00
Florian Klink
c6605992c0 feat(tvix/glue): drive builds on IO
That's one possible abstraction to drive builds.
Whenever IO into a store path is requested, we look up the root node,
and in case we don't have it in PathInfoService, but KnownPaths gives us
a Derivation for that output path, trigger a build and await the result.

This recursively might trigger builds for parent paths if they haven't
been built yet.

Another option would be to simply expose a PathInfoService interface for
a builder too, and loop all building into IO via PathInfoService
composition - but let's start with something.

Note tvix-cli doesn't have a configurable BuildService yet, it's plugged
to the DummyBuildService, so whenever it needs to do a build, it'll fail,
but that's how it can be provoked:

```
(builtins.readFile (import <nixpkgs> {}).hello.outPath + "/bin/hello")
[…]
error[E029]: I/O error: /nix/store/cg8a576pz2yfc1wbhxm1zy4x7lrk8pix-hello-2.12.1: builds are not supported with DummyBuildService
 --> [code]:1:2
  |
1 | (builtins.readFile (import <nixpkgs> {}).hello.outPath + "/bin/hello")
  |  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

Note how this fails, while pure output path calculation
(`(import <nixpkgs> {}).hello.outPath + "/bin/hello")`) still succeeds.

Change-Id: Id2075d8a2b18554d0dd608b4b29146a8cd411e7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10793
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-02-18 08:59:49 +00:00
Florian Klink
6b22d4e1fa fix(tvix/cli): regenerate Cargo.nix
I forgot to regenerate this.

Change-Id: Ia7715d69b3b6316e612c1e3947c6ab819c7bf843
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10960
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-18 08:59:06 +00:00
Florian Klink
33e68496f2 fix(tvix/cli): set max level to info on release mode
Log levels can be statically disabled at compile time via Cargo
features.

Do this for everything more verbose than info in release mode.

Change-Id: I57d4282b4e19a8d44f68da2f15c7534dd3e08977
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10955
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-18 07:12:27 +00:00
Florian Klink
28f5c13c53 fix(tvix/castore): don't emit ret as INFO
This otherwise gets a bit spammy.

Change-Id: I288350a600d79a394c239f253424ad55bc3cefc5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10954
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-18 07:12:27 +00:00
Florian Klink
8253d91eaa fix(tvix/glue): don't emit ret as INFO
This causes a bit too much spam otherwise.

Change-Id: If3ced9ddfee7f49453711cd26469d1eb81983c71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-18 07:12:27 +00:00
Peter Kolloch
e87f2a2b3a feat(tvix/nix-compat): serde for StorePath[Ref]s
Necessary, if we want to use it inside of `Derivation` etc.

Change-Id: I8888060417b2ee83ac52d7ec3e7b27c393271d8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10947
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-17 08:11:46 +00:00
Florian Klink
d293304667 feat(tvix/cli): set up tracing[-subscriber]
No otlp yet, this simply gives us structured log output and a cli
argument for the log level.

Change-Id: Ifaa60bae419640e92baebb6ee59eedd775c769c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10853
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-02-17 08:06:42 +00:00
Florian Klink
87bda3ae7a feat(tvix/glue): tune instrumentations in TvixStoreIO
Print store paths with their ToString implementation for brevity, and
don't log the sucessful return value of read_to_string.

Change-Id: I01b6838398acd66b8818095622f361fcca26fa77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10854
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-02-17 08:01:04 +00:00
Florian Klink
44624284fd test(tvix/glue/known_paths): add tests
Change-Id: I02dc891c49ed650c66c403107767b11b6d5e2734
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10949
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: flokli <flokli@flokli.de>
2024-02-17 07:54:05 +00:00
Florian Klink
8cd93f3db5 refactor(tvix/glue/known_paths): use StorePath
Passing a StorePathRef is annoying if we only (already) have a
StorePath.

Change-Id: Ic3b36c0041707230515a6745a57f0d25b2bafd16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10948
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-17 07:52:35 +00:00
Florian Klink
cc8b7fee57 feat(tvix/glue/known_paths): add get_drv_by_output_path
This allows getting a Derivation struct producing the passed output
path.

Change-Id: I89858d91bffc2ef7f1d86314c16fa4f850f21426
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10791
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-17 07:45:28 +00:00
Peter Kolloch
48d4d10bac refactor(tvix/nix-compat): derivation_or_fod_hash w/o self.clone()
Instead of constructing a completely new derivation for hashing,
allow to call hashing with substituted input_derivations.

This

1. reduces the number of allocations,
2. prepares substituting `String`s for store paths with proper `StorePath`s without needing a separate derivation struct that allows hashes without `/nix/store` in input_derivations, and
3. keeps the change local to the `Derivation` implementation.

Change-Id: I36732c78f98fc59f0925b65823773222782017b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10935
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
2024-02-17 07:30:22 +00:00
Florian Klink
d4d46b9d8a feat(nix-compat/derivation): make fod_digest public
This will be used inside glue, to populate the CA field.

Change-Id: I52350d66ad09c90c4e9f3bc6ea44fba4427f1576
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10946
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-17 07:00:41 +00:00
Florian Klink
34a1ff291a feat(tvix/castore/fs): make allow_other configurable
Also add a cli argument to the `tvix-store` binary.

Change-Id: Id07d7fedb60d6060543b195f3a810a46137f9ad5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10945
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-17 07:00:41 +00:00
Florian Klink
58f474041e refactor(tvix/glue/KnownPaths): track Derivation struct too
We need to not only store a map from drv path to hash derivation modulo,
but also keep the original Derivation struct - because we'll use that
later to trigger builds.

Change-Id: I78e2e8431ab5ae853188866b797a79025200de98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10790
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-02-17 04:49:03 +00:00
Florian Klink
e7815945df feat(tvix/cli): support configuring BuildService
Allow using a BUILD_SERVICE_ADDR env var, or cmdline args to configure
it. Still default to the dummy implementation.

Change-Id: I68f34f7b09eabef2b0491103857bbc798398ebfc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10846
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-02-17 04:49:03 +00:00
Peter Kolloch
da3ce74f73 docs(tvix/nix-compat): on store paths in fields
Change-Id: Ie78b1c49d7386c70833a3b751799993ad9121aa6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10933
Tested-by: BuildkiteCI
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-17 04:47:49 +00:00
Florian Klink
3d51b20cca feat(nix-compat/derivation): simplify derivation_or_fod_hash
Populate the input_derivations struct field in-place, using
self.input_derivations.from_iter, which seems a bit more readable.

Change-Id: I4aa12cfd66e08486ef27aedab1bb731d94b63657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10934
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-17 04:46:20 +00:00
Florian Klink
8b06566d43 fix(third_party/nixpkgs/crate2nix): reintroduce tests in debug patch
This is was dropped accidentially, but we actually want to run tests in
debug mode, to check for overflows.

Change-Id: Ic0cf2d4a263bfd71f3453ed335d8be9dacd945e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10932
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-16 11:11:08 +00:00
Aspen Smith
98b89e53e2 test(tvix/eval): Add test asserting size of Value
Now that I've done a ton of things to make sure Value is small on the
stack (16 bytes, which is a perfectly reasonable size for a programming
language Value enum), add a test asserting it stays that way.

These size improvements have a measurable impact, too - here's the
`hello outpath` benchmark compared between canon (as of r/7495) and this
commit:

hello outpath time:   [990.56 ms 995.83 ms 1.0070 s]
              change: [-7.1397% -6.1302% -5.1651%] (p = 0.00 < 0.05)
              Performance has improved.

Change-Id: If99a0976eab28eb5e516fcd2f4a0e068145af23e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10799
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-02-13 16:49:53 +00:00
Aspen Smith
7e286aab1a feat(tvix/eval): Box Value::Catchable
This is now the only enum variant for Value that is larger than 8
bytes (it's 16 bytes), so boxing it (especially since it's not
perf-critical) allows us to get the Value size down to only 16 bytes!

Change-Id: I98598e2b762944448bef982e8ff7da6d6683c4aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10798
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: aspen <root@gws.fyi>
2024-02-13 16:49:53 +00:00
Aspen Smith
dd26177319 revert(tvix/eval): Don't double-box Path values
This reverts commit d3d41552cf.

This was well-intentioned, but now the boxed Path values are actually
the *largest* Value enum variants, at 16 bytes (because they're
fat-pointers, with a len) instead of 8 bytes like all the other values.
Having the double reference is a reasonable price to pay (it seems; more
benchmarks may end up disagreeing) for a smaller Value repr.

Change-Id: I0d3e84f646c8f5ffd0b7259c4e456637eea360f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10797
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-02-13 16:49:53 +00:00
Aspen Smith
e3c92ac3b4 fix(tvix/eval): Replace inner NixString repr with Box<Bstr>
Storing a full BString here incurs the extra overhead of the capacity
for the inner byte-vector, which we basically never use as Nix strings
are immutable (and we don't do any mutation / sharing analysis).
Switching to a Box<BStr> cuts us from 72 bytes to 64 bytes per
string (and there are a lot of strings!)

Change-Id: I11f34c14a08fa02759f260b1c78b2a2b981714e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10794
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-13 16:49:53 +00:00
Aspen Smith
5d72d3980f refactor(tvix/eval): Box the strings in CatchableErrorKind
These strings are allocated once and never changed, so they don't need
the additional overhead of a capacity given by String - instead, we can
use Box<str> and save on 16 bytes for each of these, *and* for each
Value since this is currently the largest Value variant.

Change-Id: I3e5cb070fe6c5bf82114c92d04f6bae775663a7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10796
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-02-10 20:35:28 +00:00
Aspen Smith
5d2ae840f1 refactor(tvix/eval): Box the inside of Value::Json
serde_json::Value is pretty large, and is contributing (albeit not
exclusively) to the large size of the Value repr. Putting it in a box
is *especially* cheap (since it's rarely used) and allows us
to (eventually) cut down on the size of Value.

Change-Id: I005a802d8527b639beb4e938e3320b11ffa1ef23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10795
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-02-10 20:34:25 +00:00
Ryan Lahfa
7b1632ec71 feat(tvix/eval): strengthen significantly catchable test suite
Adds a bunch (notably certain overlapping) tests for catchable situations.

This should cover many scenarios, argument is catchable, element in argument is catchable, function returns
catchable in the middle of the processing, etc.

Co-authored-by: Aspen Smith <root@gws.fyi>
Change-Id: Icd722cf8dbc91a24f45cd540a328711e5826f76c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10621
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-02-10 15:34:15 +00:00
Florian Klink
d10c5309bc feat(tvix/castore/blobsvc): add Chunked{Blob,Reader}
These provide seekable access into a Blob for which we have more
granular chunking information.

There's no support for verified streaming in here yet, this simply
produces a stream of readers for each chunk, skipping irrelevant chunks
and data from the first chunk at the beginning.

A seek simply does produce a new reader using the same process.

Change-Id: I37f76b752adce027586770475435f3990a6dee0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10731
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-10 14:24:51 +00:00
Aspen Smith
3fb0697a71 fix(tvix/eval): Propagate catchables in NixAttrs::construct
Correctly propagate the case where the *key* of an attrset is a
Value::Catchable (eg { "${builtins.throw "c"}" = "b"; }) in
`NixAttrs::construct`, by converting the return type to
`Result<Result<Self, CatchableErrorKind>, ErrorKind>` (ugh!!) and
correctly handling that everywhere (including an `expect` in the
Deserialize impl for NixAttrs, since afaict this is impossible to hit
when deserializing from stuff like JSON).

Change-Id: Ic4bc611fbfdab27c0bd8a40759689a87c4004a17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10786
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-02-09 19:11:09 +00:00
Aspen Smith
8446cd1c8b fix(tvix/eval): Inline List.sort_by, and propagate errors
In order to correctly propagate errors in the comparator passed to
builtins.sort, we need to do all the sorting in a context where we can
short-circuit return `Value`s (because catchables are Values on the `Ok`
side of the Result , not `Err`s). Unfortunately this means we have
to *inline* the List `sort_by` implementation into the builtin_sort
function - fortunately this is the only place that was called so this is
relatively low cost. This does that, and adds the requisite `try_value!`
invocation to allow us to propagate comparator errors here.

As before, this doesn't include tests, primarily since those are coming
in the next commit.

Change-Id: I8453c3aa2cd82299eae89828e2a2bb118da4cd48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10754
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-02-08 22:50:27 +00:00
Aspen Smith
ddb7bc8d18 fix(tvix): Catch errors for generator in some builtins
Nix doesn't propagate errors for the function argument to some builtins,
like genList and map:

    ❯ nix repl
    Welcome to Nix version 2.3.17. Type :? for help.

    nix-repl> (builtins.tryEval (builtins.genList (builtins.throw "a") 10)).success
    true

    nix-repl> (builtins.tryEval (builtins.map (builtins.throw "a") [ "" ])).success
    true

Note that this is untested as of this particular commit, only because a
big test suite covering all sorts of catchable error propagation issues
is coming next

Change-Id: I48c8eb390a541204b1a6d438c753fa1ca9b3877e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10753
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-02-08 21:58:53 +00:00
Aspen Smith
b77a103a2a fix(tvix/eval): Propagate catchables in a few more places
Propagate catchables that we get from forcing thunks in builtins in a
few more places using the new try_value! macro

Change-Id: I95fd41a231f877ff153f4adbabd944372d4cc7eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10738
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-02-08 19:59:21 +00:00
Aspen Smith
780b47193a refactor(tvix/eval): Generalize propagation of catchable values
Rather than explicitly checking for Value::Catchable in all builtins,
make the #[builtin] proc macro insert this for all strict arguments by
default, with support for a #[catch] attribute on the argument to
disable this behavior. That attribute hasn't actually been *used*
anywhere here, primarily because the tests pass without it, even for
those builtins which weren't previously checking for Value::Catchable -
if some time passes without this being used I might get rid of support
for it entirely.

There's also a `try_value` macro in builtins directly for the places
where builtins were eg forcing something, then explicitly propagating a
catchable value.

Change-Id: Ie22037b9d3e305e3bdb682d105fe467bd90d53e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10732
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-02-08 19:59:21 +00:00
Florian Klink
4e040e8bc4 chore(tvix/tools/turbofetch): bump magic-buffer to 0.1.1
This contains https://github.com/sklose/magic-buffer/pull/4, so we don't
have to impl Send ourselves.

Change-Id: If046596e13345ad4fec22209440e65859e44d540
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10748
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-02-07 20:03:24 +00:00
Nikita Voloboev
70c068df51 fix(tvix): fix build on darwin
on macOS additional framework was needed to build tvix build

Change-Id: I0e327378d1bd4837f62496c4c6e66bc489ddedb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10747
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-06 19:10:59 +00:00
Florian Klink
40d81d0c74 docs(tvix/castore/blobstore): reorganize docs
docs/verified-streaming.md explained how CDC and verified streaming can
work together, but didn't really highlight enough how chunking in
general also helps with seeking.

In addition, a lot of the thoughts w.r.t. the BlobStore protocol, both
gRPC and Rust traits, as well as why there's no support for seeking
directly in gRPC, as well as how clients should behave w.r.t. chunked
fetching was missing, or mixed together with the verified streaming
bits.

While there is no verified streaming version yet, a chunked one is
coming soon, and documenting this a bit better is gonna make it easier
to understand, as well as provide some lookout on where this is heading.

Change-Id: Ib11b8ccf2ef82f9f3a43b36103df0ad64a9b68ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10733
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-06 18:28:00 +00:00
Florian Klink
0d55a6dcc8 docs(tvix/store/pathinfosvc): document nix+http[s]
Change-Id: I5f4e93aceec373eb3377e3f8fbb55a5142cc1363
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10740
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-05 18:40:52 +00:00
Florian Klink
172173d744 docs(eval/docs): toJSON is done
Change-Id: I67381b19eedb651f4049e20e7b1e43641a116ae5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10739
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-02-04 17:13:52 +00:00
Florian Klink
cb2cf3f6b7 fix(tvix/castore/grpc/svc_wrapper): expose chunks() over gRPC
The Stat() method was just always signalling no granular chunks are
available. However, as we now have a .chunks() method, we can expose it
over gRPC.

Change-Id: I74f0890ae083f301bb0cec62f1ea4a95463ac590
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10736
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-02 16:27:10 +00:00
Florian Klink
9504015031 feat(tvix/castore/blobsvc): validate StatBlobResponse
All chunks must have valid blake3 digests. It is allowed to send an
empty list, if no more granular chunking is available.

Change-Id: I7ecb53579cdf40fd938bb68a85685751b4d3626f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10726
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-02-02 16:26:38 +00:00
Florian Klink
5ad5a0da00 refactor(tvix/castore/grpc/blobsvc): inline stream_mapper
This can be written without the additional function.

Change-Id: Ib11c5d5254d3e44c8fa9661414835b0622eb1ac4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10735
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-02 16:25:37 +00:00
Florian Klink
1157eea710 docs(tvix/castore/blobsvc): fix doc comments on trait
The readers implement AsyncRead/AsyncSeek, not their sync counterparts.
Also update expectations around chunks.

Change-Id: Ic266688039d80d16d33f651b96ce2bcdedecfa00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10734
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-02 16:24:06 +00:00
Aspen Smith
5f0f4ea374 refactor(tvix/eval): Box Value::String
NixString is *quite* large - like 80 bytes - because of the extra
capacity value for BString and because of the context. We want to keep
Value small since we're passing it around a lot, so let's box the
NixString inside Value::String to save on some memory, and make cloning
ostensibly a little cheaper

Change-Id: I343c8b4e7f61dc3dcbbaba4382efb3b3e5bbabb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10729
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-02-02 16:16:56 +00:00
Florian Klink
4c5d9fa356 feat(tvix/castore/docs/verified-streaming): clarify reply
"given chunksize" is misleading here. It's up to the backend to decide
if it does chunking at all, and how it chunks.

Change-Id: I4f130ca9ac34db79f18ef1d6475295806ac7f9a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10728
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-02-02 08:56:48 +00:00
Florian Klink
459a564ff1 refactor(tvix/castore/blobsvc/combinator): compact trait bounds
BlobService already implies Send and Sync, we don't need to explicitly
list it here.

Change-Id: I58a4c5912be61a60acd961565979aa01d94ee0f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10727
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-02-02 08:55:16 +00:00
Aspen Smith
8caa097ba8 feat(tvix/eval): Don't emit OpForce for non-thunk constants
In the compiler, skip emitting an OpForce if the last op was an
OpConstant for a non-thunk constant. This gives a small (~1% on my
machine) perf boost, eg when evaluating hello.outPath:

    ❯ hyperfine \
        "./before --no-warnings -E '(import <nixpkgs> {}).hello.outPath'" \
        "./after --no-warnings -E '(import <nixpkgs> {}).hello.outPath'"
    Benchmark 1: ./before --no-warnings -E '(import <nixpkgs> {}).hello.outPath'
      Time (mean ± σ):      1.151 s ±  0.022 s    [User: 1.003 s, System: 0.151 s]
      Range (min … max):    1.123 s …  1.184 s    10 runs

    Benchmark 2: ./after --no-warnings -E '(import <nixpkgs> {}).hello.outPath'
      Time (mean ± σ):      1.140 s ±  0.022 s    [User: 0.989 s, System: 0.152 s]
      Range (min … max):    1.115 s …  1.175 s    10 runs

    Summary
      ./after --no-warnings -E '(import <nixpkgs> {}).hello.outPath' ran
        1.01 ± 0.03 times faster than ./before --no-warnings -E '(import <nixpkgs> {}).hello.outPath'

Change-Id: I2105fd431d4bad699087907e16c789418e9a4062
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10714
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-01 21:08:35 +00:00
Aspen Smith
d3d41552cf refactor(tvix/eval): Don't double-box Path values
PathBuf internally contains a heap pointer (an OsString), so we were in
effect double-boxing here. Removing the extra layer by making
Tvix::Value represented by a Box<Path> rather than a Box<PathBuf> saves
us an indirection, while still avoiding the extra memory overhead of the
capacity which was the reason we were boxing PathBuf in the first place.

Change-Id: I8c185b9d4646161d1921917f83e87421496a3e24
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10725
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-02-01 17:25:42 +00:00
Aspen Smith
201173afac fix(tvix): Represent strings as byte arrays
C++ nix uses C-style zero-terminated char pointers to represent strings
internally - however, up to this point, tvix has used Rust `String` and
`str` for string values. Since those are required to be valid utf-8, we
haven't been able to properly represent all the string values that Nix
supports.

To fix that, this change converts the internal representation of the
NixString struct from `Box<str>` to `BString`, from the `bstr` crate -
this is a wrapper around a `Vec<u8>` with extra functions for treating
that byte vector as a "morally string-like" value, which is basically
exactly what we need.

Since this changes a pretty fundamental assumption about a pretty core
type, there are a *lot* of changes in a lot of places to make this work,
but I've tried to keep the general philosophy and intent of most of the
code in most places intact. Most notably, there's nothing that's been
done to make the derivation stuff in //tvix/glue work with non-utf8
strings everywhere, instead opting to just convert to String/str when
passing things into that - there *might* be something to be done there,
but I don't know what the rules should be and I don't want to figure
them out in this change.

To deal with OS-native paths in a way that also works in WASM for
tvixbolt, this also adds a dependency on the "os_str_bytes" crate.

Fixes: b/189
Fixes: b/337
Change-Id: I5e6eb29c62f47dd91af954f5e12bfc3d186f5526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10200
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-01-31 14:51:49 +00:00
Aspen Smith
6f9e25943f feat(tvix/eval/observer): Allow capturing timing of events
Add a new --trace-runtime-timing flag (probably a better bikeshed for
this) that enables capturing the time, relative to the last event, of
each event recorded with the tracing observer.

This probably isn't *super* useful yet, but I'd like to start here in
adding new profiling tools to the VM, specifically based on the runtime
observer

Change-Id: Id7f12077291c39bf3eef42ab6744bfba53687a65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10713
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-01-31 04:41:37 +00:00
Florian Klink
b38be028d9 feat(tvix/tools/crunch-v2): add CLI args
Use clap derive to make the input and output files configurable, as well
as the chunk size parameters.

Change-Id: I02b29126f3bd2c13ba2c6e7e0aa4ff048ff803ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10691
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-01-27 18:40:48 +00:00
edef
4f22203a3a feat(tvix/tools/crunch-v2): init
This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format.
Currently, produced chunks are not preserved, and this purely serves as a way of measuring
compression/deduplication ratios for various chunking and compression parameters.

Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-27 18:23:40 +00:00
Florian Klink
e0a1c03b24 feat(tvix/eval/tvix_tests): add some more xml tests
https: //cl.tvl.fyi/c/depot/+/10686/comment/ea582dae_574d6c3f/
Change-Id: Iac74abbf2f2e0327bc9ddf9dcc6bb43f918a1c63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10689
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-25 12:06:44 +00:00
Florian Klink
850e50cd3f feat(tvix/eval/tests): compare .xml outputs
In case a .exp.xml file is provided alongside the test, compare its
output with the desired state.

Also, add some function .exp.xml that were presumably moved out of the
way back to src/tests/nix_tests, as they now produce the correct XML
output.

Change-Id: Ibd8123f3e6ed7bae3a44407d2284a2b2c8ce9a28
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10687
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-25 11:37:35 +00:00
Florian Klink
023e372583 feat(tvix/eval): track pattern binding names
These need to be preserved at least for builtins.toXML.

Also, we incorrectly only wrote an <attrspat> in case ellipsis was true,
but that's not the case.

Change-Id: I6bff9c47c2922f878d5c43e48280cda9c9ddb692
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10686
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: aspen <root@gws.fyi>
2024-01-25 11:37:35 +00:00
Florian Klink
e1d2589163 fix(tvix/eval/value/function): use BTreeMap for function arg names
At least toXML wants to get these out in a sorted fashion.

Change-Id: I6373d7488fff7c40dc2ddeeecd03ba537c92c4af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10685
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-24 14:00:44 +00:00
Florian Klink
adff7be4d1 feat(tvix/eval): expose value_to_xml for test cases
It's debateable on whether the serialization code should be exposed a
bit more prominently or not.

Change-Id: Iff7a28f884b1490b12b145dfdadbedacb84fd387
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10684
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-24 14:00:44 +00:00
Florian Klink
8e517bc8d0 fix(tvix/eval/tests): fix eval-okay-getenv
This relies on TEST_VAR=foo being set to "foo".

Nix does this in tests/functional/lang.sh, we do it in the test suite.

Change-Id: I7ffa9ed27124530b7758aeadf07c79477656f34f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10683
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-24 14:00:44 +00:00
Connor Brewster
85421b7f97 feat(tvix/store/nar-info): Support async blob upload
Right now all blob uploads are performed synchronously, this means if a
NAR contains many small files, the import time is dominated by round
trip time to the blob service. For small files, we can buffer them in
memory and upload them asynchronously to the blob service. Before
returning we make sure to join all the uploads to make sure they
complete successfully before responding OK.

This reduces time to import a bash-interactive closure on my machine
from 1m19s to 7s.

Change-Id: Ica3695c159e6c8ad8769281ac20d037e3143e856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10679
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-23 17:01:16 +00:00
Connor Brewster
d056329412 feat(tvix/store/nar-bridge): Setup OpenTelemetry
Sets up OpenTelemetry integration for nar-bridge. Right now it will
export spans for HTTP server requests and all gRPC client requests.
Having the spans available will make performance work significantly
easier as it provides a high level overview of where time is being
spent.

In the future we can add application-specifc metrics and
integrate logrus.

Change-Id: Ie3860675d7ffc626a95673ba062c3c798d8bb2a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10678
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-23 15:50:53 +00:00
Florian Klink
e8061fc619 docs(tvix/eval/tests): update comment
There's no `expected_failures` feature, we run them unconditionally.

Change-Id: Ibe1c93497e040d0d5b6cbfcaa043027814c191f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10681
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-01-23 12:16:14 +00:00
sterni
842d5ed3ee fix(tvix/glue/tests): resolve logic error in hasContextInAttrKeys
Change-Id: I7aa09c10f803c2b79363fe3eb52c1127276a333b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10676
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-22 14:24:51 +00:00
sterni
bdeb6f406e docs(tvix): initial notes on a possible generic Nix lang test suite
This kind of collects points to consider which should hopefully help in
figuring out what such a lang test suite could or should look like
exactly—which is something I currently struggle somewhat.

Change-Id: If4f47546fe4b8046fb79718743fa9a72f9801876
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10657
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-22 14:21:19 +00:00
Ryan Lahfa
68bba48d59 feat(tvix/castore): process_entry cannot process unsupported nodes
In the past, we had a `todo!` on unsupported node types, this returns a proper error
that can be caught by the caller.

Change-Id: Icba4c1dab33c0d670a97f162c9b358d1ed5855cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10675
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-22 14:15:42 +00:00
Connor Brewster
4e341fb5d9 chore(tvix/store): Use BoxStream type alias
The BoxStream type alias is a more concise and easier to read than
the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type.

Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-21 19:41:02 +00:00
Florian Klink
56ba7a72d8 docs(tvix/eval): builtins.toXML is done
Our docs say it's still todo, but that's wrong, it landed in cl/7835
and cl/7962.

Change-Id: Ice374ae47ab653146be18c02dce27145e7488397
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10674
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-01-20 20:09:07 +00:00
Ryan Lahfa
3d8aafb1c7 feat(tvix/store): enable name customization in the store
Sometimes, Nix lets someone customize the `name` in the store for a
path, this is the case for `builtins.path` which takes a `name`
argument, we leave it to the caller to choose the name, which can be the
basename by default of the path.

Change-Id: Icdbf71d1d8f2dca5716b99d20aac885aab905b80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10653
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 18:35:50 +00:00
Ryan Lahfa
4c3ba46ba3 refactor(tvix/store): import_pathimport_path_as_nar_ca
Add multiple additional helpers such as:

- `path_to_name`: derive the basename of a given path
- `derive_nar_ca_path_info`: derive the `PathInfo` for a content
  addressed NAR

which isolates further the tree walking feature and the ingestion feature.

Additionally, we don't `expect` anymore and propagate properly ingestion errors up.

Change-Id: I60edb5b633911c58ade7e19f5002e6f75f90e262
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10574
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-20 18:26:17 +00:00
Ryan Lahfa
7275288f0e refactor(tvix/castore): break down ingest_path
In one function that does the heavy lifting: `ingest_entries`, and three additional helpers:

- `walk_path_for_ingestion` which perform the tree walking in a very naive way and can be replaced by the user
- `leveled_entries_to_stream` which transforms a list of a list of
  entries ordered by their depth in the tree to a stream of entries in
  the bottom to top order (Merkle-compatible order I will say in the
  future).
- `ingest_path` which calls the previous functions.

Change-Id: I724b972d3c5bffc033f03363255eae448f017cef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10573
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-20 18:26:17 +00:00
Ryan Lahfa
1f1a42b4da feat(tvix/castore): ingestion does DFS and invert it
To make use of the filtering feature, we need to revert the internal walker to a real DFS.

We will therefore just invert the whole tree by storing all of its
contents in a level-keyed vector.

This is horribly expensive in memory, this is a compromise between CPU
and memory, here is the fundamental reason for why:

When you encounter a directory, it's either a leaf or not, i.e. it
contains subdirectories or not.

To know this fact, you can:

- wait until you notice subdirectories under it, i.e. you need to store
  any intermediate nodes you see in the meantime -> memory penalty.
- getdents or readdir on it to determine *NOW* its subdirectories -> CPU
  penalty and I/O penalty.

This is an implementation of the first proposal, we pay memory.

In practice, we are paying O(#nb of nodes) in memory.

There's a smarter albeit much more complicated algorithm that pays only
O(\sum_i #siblings(p_i)) nodes where (p_1, ..., p_n) is the path to a leaf.

which means for:

             A
            / \
           B   C
          /   / \
         D   E   F

We would never store D, E, F but only E, F at a given time.
But we would still store B, C no matter what.

Change-Id: I456ed1c3f0db493e018ba1182665d84bebe29c11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10567
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 17:16:01 +00:00
Ryan Lahfa
e98ea31bbd fix(nix-compat): accept SRI hashes of invalid length
In cl/10468, we accepted SRI hashes of invalid padding while checking
their trailing bits.

In this commit, we accept SRI hashes of invalid padding and invalid length, as Nix does.

Real world example: `pkgs.javaPackages.openjfx11.deps`
<849e4dc5ff/pkgs/development/compilers/openjdk/openjfx/11.nix (L71)>
in nixpkgs.

Change-Id: I834437e7b94dab9fbb030163f7a2741f52bbf03a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10668
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 17:14:30 +00:00
Florian Klink
a720278843 fix(tvix/glue/benches/eval): add impure_builtins
Impure builtins need to be explicitly added. This was missed from
cl/10640.

Change-Id: I1f8a84aa02f1cf8a26247960d6eb15895e09c610
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10667
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-20 00:33:04 +00:00
sterni
526295a71d chore(3p/sources): Bump channels & overlays
- Adjust to ecl 23.9.9 release

- Regenerate go protos after protoc-gen-go update

- Drop dhall fork which hasn't kept up with 1.42.*

- Address new clippy warnings:

  - Variant naming of Error::ValidationError
  - Simplify .try_into().unwrap()
  - Drop unnecessary identity function
  - Test module must be last in file
  - Drop unused `pub use`

- Update agenix to 0.15.0. Current master has a installCheckPhase that
  doesn't work with C++ Nix 2.3.*:
  a23aa271be (commitcomment-137185861)

Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-19 21:47:32 +00:00
Florian Klink
4497ac41ab refactor(tools): move crate2nix generate here
Having something running the depot crate2nix and formatting it with
depotfmt is useful outside of tvix too.

Change-Id: Iecc8f207da38cc6995747c5ea48d3911433fd416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10658
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-18 15:02:15 +00:00
Ryan Lahfa
93afc711f6 feat(tvix/castore): convert import error to std::io::Error
So that we can just `map_err` easily in functions returning `std::io::Error` but calling functions
returning `castore::import::Error`.

Change-Id: Id181b95e8431c69e95f3a8cd569ca10306656e1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10572
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-18 14:40:06 +00:00
Florian Klink
501827db59 refactor(tvix/glue): add BuildService to TvixStoreIO
TvixStoreIO triggers builds whenever IO into a not-yet-built store path
is requested, if it knows how to build that path.

Change-Id: If30e9db6be2f2a30cbc9d0576f357f3ecfa0d35a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10645
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-18 14:30:23 +00:00
Ryan Lahfa
12ae96cff2 feat(tvix/glue): use TvixStoreIO as derivation builtin state
We propagate a `TvixStoreIO` as the `state` of our derivation-specific
builtins in the glue crate.

The evaluators `io_handle` itself is using a Rc<dyn EvalIO>.

An earlier version of TvixStoreIO was also introducing generics over the
different internal services themselves, but we opted for instead
hardcoding this to Arc<dyn …> for the sake of less macro voodoo.

Change-Id: I535c476f06b840858fa3070c4a237ece47f7a15b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10636
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-18 14:28:00 +00:00
Florian Klink
43b9e25025 refactor(tvix/eval): move Evaluation::{default,new_pure}() again
Have a Evaluation::new() function that's used to set up the Evaluation
struct initially - which is also used by both new_pure and new_impure
internally.

It's generic over the exact type of IO, making it easier to instantiate
Evaluation with non-tvix-eval EvalIO implementations, that might not be
in a Box.

Change-Id: Ibf728da24aca59639c5b6df58d00ae98c99a63f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10640
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-18 14:24:40 +00:00
Ryan Lahfa
ea03ff374b chore(tvix/shell): add cargo-expand to debug proc-macro
I lost a lot of hope and had to read the source code of `quote!`, `cargo expand` was invaluable
in this adventure. We should keep it IMHO.

Change-Id: Icfb4c80d413602f2bdc6deab0d595183825d88ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10635
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-18 12:29:46 +00:00
Florian Klink
e0a867cabf refactor(tvix/eval): generalize EvalIO container
Don't restrict to a Box<dyn EvalIO>.

There's still one or two places where we do restrict, this will be
solved by b/262.

Change-Id: Ic8d927d6ea81fa12d90b1e4352f35ffaafbd1adf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10639
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-18 09:09:15 +00:00
Ryan Lahfa
44d24852c3 fix(tvix/eval): catchable-aware throw
`throw (throw "a")` should work and propagate the internal throw.
Before this commit, it didn't work.

Change-Id: Id5d46f74e484dba99e912ad9fa211f3bf1617bac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10600
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 17:39:32 +00:00
Ryan Lahfa
e5e33611d7 fix(tvix/eval): catchable-aware elem
`elem` did not catch the list being a catchable.
This surfaced during Nixpkgs evaluation.

Change-Id: Icf19b94e914e35a435c4412d769ee63ba59ab7b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10599
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-17 17:39:28 +00:00
Ryan Lahfa
f71bb351d2 feat(tvix/glue): introduce test suite for context strings
This is an additional test suite on the top of the Nix ones
for context strings matters.

It already smoked out multiple mistakes and potential bugs and non-deterministic result from the evaluator.

It uses a similar technology as the one in the tvix-eval albeit we instantiate a fully fledged evaluator
with in-memory store.

We copy the files instead of symlinking them because crates are built in
isolation, so symlinks cannot work.

Change-Id: I63ae225ce4f83c6e2c8ccd60d779c2f8eb9d08fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10619
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 17:31:07 +00:00
Ryan Lahfa
75cc52ddb1 fix(tvix/eval): getContext merges underlying values
Previously, we were assembling very naively an attribute set composed of context we saw.

But it was forgetting that `"${drv}${drv.drvPath}"` would contain 2 contexts with the same key, but
with different values, one with `outputs = [ "out" ];` and `allOutputs = true;`.

Following this reasoning and comparing with what Nix does, we ought to merge underlying values systematically.

Hence, I bring `itertools` to perform a group by on the key and merge everything on the fly, it's not
beautiful but it's the best I could find, notice that I don't use
`group_by` but I talk about group by, that is, because `group_by` is a
`group_by_consecutive`, see
https://github.com/rust-itertools/itertools/issues/374.

Initially, I tried to do it without a `into_grouping_map_by`, it was akin to assemble the final `NixAttrs` directly,
it was less readable and harder to pull out because we don't have a lot of in-place mutable functions on
our data structures.

Change-Id: I9933c9bd88ffe04de50dda14f21879b60d8b8cd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10620
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-17 17:25:24 +00:00
Ryan Lahfa
97e6c39dcd fix(tvix/eval): context-aware dirOf
`dirOf` forgot to accepts contextful strings, e.g. derivations and propagates this context
further.

Change-Id: I6c05944a3ce5073e243e7676c9be56c48407d657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10618
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-17 07:57:43 +00:00
Ryan Lahfa
c2d0e245e2 fix(tvix/eval): context-aware… hasContext
Yes, `hasContext e` should work where `e` is a contextful strings, otherwise, it is really useless.

Change-Id: I5eb071fc257217d6e8a63fe519132ebd98186696
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10617
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 07:57:38 +00:00
Ryan Lahfa
bc8fb825c7 feat(tvix/eval): move away from test_generator to rstest
`test-generator` has not been updated in the past 2 years.
`rstest` has not been updated in the past 5 months.

This is an improvement in the maintenance state… I guess?
We get also new features, it changes the name of the tests with numbers too.

Change-Id: I5376104c7704f525dba7524da78daa09867cc669
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10623
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 07:34:02 +00:00
Florian Klink
850a4bfc7b chore(third_party/crate2nix): run tests in /build/source
Cherry-pick of https://github.com/nix-community/crate2nix/pull/328. This
should fix rstest runs inside crate2nix-generated derivations.

Change-Id: I9d393768f7f764e33c5938bd8fa14bd1bb0b72e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10650
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-17 07:33:12 +00:00
edef
b624edb2ae fix(tvix/eval): lift VM ops over Catchable
We want to handle bottoms in a consistent fashion. Previously this was
handled by repetitive is_catchable checks, which were not consistently
present.

Change-Id: I9614c479cc6297d1f64efba22b620a26e2a96802
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10485
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-16 19:20:28 +00:00
sterni
99615002a7 fix(tvix/cli/ci): prevent unnecessary dep on compared out/drv path
Change-Id: Ieab16fd5fad67a4ed0124f118aef8abbefd51cdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10646
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-16 16:50:59 +00:00
Florian Klink
e2c79e39f0 refactor(nix-compat): use StorePathRef for hash derivation modulo
Rather than passing strings around, use a StorePathRef.

This makes things a bit more typesafe, and more aligned with what we
want to do in b/264.

Change-Id: Ib7080addf27e7f1a9c8da1d8aaa66744468e3b5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10633
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-16 08:37:07 +00:00
Florian Klink
c8114810c9 chore(third_party/overlays): bump crate2nix to 0.13.0
We need to vendor in the package expression, as it's not possible to
override cargoHash.

Change-Id: Ib123647bb9b96d41f4630daa431d020f1cb8d4fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10624
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-15 23:14:21 +00:00
Florian Klink
f0a750bcb7 feat(tvix/build): add CLI entrypoint
This starts a BuildService as a separate process, currently defaulting
to the DummyBuildService.

Change-Id: Ic206f00831641d3ffebaa44883b7dc053700b9ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10631
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-15 18:54:35 +00:00
Florian Klink
170e0cdfad feat(tvix/build): add from_addr method
This allows constructing a BuildService from a URI, similar to how it's
done in tvix-[ca]store.

Change-Id: Ib962b329535c6c7e378ab7ac7f4dd254366497b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10630
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-15 18:49:33 +00:00
Florian Klink
5aa9b29d8c feat(tvix/build): add gRPC client
Change-Id: I0d917a9a308227b13d096def22945db917830d18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10629
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-15 18:41:58 +00:00
Florian Klink
c01ec8ee38 feat(tvix/build): add GRPCBuildServiceWrapper
This produces a gRPC BuildService server for anything implementing our
BuildService trait.

Change-Id: I59c690a432b5e1f59209fd67e2718cb8c935adf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10628
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-15 18:41:58 +00:00
Florian Klink
4fb4fc263d feat(tvix/build): add BuildService trait
Also provide a dummy implementation that just fails on any build that's
requested.

Change-Id: I0df743a730c5331ec9ce6e97a966abe18ce067f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10627
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-15 18:25:19 +00:00
Florian Klink
9fd15ba506 refactor(tvix/glue): have derivation_to_build_request consume inputs
Determining the inputs might trigger additional builds/substitutions,
so answering these lookups via a lambda in a lazy fashion gets
complicated.

You end up assembling the list of input nodes upfront, and the lambda
will just be a dumb lookup into that preassembled list.

Rather than doing that, simply have derivation_to_build_request leave
the work of determining the inputs to the caller.

Change-Id: I75880132916c76b930807c989090da298b6891bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10626
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-15 18:25:19 +00:00
Florian Klink
83291ff51e refactor(tvix/glue/known_paths): drop some unused stuff
This are leftovers from the "reference scanning" approach (which we
didn't end up using).
We still want a concept of known paths, so we can trace IO into
storepaths back to the build recipe that'll produce it, so let's keep
the rest of this struct around.

Change-Id: I73d38e21e5b97950b8fc2a42176cae5f80d371c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10632
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-15 18:25:19 +00:00
Florian Klink
c5e2832cbd feat(tvix/castore): implement Ord for node::Node
This allows assembling BTreeSets of node::Node.

Change-Id: I97b83be5ffc3e891307a8ef2b5fc31e38b747a62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10625
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-15 18:19:15 +00:00
Ryan Lahfa
2750e1e640 fix(tvix/eval): catchable-aware builtins
A bunch of operations in Tvix are not aware of catchable values
and does not propagate them.

In the meantime, as we wait for a better solution, we just offer this
commit for moving the needle.

Change-Id: Ic3f0e1550126b0847b597dfc1402c35e0eeef469
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10473
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-14 03:37:17 +00:00
Ryan Lahfa
61209f0775 feat(tvix/glue): input derivation context tests
`args` was not propagating context, here's a regression test for it.

Change-Id: I8b6a3148508d40df0077128f0bafe68c098a03bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10610
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-12 22:32:22 +00:00
Florian Klink
d516ce56b1 feat(tvix/glue/derivationStrict): support __structuredAttrs
This adds support to handle the __structuredAttrs argument, which can be
passed to builtins.derivationStrict.

If __structuredAttrs is passed, and set to true, most of the arguments
passed to builtins.derivationStrict are not simply coerced to a string
and passed down to "environments", but instead kept in a more structured
fashion.

Inside ATerm, which is what's relevant as far as path calculation is
concerned, a virtual `__json` environment variable is present,
containing these structured values.

Inside Builds, these structured values are not made available as an
environment variable, but a JSON file (and source-able bash script).

This will need to be respected once we start emitting BuildRequests,
and for that we can probably just parse the `__json` key in
Derivation.environment again - or keep this additionally in
non-serialized form around during Evaluation.
No matter what, this is left for a followup CL.

The existing handle_derivation_parameters and populate_outputs helper
function were removed, as __structuredAttrs causes quite a change
in behaviour, and so handling both in the same place makes it more
readable.

There's some open questions w.r.t. string contexts for structured attrs
itself. A TODO is left for this, but at least path calculation for
individual structured attrs derivations are correct now.

Part of b/366.

Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 22:25:35 +00:00
Florian Klink
82540717d6 feat(tvix/eval): make into_json public
Allow other crates (like tvix-glue) to look at a Value in JSON, which is
used by the structured attrs feature.

Change-Id: Iba02ace6e11a74c3f9b19dcbef4b008b76dec046
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10602
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-12 22:25:35 +00:00
Ryan Lahfa
c955560767 docs(tvix/glue): misc reflowing of comments
My OCD could not be stopped.

Change-Id: I2bf504fe0865a5084ad02aee18e6180a8a3e19d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10609
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-12 22:08:34 +00:00
Florian Klink
ce66af09a7 fix(tvix/glue/tvix_build): fn_input_drvs_to_output_nodes
The Derivation input_derivations field contains a list of input
derivations and (a subset of their) output names.

This means, multiple nodes can be returned, so return a Vec.

Also, update the name to better reflect the nodes are the nodes of the
selected outputs, not a node representing the .drv file itself.

Additionally, use a proto::node::Node (the naked enum), rather than
proto::Node, which wraps this in an optional struct field until
realizing the BuildRequest.

Change-Id: Iec5620b5d7ac0462f2c76acac4abcaeea2de0aad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10608
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-12 20:38:38 +00:00
Florian Klink
639ee19101 refactor(tvix/glue/tvix_store_io): async store_path_to_node
Provide a store_path_to_node_sync function which uses the runtime handle
to block on the async function internally, but make store_path_to_node
itself async, so it can call async functions internally.

We'll use that later when triggering builds and waiting on their
results.

Change-Id: Idae9da7aa5b0878e0d3a2eba34ea2623e1ba84b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10607
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 20:38:36 +00:00
Florian Klink
b59df53774 refactor(tvix/store/pathinfoservice): make more generic
We don't need Arcs in most of the cases, we're fine with some container.

Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 20:37:35 +00:00
Florian Klink
7d51193f7d refactor(tvix/store/nar/renderer): don't require Arc, Clone or Sync
To render NARs, we're fine with a simple AsRef to a BlobService and
DirectoryService. We just need to have the function pass back the
references, so we can reuse it after the recursion.

Change-Id: I8a1b899134ddda26cf14aa829a08383986101850
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10605
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-12 20:37:03 +00:00
Florian Klink
9cdb5e17a4 fix(tvix/eval): fix JSON error types
The error message is misleading. The errors we return can happen both
during serialization or deserialization, though the messages suggested
the latter only.

Change-Id: I2dafe17ec78ee75cab5937a3a81540fda3175eac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10603
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-12 09:22:37 +00:00
Florian Klink
4d135bcfa2 feat(tvix/castore): implement CombinedBlobService
First attempt on composition of BlobServices.

Change-Id: I6e70248007edfd322a503fd40c1c4b4300cbc30c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10587
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-09 17:32:03 +00:00
Florian Klink
719cbad871 feat(tvix/castore/blobsvc): add chunks method
This adds support to retrieve a list of chunks for a given blob to the
BlobService interface.

While theoretically all chunk-awareness could be kept private inside
each BlobService reader, we'd not be able to resolve individual chunks
from different Blobservices - and due to this, not able to substitute
chunks we already have in a more local store.

This function allows asking a BlobService for the list of chunks,
leaving any actual fetching up to the caller (be it through individual
calls to open_read), or asking another store for it.

Change-Id: I1d33c591195ed494be3aec71a8c804743cbe0dca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10586
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-09 17:31:32 +00:00
Florian Klink
9596c5caff refactor(tvix/castore): do clone inside a scope
Make it clear this is only used inside the scope.

Change-Id: Ie94f88d7f0fb58cd4bf9c2f1176000b272e6f2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10585
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 16:20:19 +00:00
Florian Klink
9de1ebf23e feat(tvix/castore/grpc): instrument some more functions
Change-Id: Icedb148c88c5f4a3b2242ed12df1dd8692af94fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10584
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 16:18:48 +00:00
Florian Klink
99f675ecef refactor(tvix/store/nar/import): use AsRef
We need to be a bit careful and pass the BlobService around (similar to
how we already do with the directory_putter), but that allows getting
rid of a bunch of annoying trait bounds.

We also stop spawning additional tasks where we can just use block_on.

Change-Id: If36de0ee947d2c779d20a384308241d2262d4764
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10580
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-01-09 14:18:26 +00:00
Florian Klink
0009383c07 refactor(tvix/castore/directorysvc): AsRef traverse_to
Change-Id: I641bd4ab3de591a013f03137f1e16295946315f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10579
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-09 14:15:55 +00:00
Florian Klink
b1c556b7e1 refactor(tvix/castore/blobservice/grpc): remove fn pointer hack
It looks like the workaround isn't necessary anymore.

Change-Id: Ifbcef1d631b3f369cac3db25a2c793480043f697
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10583
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:13:24 +00:00
Florian Klink
89882ff9b1 refactor(tvix): use AsRef<dyn …> instead of Deref<Target= …>
Removes some more needs for Arcs.

Change-Id: I9a9f4b81641c271de260e9ffa98313a32944d760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10578
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:08:22 +00:00
Florian Klink
8fbdf72825 feat(tvix/castore/blobsvc/grpc): rm VecDec, fix docstring
The docstrings were not updated once we made the BlobService trait async.
There's no more need to turn things into a sync reader.

Also, rearrange the stream manipulation a bit, and remove the need to
create a new VecDeque for each element in the stream. bytes::Bytes
implements the Buf trait.

Fixes b/289.

Change-Id: Id2bbedca5876b462e630c144b74cc289c3916c4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10582
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 14:03:21 +00:00
Florian Klink
17eaacb139 feat(tvix/glue): add test for output path calculation
This test serves as a minimal reproducer for output path calculation.

Derivations with the same name and output hash, but different build
recipe should end up with the same outPath.

However derivations with different name should end up with a different
outPath.

Change-Id: I555be59dd87ea675a0816188ed973f96c311e4e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10416
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-01-08 11:13:16 +00:00
Florian Klink
986e9b73c3 feat(tvix/build): add BuildRequest validation
Change-Id: I8182e4c4a7e5694c1e6f1f56ce092751c22adf4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10538
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-06 15:33:12 +00:00
Florian Klink
c9c95f4ef3 chore(tvix): bump itertools
Change-Id: I2c798a30d3e69789c7d9120422bfdd67b1560dd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10565
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-06 15:33:12 +00:00
Ryan Lahfa
cbcd078684 chore(tvix/castore): fix the docstring for process_entry
It was a `//` not a `///`.

Change-Id: Iee3e8c116d73b5dd8a41c027153714415a66695f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-06 01:39:41 +00:00
Florian Klink
00aa3260c9 fix(tvix/build): fix inconsistencies around paths
Change-Id: I76ce4e042f6fbe4bc897767b55ce4eb1f7269e5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10541
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-01-05 16:49:10 +00:00
Florian Klink
5f0360c566 refactor(tvix/glue): simplify TvixStoreIO further
We don't need to spawn in all these places, we can just block_on
directly, this is all IO bound.

This also means, we don't need to clone any of the service handles
(except preserving clone-ability of the BlobService).

Change-Id: I7d90f4d6a263a98491caa071ada538a5197a5472
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10540
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-05 16:49:10 +00:00
Florian Klink
4284cd82ef refactor(tvix/glue): simplify store_path_to_[root]_node
This was wrongly named, it returns a specific node at a subpath.

Also, this code can be simplified a lot - we don't need to spawn
additional tasks, and can get rid of some clones too.

This is also where we need a certain build - so add some TODO to block /
fetch here.

Change-Id: Id26d7bd80f7a2095121e642b3f7716de78d6b6a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10539
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-05 16:43:37 +00:00
Florian Klink
5a82736122 chore(tvix): bump test-case dep to 3.3.1
Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-01-05 16:43:34 +00:00
Florian Klink
70e40f5ce8 docs(tvix/build): document requirements for paths and sorting
Change-Id: I466fac025b9b5b2279a6188fa35cb1f7d8ab7bc5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10536
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-05 16:43:34 +00:00
Florian Klink
8e794478bf feat(tvix/store/pathinfosvc): call validate
Change-Id: I557ed67a04112ba3e363303637d1988309250a93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-05 16:43:34 +00:00
Florian Klink
6b42aef88d fix(tvix/castore): validate Option<Node>
Extend our validation function to also check for the None case.

Change-Id: Ib75f880646d7fb3d66588f1988e61ec18be816a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10534
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-05 16:43:34 +00:00
Florian Klink
f20969de9b refactor(tvix/castore): relax trait bounds for DS
Make this an `AsRef<dyn DirectoryService>`.

This helps dropping some Clone requirements.

Unfortunately, we can't thread this through to TvixStoreIO just yet.

Change-Id: I3f07eb28d6c793d3313fe21506ada84d5a8aa3ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-05 16:43:34 +00:00
Florian Klink
3297843bf1 chore(tvix): run clippy with --{benches,examples}
While doing the obvious thing, ensuring clippy doesn't complain on our
benchmarks and examples, this will also flag build failures on them,
and contrary to cl/10301 not pollute the /nix/store with stuff we can't
execute anyways.

Change-Id: I7ee1c2babbc67dac9794274d05bb9d6e1251ed01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10544
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-05 16:37:30 +00:00
Aspen Smith
e66dcba195 fix(tvix/eval): Update eval benches for new API
cl/10475 updated the API of tvix_eval::Evaluation, but didn't update
this benchmark to use that new API.

Also, fixes the docstring to no longer specify that there is a "given
snippet".

Change-Id: Ibb8285731849dbeec814e2585bbaa36f22368afe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10542
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2024-01-05 16:30:51 +00:00
Ryan Lahfa
54ad5a19e7 feat(tvix/cli): test firefox (drv|out)Path
With context strings, we are now able to reproduce up to firefox.

Until a more problematic example can be found, we should at least
lean on this particular example.

Change-Id: Ibeaf799d26e1f160b6c93f8ccd978702a2a7e3e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10460
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:48:03 +00:00
Ryan Lahfa
de6c8b7546 feat(tvix/cli): context-aware raw printing
Raw printing will transform the result into a string
and print it.

In case of a contextful string, this will fail by default, as expected.

Change-Id: I5e564329e7b001adc57a77a9153b4425cb332bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10457
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:48:03 +00:00
Ryan Lahfa
f44ac2a594 feat(tvix/eval): context-aware split
Nix does something like:

```cpp
        NixStringContext context;
        const auto str = state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.split");
```

And then do nothing with that context, therefore, we follow them and
make `split` aware of the context but still do nothing with it.

Change-Id: I4fee1936600ce86d99d00893ca3f64013213935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10428
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 23:24:07 +00:00
Ryan Lahfa
556e52c9cb feat(tvix/eval): impl unsafeDiscardStringContext
Change-Id: I7f0cc42cbebfe5cd27bf6d4f58a4af927b83646a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10423
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:24:07 +00:00
Ryan Lahfa
099ca6b7c0 feat(tvix/glue): contextful derivation
We calculate the input context by performing the union of context
over all input of the derivation.

Then, we just pass the rest of it to the remaining machinery.

Finally, we re-emit an `outPath` and a `drvPath` containing the expected
contexts.

Change-Id: I74905fb258b5bee8b08d1208c9eb87f51b92a890
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10436
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 23:23:33 +00:00
Ryan Lahfa
20c894e232 feat(tvix/glue): context-aware toFile
This removes the reference tracking and uses instead the context for
references and returns some.

Change-Id: Ic359ca6b903b63f1a9c679c566004c617b792442
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10435
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 19:43:10 +00:00
Ryan Lahfa
cc098b9aaa feat(tvix/eval): contextful coercion of files
In the past reference tracking system, `tvix-io` glue was appending
plain paths in the known path state.

Now, we make up for this by just making contextful coercion of file
imports.

Change-Id: Ieb9b04dd83302c77909252d5f7733857ac3cf8fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10443
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:50:29 +00:00
Ryan Lahfa
951854defc feat(tvix/eval): contextful string coercion
String with contexts are always coerced to a string with the same
context.

Change-Id: I224814febd9cad196bb28876793e76bed564dc72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10440
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-01-03 18:49:57 +00:00
Ryan Lahfa
9d43c26576 feat(tvix/eval): contextful == of derivations
Otherwise, you just fail because they are not... contextless strings!

Change-Id: I0b8f63a18cd89c3841b613d41c12ec4ee336f953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10442
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-03 18:49:57 +00:00
Ryan Lahfa
207f6fed46 feat(tvix/eval): match DO NOT propagate context
`match` silently ignore the input context and do not propagate
it and successful matches.

The why is unclear but nixpkgs does rely implicitly on this behavior
because dynamic attribute selection cannot be done with contextful
strings.

Change-Id: I5167fa9b2c2db8ecab0c2fb3e9895c9cfce6eeb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10441
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:49:57 +00:00
Ryan Lahfa
09ec8b6fcf feat(tvix/eval): implement getContext primop
Change-Id: I2c5068a28f9883a01b0ff80a5e5ab32ba18bfc1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10437
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-03 18:20:43 +00:00
Ryan Lahfa
96d06031af feat(tvix/eval): context-aware replaceStrings
And it also preserve the original context if it exists.

Change-Id: I904f7c13b7f003a267aace6301723780fccaafb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10434
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 18:16:09 +00:00
Ryan Lahfa
cbd22af2b5 chore(tvix/eval): note on context-aware hashString
It must propagate context too.

Change-Id: If57c22c9723ea02aa013f69d3dcf96054476d8de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10433
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:15:36 +00:00
Ryan Lahfa
e372b1d1a5 feat(tvix/eval): ${} propagates contexts
We just perform union of contexts of every pieces.

Change-Id: Ief925c1818cd8bbec0503e9c625b0630feebfdda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10432
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 18:15:36 +00:00
Ryan Lahfa
0e6285ec8b feat(tvix/eval): context-aware concatStringsSep
Change-Id: Id0e169084d26dc598091d157563c4d959b66279b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10431
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:15:04 +00:00
Ryan Lahfa
88fcab68d4 chore(tvix/eval): note on context-aware toString
Change-Id: Ie26ebd16e95e6a7b6f81051d8269169842978058
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10430
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 18:14:02 +00:00
Ryan Lahfa
c7d6d08799 feat(tvix/eval): context-aware throw
Change-Id: Ie552dabe4cf93cc396c883268a3bee67796dbbd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10429
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 18:09:30 +00:00
Ryan Lahfa
7dcb37fc52 feat(tvix/eval): context-aware substring
`substring` has a very funny behavior when it comes to empty strings,
it propagates the context too, this is used in nixpkgs to attach context
to strings without using any builtin: `lib.addContextFrom`.

Change-Id: Id655356799b3485f7519b3d1914c630f9d8416c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10448
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 18:08:58 +00:00
Ryan Lahfa
375f7eaa59 feat(tvix/eval): context-aware abort
Change-Id: Id5a435961ce3a2a2240b3936ea48515650d445d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10427
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 18:07:56 +00:00
Ryan Lahfa
e259a5c42b feat(tvix/eval): context-aware dirOf
Change-Id: If73a82a7106de9b479c950741efb70bffabd470a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10424
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-03 16:57:35 +00:00
Ryan Lahfa
db748741da feat(tvix/eval): context-aware baseNameOf
Change-Id: I9f0a8143070805b85276f721bdfbdf7ede2cf615
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10421
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-03 16:54:33 +00:00
Ryan Lahfa
e15b6580cc chore(tvix/eval): notes on coercion and contexts
We make a case for adding a `reject_context` `CoercionKind` here.
It does happen during concatenation actually for path concats.

Change-Id: I0c196aad917550b9bcd0896cd2127a94f8181ffb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10444
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 16:54:00 +00:00
Ryan Lahfa
743c362049 feat(tvix/eval): context-aware coerce_to_string
I am still undecided whether we need a CoercionKind to control
the coerced context, here's a simple attempt.

Change-Id: Ibe59d09ef26c519a6acfdfe392014446646dd6d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10426
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-03 16:45:26 +00:00
Florian Klink
802f374a90 feat(tvix/glue): handle passAsFile
This extends derivation_to_build_request to handle passAsFile the same
way Nix does, and adds a unit test for it.

I opted to making this function fallible (if passAsFile contains a
non-existent env var), rather than pushing all of this into the
Derivation validate function.

Change-Id: I75b635f1f6c0c78d72b9a8fc7824f77e97b69951
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10522
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-03 14:15:15 +00:00
Florian Klink
a82214b3ad fix(tvix/store): set service.name
While this can be influenced with `OTEL_SERVICE_NAME` or
`OTEL_RESOURCE_ATTRIBUTES`, "unknown_service" is an annoying name.

Let's pick tvix.store as a default.

Change-Id: I9b5c45703d9546775bb9266b29ab54daf7c54a9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10524
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-03 13:29:35 +00:00
Florian Klink
9abe735061 refactor(tvix/glue): create environment_vars Vec at the end
Prevent code in this function from accessing the Vec, realize it only
when returning the BuildRequest struct.

Change-Id: I385c1adca552c9cb261cd70c5ca3ec45f0d7a9c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10521
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 13:19:02 +00:00
Florian Klink
7f030fe265 test(tvix/glue): add test with passAsFile
This already succeeds, as the output path calculation path does not need
to be aware of any builder-specific custom handling for passAsFile.

Change-Id: I9cc9e4e4351cdeaa3ec33ba58ee1569e7a368150
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10520
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-03 13:19:02 +00:00
Florian Klink
e6782bb7c4 feat(tvix/build/protos): add additional_files field
Nix builds also have some other inputs, outside of inputs_dir, used in
derivations setting `passAsFile` and `__structuredAttrs`.

This extends our `BuildRequest` structure to accomodate these usecases.

Change-Id: I8de6516eb467fa48a4961d88933ebd548f131049
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10519
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2024-01-03 13:18:31 +00:00
Florian Klink
6b136dfd23 feat(tvix/glue): emit a warning in case of bad SRI hashes
And include a test to ensure we show the warning.

Change-Id: Ib6a436dbba2592b398b54e44f15a48d1aa345099
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10470
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-03 13:01:02 +00:00
Ryan Lahfa
d5aa75bbcf feat(tvix/eval): emit warnings with kinds and spans
In the past, we had `emit_warning` be no-op and we used `push_warnings` exclusively
but as we have consumers of this function, we need it to work somewhat.

Change-Id: I78a5ece199a473dec9ef5ea1fae60b36e35137b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10477
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-03 13:01:02 +00:00
Florian Klink
7a84a8fe89 fix(nix-compat/nix-hash): relax padding requirements
Nix is quite tolerant when it comes to parsing SRI hashes and their
padding (and only for SRI hashes, it funnily is strict about that in the
non-SRI-hash case).

Nix essentially accepts any number of padding characters, no matter if
it's too much or too little. So we do the only sane thing - simply strip
all padding characters, and parse it with BASE64_NOPAD and the length
the algo uses.

Change-Id: I6a721aa289b06cc36741589792b9dd4c4f930b86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10468
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-03 13:01:02 +00:00
Florian Klink
3307791855 feat(tvix/store): add opentelemetry
This is behind the otlp feature flag (currently enabled by default).

By default, this will try to push traces to a OTLP collector running at
https://localhost:4317.

You can make one available by running:

```
docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 --rm \
  jaegertracing/all-in-one:1.42
```

Started like that, jaeger brings a web interface at
http://localhost:16686/search

As documented in
https://docs.rs/opentelemetry-otlp/latest/opentelemetry_otlp/, you can
point this to another location by setting `OTEL_EXPORTER_OTLP_ENDPOINT`.

Change-Id: Id1dca367d70027b2ea98bb70bcf99a68363ec2be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8194
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: aaqaishtyaq <aaqaishtyaq@gmail.com>
2024-01-01 17:27:39 +00:00
Florian Klink
597a6b6205 refactor(tvix/castore/tests): let gen_*_service return Boxes
Only convert to and reuse an Arc<…> where needed.

Change-Id: I2c1bc69cca5a4a3ebd3bdb33d6e28e1f5fb86cb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10514
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 14:45:17 +00:00
Florian Klink
09a92b78d2 refactor(tvix/store/nar/import): be a bit more generic
Change-Id: If9a536949f36f428abea1a893f937fe7063e2f41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10517
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 14:43:07 +00:00
Florian Klink
1b62f82b10 refactor(tvix/castore/blobsvc/grpc/wrapper): don't require Arc<_>
Change-Id: I9655f5588c7dc98427de6af47d74b4ab7ce22071
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10516
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 14:42:36 +00:00
Florian Klink
96aa220dcf refactor(tvix/castore/directorysvc/grpc/wrapper): no Arc<_>
We can also drop the Clone requirement. Because the trait is async since
some time, there's no need to clone before moving into an async closure,
allowing us to simplify the code a bit.

Change-Id: I9b0a0e10077d8c548d218207b908bfd92c5b8de0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10515
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-01-01 14:40:35 +00:00
Florian Klink
54fe97e725 refactor(tvix/castore): make directorysvc more generic
This works on Box<dyn DirectoryService> too.

Change-Id: Ib869f0f4d963ef4dbaeab22db03ff6afb71ede04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10513
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-01 02:09:20 +00:00
Florian Klink
2c2fdfedc6 refactor(tvix/glue/tvix_store_io): use utils::import_path
This is doing the exact same thing.

Change-Id: Iadc5e13dd192efc91cc3d36b2bdf4b8b99a312b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10512
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 01:55:46 +00:00
Florian Klink
52a61e353b feat(tvix/store/bin): factor out import
While at it, make it a bit more generic.

Change-Id: Ic4caefda93aca3ffb656a09f8b4d648b41415532
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10511
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-01-01 01:55:46 +00:00
Florian Klink
ddae4860c2 feat(tvix/castore/import): generalize ingest_path
We don't actually care if it's an Arc<dyn BlobService>, or something
else, as long as we can Deref to a BlobService and clone.

Change-Id: I0852aaf723f51c5e6b820be8db1199d17309ab08
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10510
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-01 00:54:14 +00:00
Florian Klink
fdd7817aad feat(nix-compat/store_path): build_ca_path may fail
Change-Id: Ia74ee870f38b7966501458bace541092256c3213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10509
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-01 00:52:41 +00:00
Florian Klink
20c98a51bf refactor(tvix/store/bin): don't reconstruct StorePath twice
We already have that data in output_path.

Change-Id: Iadf615551f0b44d539098446ff5926994deb606d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10508
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 00:42:38 +00:00
Florian Klink
fa335aaa68 fix(tvix/store/bin): don't unwrap in case of invalid paths
Instead, return an error, and move the entire check before starting to
ingest the data underneath.

Change-Id: Idcfba115cb7d599f5fc72a156aaad9d4d4714fcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10507
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-31 22:23:49 +00:00
Florian Klink
694ed7ea1a refactor(tvix/store/bin): condense subscriber setup a bit
We can use cli.json.then_some(…) to create a Some(…), allowing us to
omit the else { None } lines.

Change-Id: I6c8142a08d8cb88d6c8302e5ca7570698fcf2aa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10505
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:23:49 +00:00
Florian Klink
f6d1a56c8c refactor(tvix/store): move construct_services helper here
This takes three URLs, and constructs Arc'ed
{Blob,Directory,PathInfo}Service, allowing to remove some of the
boilerplate.

Change-Id: I40e7c2b551442ef2acdc543dfc87ab97e7c742bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10484
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-31 22:23:49 +00:00
Florian Klink
41935fab70 refactor(tvix/castore/directorysvc): return Box, not Arc
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.

Change-Id: Ia05bb370340792a048e2036be30e285ef1e63870
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10483
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:18:14 +00:00
Florian Klink
9ca1353122 refactor(tvix/castore/blobsvc): return Box, not Arc
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.

Change-Id: I7bd337cd2e4c2d4154b385461eefa62c9b78345d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10482
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 22:18:14 +00:00
Florian Klink
e2b6c77bfc feat(tvix/cli): allow configuring different services
At some point, tvix-cli needs to talk to the outside world to persist
things into a real store.

Introduce the same CLI options to configure
{Blob,Directory,PathInfo}Service URLs.

We need to be a bit careful with how we set up stores, and make
this separate from setting up TvixStoreIO, as it's holding a
Rc<RefCell<KnonPath>> which not Send.

At some point, we might make this a Arc<RwLock<_>> later anyways, and
then this can be simplified a bit, but for now, this is sufficient.

Change-Id: I87d84ca3a10ce947e194ff985073791469773f35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10474
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 21:55:18 +00:00
Florian Klink
12f9b95a2c feat(tvix/eval): accept impl AsRef<str> for code
We're also happy to consume strings, or other owned stringy types.

Change-Id: I5bead4407976134815d8f879f9f70468e6af1dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10476
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-31 13:15:24 +00:00
Florian Klink
4fba57c2c9 refactor(tvix/eval): remove code and location from struct
Instead, it's passed in the evaluate/compile_only functions, which feels
more naturally. It lets us set up the Evaluation struct long before
we actually feed it with data to evaluate.

Now that Evaluation::new() would be accepting an empty list of
arguments, we can simply implement Default, making things a bit more
idiomatic.

Change-Id: I4369658634909a0c504fdffa18242a130daa0239
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10475
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-12-31 13:15:24 +00:00
Florian Klink
4d6137964b feat(tvix/build): don't force outputs to be in store_dir
There's no need to require builds to also put their outputs in store_dir.

So rename store_dir to inputs_dir and make outputs consume paths, not
basenames.

Also document the possibility for builds to write to inputs_dir. Let's
see if we want to make this a constraint later.

Change-Id: Ib4df20afcdde2d771c269c422f04c7e95587cd0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10447
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-12-29 22:14:08 +00:00
Ryan Lahfa
3e63d7be42 feat(tvix/eval): context-aware casting to strings
By default, we don't want contextful strings and we almost always want contextless strings.

To this end, we make taking a contextful string a very explicit operation under `to_contextful_str`
and we implement manually the `to_str` cast which requires a `if !s.has_context()` guard that
the macro cannot cover.

Change-Id: I7aae8e57a7d73e547e62b1edb0b1cc7e8c0c69b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10425
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-29 22:00:04 +00:00
Ryan Lahfa
343ee0de57 feat(tvix/eval): implement hasContext primop
`hasContext` is now functional.

Change-Id: I23b128afc9150b833bc0d9b042d31fee35badadb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10422
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-29 21:58:32 +00:00
Ryan Lahfa
5dfb15d2c8 feat(tvix/eval): introduce NixContext
This prepares the data structures to implement string contexts
in Nix.

Change-Id: Idd913c9c881daeb8d446907f4b940e462e730978
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10420
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-29 21:57:30 +00:00
Ryan Lahfa
fc182061c6 docs(tvix/eval): why context strings now
Explain why we had to take the bullet on context strings now
and move the historical approach in a historical section.

Change-Id: Ie3bcc2213b391c6ba06547cc05c850891a41d06b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10446
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-29 21:55:58 +00:00
Adam Joseph
4b3f27ebc9 fix(tvix/eval): propagate catchables through builtins.attrNames
Change-Id: Id14e39543239272aed041998fd9a78465c9cb8b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10359
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
9cb3cd10f5 fix(tvix/eval): propagate catchables through builtins.intersectAttrs
Change-Id: I4ada8cf10611e98519cb1b1da11bfd06815f2932
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10358
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
2883b2d4bd fix(tvix/eval): propagate catchables in string interpolations
Change-Id: I13d7ce0c7328a8e6fbc6d2c4ff5c4fe6095b96ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10357
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
7ddea7340f fix(tvix/eval): catchable in type field of nix_eq()
Change-Id: I165ff77764e272cc94d18cb03ad6cbc9a8ebefde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10348
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
2af8174e2e fix(tvix/eval): builtins.match: propagate catchables
Change-Id: I14c9e625c91369e10d0c00380dca992811ae9059
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10346
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
11e35a77a6 refactor(tvix/eval): let OpCoerceToString select the CoercionKind
Change-Id: I92d58ef216d7e0766af70f019b3dcd445284a95d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10344
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Adam Joseph
0c22454bb9 fix(tvix/eval): add stack depth assertion to OpReturn
I'm still trying to work out the exact stack invariants for tvix.
We really should add assertions for them; getting the stack messed
up is no fun.  This commit adds one simple assertion.  It also adds
a missing stack-push (my mistake) in one place, which was uncovered
by the assertion.

Change-Id: I9d8b4bd1702d954e325832c5935b0d7e3eb68422
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10369
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-29 21:34:45 +00:00
Florian Klink
8a52c7f1c5 feat(tvix/castore/fs): borrow some matches
We only do things with the reference, so we don't need to locally borrow
it.

Change-Id: I6073f7ec7aff717ae3069e28a00b1cb408a50ceb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10455
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-29 17:18:14 +00:00
Florian Klink
5e3683cd53 feat(tvix/shell): add grpcurl to shell
This is a bit more useful than evans when sending over the same message
over and over again.

Change-Id: I600b6b9f591c0c963c5d270005aa1cc84d2a0770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10453
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-12-29 15:57:26 +00:00
Florian Klink
22427dc8d0 refactor(tvix/castore/bin/tvix-store) don't import std::io
Make it more clear this is not tokio::io.

Change-Id: Ic2fa56f0baf1c200b6631098d556388a19629a45
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10452
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-29 15:57:26 +00:00
Florian Klink
46a372d5d7 feat(tvix/castore/fs): instrument FuseDaemon functions
Change-Id: I696b7ab6b4c08004db147c0fda7312bbebaa0eec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10451
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-29 15:57:26 +00:00
Florian Klink
0af445c2f4 feat(tvix/store): emit more structured log messages
Change-Id: If4748cb800276fc42779ba8973d1eef72df0503e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10450
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-29 14:37:58 +00:00
Florian Klink
f6c94430c8 feat(tvix/build/protos): add some missing fields
- directory in which the castore input nodes are mounted
 - working directory for the build command
 - scratch paths
 - network access y/n
 - whether a (static) /bin/sh should be provided

Populate these fields appropriately, and extend the tests in tvix-glue
with a FOD example.

Change-Id: I4f9de1483d6696d74694a09784910c407acb0be0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10412
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-26 10:20:41 +00:00
sterni
d07600dbca fix(tvix/eval/value): correctly emit spaces when coercing lists
r/7176 introduced an incorrect assumption was the benefit of the
nonrecursive coercion algorithm, namely that a coercion operation always
returns a non empty string. This allows to detect whether we are
coercing a list or not by checking if the intermediate result is empty
or not. Unfortunately, coercing null and false yields an empty string,
so we need to explicitly track whether we are coercing a list.

Updated the test case to hopefully catch similar bugs in the future. I'm
not a hundred percent certain I have not introduced a new edge case with
this, so it may be interesting to add a prop test case for this to
nix_oracle down the line. At least lists are the only nested data
structures that can be serialized as nested data structures, so the
problem is kind of limited.

Change-Id: Ia41e904356f1c41a9d35e4e65ec02f2fe5a4100e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10418
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-25 15:06:23 +00:00
Florian Klink
d4dc28b675 chore(tvix/turbofetch): switch to futures 0.3.30
The bugs have been fixed,
https://github.com/rust-lang/futures-rs/pull/2801 and
https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended
up in that release.

Change-Id: I301c0ffc951f04a5b3b7267e922771c837a3f5a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10415
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-24 21:45:04 +00:00
Florian Klink
acbb613e61 chore(tvix): switch to upstream futures 0.3.30
The bugs have been fixed,
https://github.com/rust-lang/futures-rs/pull/2801 and
https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended
up in that release.

Change-Id: Iefd990d2d1719b884504093343e54e9c5258e2e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10414
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-24 21:45:04 +00:00
Florian Klink
a5865ec7fa refactor(tvix/castore/fs/tests): drop unused args
There's no need to pass in an unused directory service into the
populate_blob_* method, and considering we have one or two invocation of
each of these, we don't really gain much from having all these functions
follow the same structure, at least for now.

Also, update some function names to better describe what they're doing.

Change-Id: I92f680745c157fb0a602b07342f8838bfad23ecd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10411
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-24 16:05:52 +00:00
Florian Klink
8d86d2f409 refactor(tvix/castore): add RootNode impl for BTreeMap, mv fs tests
cl/10378 did already move store/fs to castore/fs, but we kept the tests
in tvix-store, as they were populating a PathInfoService to make nodes
appear in the mount root.

Update these tests to now just insert root nodes into a BTreeMap, and
ensure we can use that as a RootNodes too.

Change-Id: Iad7d1ee4f9423eb6e3a1da33f433842c9ae0de1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10410
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-24 15:44:30 +00:00
Aspen Smith
6af17545bf feat(tvix/website): Link to windtunnel results page
Change-Id: If4456c565abf32385b1239467b96e7398190ea4b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10383
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-23 17:12:25 +00:00
Florian Klink
a5167c508c chore(tvix): move store/fs to castore/fs
With the recent introduction of the RootNodes trait, there's nothing in
the fs module pulling in tvix-store dependencies, so it can live in
tvix-castore.

This allows other crates to make use of TvixStoreFS, without having to
pull in tvix-store.

For example, a tvix-build using a fuse mountpoint at /nix/store doesn't
need a PathInfoService to hold the root nodes that should be present,
but just a list.

tvix-store now has a pathinfoservice/fs module, which contains the
necessary glue logic to implement the RootNodes trait for a
PathInfoService.

To satisfy Rust orphan rules for trait implementations, we had to add a
small wrapper struct. It's mostly hidden away by the make_fs helper
function returning a TvixStoreFs.

It can't be entirely private, as its still leaking into the concrete
type of TvixStoreFS.

tvix-store still has `fuse` and `virtiofs` features, but they now simply
enable these features in the `tvix-castore` crate they depend on.

The tests for the fuse functionality stay in tvix-store for now, as
they populate the root nodes through a PathInfoService.

Once above mentioned "list of root nodes" implementation exists, we
might want to shuffle this around one more time.

Fixes b/341.

Change-Id: I989f664827a5a361b23b34368d242d10c157c756
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-22 16:55:18 +00:00
Florian Klink
52cad86195 refactor(tvix/store): remove Arc<> from PathInfoService::from_addr
This makes PathInfoService::from_addr return a Box<dyn PathInfoService>,
rather than an Arc<dyn …>, and leaves it up to the consumers to rewrap
it into an Arc where needed.

This allows us to drop the Arc for the tvix-store daemon subcommand.

Change-Id: Ic83aa2ade6c51912281bd17c7eef7252e152b2d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10409
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-22 16:55:18 +00:00
Florian Klink
93a228b9a4 refactor(tvix/store/bin): move service.clone right before async move
Change-Id: I7a2b951d9c9251a053a0de40f31836bda03a922d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10408
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-22 16:55:11 +00:00
Florian Klink
39cddb95be refactor(tvix/store/bin): remove unneeded clone
Change-Id: Ib94376db3a57853d980112919c2d6fcc0f566883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10407
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-12-22 13:29:04 +00:00
Florian Klink
9627ef15de docs(tvix/castore/protos): remove reference
This is not gonna end up as a interlinked docstring.

Change-Id: I2b0ca106aa75bae0156c0b411da5931da60c725d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10406
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-12-21 16:44:48 +00:00
Florian Klink
3b274f47af docs(nix-compat/nar/reader): fix reference
Change-Id: I7ccdfe3c042f49aa962d1c7ba6337bd656ef05c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10405
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:44:16 +00:00
Florian Klink
12406fe241 docs(nix-compat/derivation): fix docstring
Change-Id: I76200aa474b86cf655308f6710a2298b77fa4b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10404
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:44:16 +00:00
Florian Klink
b04fdeaa10 docs(nix-compat/derivation): fix reference
Change-Id: I0ebe256c62435901c12ca1895f30e380280e9ea8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10403
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:44:16 +00:00
Florian Klink
410abd67f8 docs(nix-compat/aterm/parser): fix docstring
Change-Id: I7a65ce320163c5898df402af3a639f6cee2e17c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10402
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:43:44 +00:00
Florian Klink
783a1e314c docs(tvix/castore): fix reference
Change-Id: I00b1d56d58c4d3779b57ab0056cff1c7e6053b9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10401
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-12-21 16:43:44 +00:00
Florian Klink
3fc4d295d5 docs(tvix/store): fix reference
Change-Id: If1970cb8eb9a21c30011c9303f2fcbff646ad12b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10400
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:43:44 +00:00
Florian Klink
5bee6c6be0 docs(tvix/store): fix reference
Change-Id: Ifb77097755f25aa6536c60365941507a592cac3e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10399
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:43:11 +00:00
Florian Klink
6838dc3fff docs(tvix/glue): fix docstring
This is not an unclosed <html> tag.

Change-Id: I2bd2426fc600de2d96dbab47743f1c7bd5fed35e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10398
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:43:11 +00:00
Florian Klink
ca0a8f5603 docs(nix-compat/nixhash): fix reference
Change-Id: I95c35a1405601c6f79cbc03462ab7c7cd6f08f2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10397
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-12-21 16:43:11 +00:00
Florian Klink
1c3ea9d83d docs(nix-compat/drvfmt): fix reference
Change-Id: Ic0403e028cc968bef4023ef78dbfa0d502da90b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10396
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:39:07 +00:00
Florian Klink
8c83838c40 docs(nix-compat/narinfo): fix reference
Change-Id: Ib3a5414afed738ba6178ec69d38a4169c9b24714
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10395
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:36:34 +00:00
Florian Klink
18702c740e docs(nix-compat/narinfo/fingerprint): fix reference
Change-Id: Icc56aa178247b3985a406421b7d8d449c63f8231
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10394
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-12-21 16:36:34 +00:00
Florian Klink
e4eb0b9bf4 docs(nix-compat/derivation): fix references
Change-Id: If6962931350edfbae206fa739a673e7ccbc3ee89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10393
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:36:34 +00:00
Florian Klink
329a7d30a7 refactor(nix-compat/store_path): centralize self_reference check
self_reference being set to true is only allowed for
`CAHash::Nar(NixHash::Sha256(_))`, so we can handle this in a check at
the front.

Change-Id: Ic363ade4789a7767cbe26a6959b143bb53e50e5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10391
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-21 16:36:29 +00:00
Florian Klink
88adaea12b refactor(nix-compat/store_path/utils): restructure build_ca_path
All match cases essentially construct `ty` and `hash`, which is then
passed to the `build_store_path_from_fingerprint_parts` function.

Change-Id: I01dfd219f9b0ac1afe8af7c6e361ea048117a0e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10390
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:28:53 +00:00
Florian Klink
9065089b0c refactor(tvix/nix-compat): have helpers interact with StorePathRef
In most case, we don't actually need an owned `StorePath` struct, but a
`StorePathRef<'_>` is sufficient. The lifetime is only due to it holding
onto the name, but that one is mostly left untouched.

`Derivation::calculate_derivation_path` still needs to return
`StorePath`, as its name has a `.drv` appended.

Change-Id: Ie0d52f369d785711bb0658ea2b0bd2617fd9f45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10389
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:28:53 +00:00
Florian Klink
ea8fdb9304 refactor(nix-compat/narinfo/fingerprint): move to StorePathRef
With the introduction of StorePathRef::to_absolute_path(), there's no
need to convert this StorePathRef to a StorePath first.

Change-Id: I634c977c4b63858e4f329fd21726e0611b99da4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10388
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-12-21 16:28:53 +00:00
Florian Klink
91deb67de9 refactor(nix-compat/store_path): use StorePathRef::to_absolute_path
Keep the method around in StorePath for convenience, but move the
implementation to StorePathRef.

Change-Id: Ie1844fa01ce6529dc1a58907563c95c3112c831d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10387
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:28:41 +00:00
Florian Klink
c7ea57df46 feat(nix-compat/store_path): derive [Partial]Eq
This allows comparing StorePathRef structs.

Change-Id: Ia69967ea9358052e2d6e76042a7e6d394f7f29a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10386
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2023-12-21 16:19:46 +00:00
Luke Granger-Brown
e20848ecf1 chore(depot): update OWNERS files for aspen
Change-Id: Id94b646a6ea035782298c421d6667530da6fc5b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10384
Tested-by: BuildkiteCI
Owners-Override: lukegb <lukegb@tvl.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2023-12-20 18:35:58 +00:00
Aspen Smith
c236231910 chore(tvix): Run criterion benchmarks for the whole workspace
Run criterion benchmarks (via `cargo bench`) for the *whole tvix
workspace*, not just `//tvix/eval`, and report the results to
windtunnel.

Change-Id: I9235c3c166ed9121f35c5bb4c46d59dc1f4c4055
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10382
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-20 18:04:43 +00:00
Florian Klink
9c1358fe44 refactor(tvix/store): remove more clones on pathinfoservice
These are still leftovers from before the traits being async, where we
had to clone before moving into an async closure.

Change-Id: I1b3937edf61ce3e23bb07803306622c37a3572c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10381
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-12-18 14:10:40 +00:00
Ryan Lahfa
0ae32d45f6 feat(tvix/castore): simple filesystem blob service
The simple filesystem `BlobService` enable a user to write blob store
on an existing filesystem using a prefix-style layout in the provided root directory,
e.g. the two first bytes of the blake3 hashes are used as directories prefixes.

Change-Id: I3451a688a6f39027b9c6517d853b95a87adb3a52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10071
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-17 14:34:13 +00:00
Florian Klink
923a5737e6 refactor(tvix/castore): drop is_closed() from impl DirectoryPutter
This is only used in the gRPC version (GRPCPutter), during the test
automation.

So define it as a method there, behind #[cfg(test)], and remove from
the trait.

Change-Id: Idf170884e3a10be0e96c75d946d9c431171e5e88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10340
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 23:07:31 +00:00
Florian Klink
36cc7b7088 docs(tvix/store/fs): fix comment
There's nothing store-path specific here anymore, it's just a name in
the mountpoint root.

Change-Id: I0f8004491baa03ba560d390053a42678ee81154a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10377
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 21:53:31 +00:00
Florian Klink
7749814953 docs(tvix): refer to futures-rs PR
The PR has since been replaced with another version, using a bit of
unsafe, as rustc doesn't like the lifetimes.
We're still waiting on a backport of that to 0.3.

Change-Id: I8dd344e78162cd956f6e53ff05dc4c02763fbb04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10376
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-16 21:47:01 +00:00
Florian Klink
43ab1543e2 chore(tvix/store): switch back to fuse-backend-rs upstream
The fix mentioned in Cargo.toml did land in
https://github.com/cloud-hypervisor/fuse-backend-rs/pull/162.

Change-Id: I4451f910806f05400f7e0f83581ae786f170b9e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10375
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 21:46:30 +00:00
Florian Klink
97dee277b0 refactor(tvix/store/fs): no explicitly required Arc'ed Blob/DirSvc
Change-Id: Ie55026668cd4a6117e7b07174f5ac6638f93d194
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10374
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 21:46:30 +00:00
Connor Brewster
6815572274 refactor(tvix/store/fs): Decouple TvixStoreFs from Nix concepts
To support tvix builds, we need to be able to use the `TvixStoreFs` to
materialize the sandbox's /nix/store filesystem with just the list of
inputs needed for the build. Currently we'd need to seed an in-memory
`PathInfoService`, which includes more functionality than what is
required for `TvixStoreFs`. Additionally, the `PathInfoService` is
specific to Nix. By decoupling `TvixStoreFs` and `PathInfoService`,
we allow for usage of `TvixStoreFs` with `tvix-castore` without needing
a `PathInfoService`.

This introduces a new `RootNodes` trait which provides a way for the
filesystem to look up CA nodes via their basename in the root directory
of the filesystem. We then implement `RootNodes` for any
`PathInfoService`. Additionally, the filesystem root inode tracker now
stores basenames rather than `StorePath`s since `StorePath`s are
specific to Nix.

As a followup we can rename `TvixStoreFs` to `TvixCaStoreFs` and move
it to the `castore` crate (or its own crate).

b/341

Change-Id: I928372955017c23b1bf2b37190cbc508a4ed10d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10363
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-14 21:46:25 +00:00
Aspen Smith
91b6c13f90 fix(tvix): Remove errant backslash in benchmark script
Change-Id: I485986521fd4a25770f2a398896a0f672628df9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10365
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-14 19:52:31 +00:00
Aspen Smith
1d678bab5f fix(tvix): Fix tvix nixpkgs eval benchmark
Fix the nixpkgs eval hyperfine benchmark in tvix/scripts.bench.sh to:

- Eval both a pinned (never-changing) nixpkgs commit and the nixpkgs
  commit used by the depot
- Not depend on fetchTarball, which is not yet implemented.
- Be called bench-windtunnel.sh

Change-Id: Id2ae18f983ab7327625320b5b16c082ecc369a49
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10364
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2023-12-14 19:34:58 +00:00
Florian Klink
e4d6d20aa7 test(tvix/glue): add nonexisting_path_without_import test
This moves the failing example from cl/10285 into its separate test
case.

There were multiple complications: tvix-[ca]store was panicking in some
places, rather than returning an error. This is now fixed.

It needs to live in tvix-glue, so we actually have a "proper" EvalIO
interface doing something.

> toString ({ line = 42; col = 42; file = /deep/thought; }.file)

Should not cause an error, because it shouldn't trigger an import, but
leave the path as-is, and not care about it not being present.

Change-Id: I76f70b3cb1f73a0fb05870375710fd9f67d5603c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10342
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-14 13:25:25 +00:00
sterni
7165ebc43b fix(tvix/eval): remove incorrect imports when coercing
The default behavior of string coercion in C++ Nix is to weakly coerce
and import to store if necessary. There is a flag to make it strongly
coerce (coerceMore) and a flag that controls whether path values have
the corresponding file/directory imported into the store before
returning the (store) path as a string (copyToStore). We need to
implement our equivalent to the copyToStore (import_paths) flag for the
benefit of weak coercions that don't import into the store (dirOf,
baseNameOf, readFile, ...) and strong coercions that don't import into
the store (toString).

This makes coerce_to_string as well as CoercionKind weirder and more
versatile, but prevents us from reimplementing parts of the coercion
logic constantly as can be seen in the case of baseNameOf.

Note that it is not possible to test this properly in //tvix/eval tests
due to the lack of an appropriate EvalIO implementation being available.
Tests should be added to //tvix/glue down the line.

Change-Id: I8fb8ab99c7fe08e311d2ba1c36960746bf22f566
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10361
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-14 13:15:23 +00:00
sterni
a30dd0905a fix(tvix/eval): determine meaning of + exprs based on first type
This matches the behavior of C++ Nix more closely where the decision is
made based on the first type based to ExprConcatStrings:
1f93fa2ed2/src/libexpr/eval.cc (L1967-L2025)

Note that this doesn't make a difference in any successful
evaluation (at least to my knowledge), but ensures that our error
messages will match C++ Nix more closely, e.g. in the case of
`1 + "string"`.

Change-Id: I8059930788f9c8d98baf98e3d93d8a060ef961f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10360
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-14 02:59:58 +00:00
Florian Klink
8b0047e277 docs(tvix/castore/directorysvc): update comment
This comment didn't make a lot of sense before.

Change-Id: Ie057a133ca4b1a099ed3c885e32316b0d87c5eb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10339
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-12-13 21:11:58 +00:00
Florian Klink
3a32963b78 docs(tvix/castore): document expectations about DirectoryService
Namely, all trait implementations should reject invalid data being fed,
and detect invalid data being returned.

b/355 tracks writing some more tests for this, to ensure we're compliant
with this.

Change-Id: I3b05752932837ce208785efb21ffc21508b4b33a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10338
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2023-12-13 19:57:45 +00:00
Florian Klink
d236b08916 docs(tvix/castore): fix docstrings
There's been some copypasta errors.

Change-Id: I8fcad6cfc951ead6c789e0dce823c798adbfcf97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10337
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-13 19:40:40 +00:00
Florian Klink
9c5c717a99 feat(tvix/glue): add single-file import tests
This ensures importing these paths also behave the same way as Nix.

Change-Id: Icaa507bbe3d9867a301fc7a300c5d2b3f9feb911
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10355
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-13 13:42:21 +00:00
Florian Klink
3cdde8ad5a feat(tvix/glue): add some import tests
This creates a directory with a .keep file inside, and uses
"${path/to/there}" to coerce it to a string (and import it into the
store), ensuring it calculates the same store paths as Nix does.

Change-Id: Ie14ae075104ce278bc4f2cce93aab5762a2734d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10343
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-13 13:41:44 +00:00
Florian Klink
307675f5e7 fix(tvix/glue/tvix_store_io): don't unwrap ingest_path errors
Fixes b/344.

Change-Id: I1446726e3be3a8fc20801d466a964c4d6b8cbc02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10331
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:43 +00:00
Florian Klink
81ef26ba3f fix(tvix/castore/import): don't unwrap entry
If the path specified doesn't exist, construct a proper error instead
of panicking.

Part of b/344.

Change-Id: Id5c6a91248b0a387f3e8f138f8e686e402009e8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10330
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:11 +00:00
Florian Klink
afd09c3290 feat(tvix/castore/import): log returned errors
This will emit a log event / trace in case this function returns an
error-y type.

Change-Id: I48db6807f3e42304357c422a2b6e177cb8b95228
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10329
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-12 18:07:11 +00:00
Florian Klink
30d82efa77 refactor(tvix/castore/blobservice): use io::Result in trait
For all these calls, the caller has enough context about what it did, so
it should be fine to use io::Result here.

We pretty much only constructed crate::Error::StorageError before
anyways, so this conveys *more* information.

Change-Id: I5cabb3769c9c2314bab926d34dda748fda9d3ccc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10328
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-12 18:06:40 +00:00
Adam Joseph
91456c3520 refactor(tvix/eval): vm::add_values(): be less verbose
Change-Id: Icf328649fd70bdf0fc3ba6cd7754ae29735f11f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10035
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 18:00:37 +00:00
Adam Joseph
243a4b5699 fix(tvix/eval): substring: propagate catchables
Change-Id: Ia9b7858c817fbc9c95a3d1c2855b2445f7830e8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10326
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:51:29 +00:00
Adam Joseph
c5f58d4af3 fix(tvix/eval): baseNameOf: propagate catchables
Change-Id: Id8dc772ea8f338dfd243210f4108f79072570c3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10324
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:50:56 +00:00
Adam Joseph
2565a21aa9 fix(tvix/eval): builtins.length: propagate catchables
This commit fixes out builtins.length so it propagates catchables
like cppnix does.

Change-Id: I7670bec5eee1d4cd3f67a04c9a6808979fb56a8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10315
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-12 17:49:54 +00:00
Adam Joseph
1ddfed9825 fix(tvix/eval): builtins.filter: propagate catchables
This commit fixes builtins.filter so it propagates catchables
correctly.

Change-Id: Ib23a383bc5e272e42052205ffd1e94649a0ebc47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10313
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:35:39 +00:00
Adam Joseph
25dd1458b9 feat(tvix/eval): builtins.hashString: add placeholder
This adds an unimplemented placeholder for builtins.hashString.

Change-Id: Ibc770103acf5dbc3ea7589ab5ca23fe6e07bd91a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10311
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 17:35:06 +00:00
Adam Joseph
8c8409c0d2 fix(tvix/eval): builtins.getAttr: propagate catchables
Change-Id: I84b6b8f8568d57614a03aff0d6069e0bc27357bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10310
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 17:34:34 +00:00
Adam Joseph
afba150036 fix(tvix/eval): builtins.elemAt: propagate catchables
This commit fixes builtins.elemAt so it propagates catchables like
cppnix does.

Change-Id: Ieca5e128da17e78af0b14dae4a28a1ff8796e4f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10308
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:34:02 +00:00
Adam Joseph
07ca556a94 feat(tvix/glue): make unimplemented-structuredAttrs catchable
This commit adjusts the error produced by STRUCTURED_ATTRS so that
it is catchable.  This way we are able to enumerate the release
packageset, and the enumeration process will simply treat the few
derivations using structured attributes as being broken, rather than
killing the whole eval session.

Change-Id: I2e17638b8e3227f88543c3718aaf505deaec22ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10306
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 17:21:53 +00:00
Adam Joseph
29464fc65b fix(tvix/eval): propagate catchables through builtins.splitVersion
This fixes our implementation of builtins.splitVersion so it
propagates catchables like cppnix does.

Change-Id: Id5d83ea76229f8c8f202aa42353cb609e67de43f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10305
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:21:53 +00:00
Adam Joseph
012b44244b feat(tvix/eval): builtins.hasContext: placeholder implementation
Currently this just `throw`s a message explaining that it is not
implemented.  This is necessary in order to allow enumerating the
nixpkgs release attrset (afaict only one package uses this builtin).

Change-Id: I45266d46af579ddb5856b192b6be4b481369543c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10302
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:21:53 +00:00
Adam Joseph
531f0c0c42 fix(tvix/eval): baseNameOf should not coerce paths into strings
... since this may import them to the store which changes their
basename.

Fixes b/350.

Change-Id: Iabd08ff4d6a424c66d6d7784d7a96b0c078f0a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10298
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 17:21:21 +00:00
Adam Joseph
8d4aa2c15c fix(tvix/eval): add unimplemented __curPos and builtins.filterSource
This commit adds __curPos (to the global scope, yuck) and
builtins.filterSource.  These are not implemented; forcing them will
produce the same result as `throw "message"`.

Unfortunately these two post-2.3 features are used throughout
nixpkgs.  Since an unresolved indentifier is a catchable error, this
breaks the entire release eval.  With this commit, it simply causes
those broken packages that use these features to appear as they are:
broken.

Change-Id: Ib43dea571f6a9fab4d54869349f80ee4ec5424c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10297
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 17:20:50 +00:00
Adam Joseph
bb1e79e5d1 fix(tvix/eval): propagate catchables through &&
Change-Id: I7bb5ac1ef47b41c47269e64cee0e90eb64c6fcce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10322
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:04:07 +00:00
Adam Joseph
bcc1ea8552 fix(tvix/eval): make || propagate catchables
Change-Id: I42f994d7c9228368d5f6c30c4730c24666f7bc69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10320
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:04:07 +00:00
Adam Joseph
7b9eea71d0 fix(tvix/eval): fix nested assertions b/340
This commit fixes our handling of `throw` within an `assert`
condition.

Fixes: b/340
Change-Id: I40a383639ec266da50a853f16216b1b7868495da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10318
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 17:03:03 +00:00
Adam Joseph
fc963033ae fix(tvix/eval): ?: propagate catchables
This commit fixes out `?` operator so it correctly propagates
catchables.

Change-Id: Iebaa153a8492101ee3ddd29893c98730ff331547
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10317
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 16:57:55 +00:00
Adam Joseph
990c4e727f feat(tvix/eval): OpAttrsSelect should propagate catchables
Previously, using a catchable as either argument of OpAttrsSelect
would result in an unrecoverable error.  This commit matches cppnix
behavior by propagating the catchable.

Change-Id: I4877f4068ec2b823225f185290693c101d0b9c9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10303
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 16:09:47 +00:00
Adam Joseph
2949ee08f1 fix(tvix/eval): calling a catchable is catchable
When attempting to call a Value, if it is a Value::Catchable we must
not cause an uncatchable failure.  This commit simply reuses the
Value::Catchable as the result of attempting to call it.  This is
safe because nix is designed so that nix code cannot distinguish
between different catchable failures -- they all look the same to
the interpreted code.

This fixes b/351.

Change-Id: Ibf763a08753e541843626182ff59fdbf15ea2959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10300
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 16:09:15 +00:00
Adam Joseph
52b68c0539 fix(tvix/eval): fix catchables in named formals
Fixes b/348.

Change-Id: I5e8d56b5fd26a19eac32ec5e11baf93765691dc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10296
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 15:55:06 +00:00
Adam Joseph
e516046ed2 test(tvix/eval): test catchables in named formals
Relates to b/348.

$ /git/depot/result/bin/tvix -E '(builtins.tryEval (({ fred }: "bob") (throw "3"))).success'
note: while evaluating this Nix code
 --> [code]:1:1
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (force)
 --> [code]:1:1
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (tryEval)
 --> [code]:1:2
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  |  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
 --> [code]:1:20
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E006]: expected value of type 'set', but found a 'internal[catchable]'
 --> [code]:1:21
  |
1 | (builtins.tryEval (({ fred }: "bob") (throw "3"))).success
  |                     ^^^^^^^^

Change-Id: I730fdd996f7e1b81dbbf83dc1524104a8cad2f78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10295
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 15:54:04 +00:00
Adam Joseph
663362f3df fix(tvix/eval): fix testing catchables for inequality
Fixes b/347.

Change-Id: Icad0251884d4d8adcdf8d690b91385bf4896f9c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10294
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 15:53:32 +00:00
Adam Joseph
1b2a1892cb test(tvix/eval): testing catchable for inequality
Relates to b/347.

    $ /git/depot/result/bin/tvix -E '(builtins.tryEval (throw "bob" != 3)).success'
    note: while evaluating this Nix code
     --> [code]:1:1
      |
    1 | (builtins.tryEval (throw "bob" != 3)).success
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    note: while evaluating this as native code (force)
     --> [code]:1:1
      |
    1 | (builtins.tryEval (throw "bob" != 3)).success
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    note: while evaluating this as native code (tryEval)
     --> [code]:1:2
      |
    1 | (builtins.tryEval (throw "bob" != 3)).success
      |  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    error[E006]: expected value of type 'bool', but found a 'internal[catchable]'
     --> [code]:1:20
      |
    1 | (builtins.tryEval (throw "bob" != 3)).success
      |                    ^^^^^^^^^^^^^^^^

Change-Id: Ide19b3ddaf314ef310efc2fe5ac36667e43011dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10293
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-12 15:49:28 +00:00
Adam Joseph
289663cac4 fix(tvix/eval): handle catchables in attribute set updates
Fixes b/346.

Change-Id: I277121d2363e605ebe09651ed9440fe1bc126c8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10292
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 15:48:25 +00:00
Adam Joseph
e54533518b test(tvix/eval): test for catchable in attribute merges
Relates to b/346.

    $ /git/depot/result/bin/tvix -E '(builtins.tryEval (throw "bob" // { })).success'
    note: while evaluating this Nix code
     --> [code]:1:1
      |
    1 | (builtins.tryEval (throw "bob" // { })).success
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    note: while evaluating this as native code (force)
     --> [code]:1:1
      |
    1 | (builtins.tryEval (throw "bob" // { })).success
      | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    note: while evaluating this as native code (tryEval)
     --> [code]:1:2
      |
    1 | (builtins.tryEval (throw "bob" // { })).success
      |  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    error[E006]: expected value of type 'set', but found a 'internal[catchable]'
     --> [code]:1:20
      |
    1 | (builtins.tryEval (throw "bob" // { })).success
      |                    ^^^^^^^^^^^^^^^^^^

Change-Id: Ib84c4ec6d2612d4f1f6066e66c3dd1bf04369b6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10291
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 15:48:25 +00:00
Adam Joseph
edf411a86c fix(tvix/eval): fix recovering from throws in implications
This fixes b/345.

Change-Id: Ic0d3b6ffacd2a5e0050d22354d08320b69a4fe13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10290
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 15:47:53 +00:00
Adam Joseph
24ff74d346 test(tvix/eval): test recovering from throw in implications
error[E006]: expected value of type 'bool', but found a 'internal[catchable]'
 --> src/tests/tvix_tests/notyetpassing/eval-okay-test-catchables-in-implications.nix:1:43
  |
1 | (builtins.tryEval (({ foo ? throw "up" }: foo -> true) { })).success
  |                                           ^^^^^^^^^^^

Relates to b/345

Change-Id: Ic331c32ea59bf67ae775f485b444dc6804ca13d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10289
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 15:47:21 +00:00
Adam Joseph
9792920f8c fix(tvix/eval): fix branching on catchable defaults (b/343)
This commit adds Opcode::OpJumpIfCatchable, which can be inserted
ahead of most VM operations which expect a boolean on the stack, in
order to handle catchables in branching position properly.

Other than remembering to patch the jump, no other changes should be
required.

This commit also fixes b/343 by emitting this new opcode when
compiling if-then-else.  There are probably other places where we
need to do the same thing.

Change-Id: I48de3010014c0bbeba15d34fc0d4800e0bb5a1ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10288
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:55:48 +00:00
Adam Joseph
e54eeda0ff test(tvix/eval): test branching on catchable defaults (b/343)
This is a test case for b/343, wherein tvix dies if you try to
branch on an argument whose defaulted value is a catchable.

Change-Id: I891ca825e39ad14dda9f220f06d9591874fcd45d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10287
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-12 14:55:48 +00:00
Adam Joseph
1ac57b0d1c feat(tvix/eval): nonrecursive coerce_to_string()
After this commit, the only non-builtins uses of generators are:

  - coerce_to_string() uses generators::request_enter_lambda()
  - Thunk::force() uses generators::request_enter_lambda()

That's it!  Once those two are taken care of, GenCo can become an
implementation detail of `builtins::BuiltinGen`.  No more crazy
nonlocal flow control within the interpreter: if you've got a GenCo
floating around in your code it's because you're writing a builtin,
which isn't part of the core interpreter.  The interpreter won't
need GenCos to talk to itself anymore.

Technically generators::request_path_import() is also used by
coerce_to_string(), but that's just because the io_handle happens to
be part of the VM.  There's no recursion-depth issue there, so the
call doesn't need to go through the generator mechanism
(request_path_import() doesn't call back to the interpreter!)

Change-Id: I83ce5774d49b88fdafdd61160975b4937a435bb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10256
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:54:46 +00:00
Adam Joseph
3701379745 feat(tvix/eval): nonrecursive deep_force()
This commit implements deep_force() nonrecursively, by maintaining
an explicit stack rather than using the call stack for recursion.

As an added bonus, we don't need to pass around the SharedThunkSet
anymore, and can in fact completely eliminate SharedThunkSet.

Change-Id: I7c4f59f37834d451a28bf6be317eb0a90eac4ee6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10252
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:36:02 +00:00
Adam Joseph
c956138ee6 docs(tvix/eval): clarify difference between ThunkSet and Blackhole
The comment explaining ThunkSet makes it seem like it does the same
think as ThunkRepr::Blackhole.  In fact neither one is a substitute
for the other.  Let's explain the difference.

Change-Id: I89ceaaa9d3c499edbc7d48f70ca5d11f97666c43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10250
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:35:00 +00:00
Adam Joseph
ddba4d4e17 test(tvix/eval): nested assertions (b/340)
Change-Id: I898d7056877a6370d5720b633df416f54e7cf65f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10222
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 14:35:00 +00:00
Adam Joseph
c92d06271d feat(tvix/eval): drop LightSpan::Delayed
LightSpan::Delayed was introduced in commit
bf286a54bc which claimed that "This
reduces the eval time for `builtins.length (builtins.attrNames
(import <nixpkgs> {}))` by *one third*!"

I am unable to reproduce this result.  In fact, dropping the
LightSpan::Delayed variant of the enum makes eval of the same
expression slightly faster!  I also tried a large evaluation
(pkgsCross...hello) and got similar results: slightly faster,
slightly less memory.  See git footers.

I suspect that there was some unrelated horrific inefficiency that
has since been fixed.  The avoided computation in `get_span()` is
nothing more than a binary search!  If this were in fact a major
performance issue we could simply precompute the mapping from
CodeIdx to Span when the Chunk becomes immutable (i.e. at the end of
the compilation process, when compiler backtracking is no longer a
concern).  Since a Span is just 64 bits this is not a space issue,
and since binary search is much simpler than compiling Nix
expressions it isn't a performance issue either.

Technically there is no longer any reason to have LightSpan since it
is now a single-variant enum.  However there is no rush to remove
it, since Rust will optimize its representation into the same thing
you'd get if you replaced LightSpan by Span.

Prev-Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
This-Benchmark: {"nixpkgs-attrnames":{"kbytes":"230192","system":"0.29","user":"2.00"}}
Prev-Benchmark: {"pkgsCross.aarch64-multiplatform.hello.outPath":{"kbytes":"458936","system":"0.73","user":"5.36"}}
This-Benchmark: {"pkgsCross.aarch64-multiplatform.hello.outPath":{"kbytes":"451808","system":"0.53","user":"5.10"}}

Change-Id: Ib9e04806850aa1fc4e66e2a042703986440a7b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10254
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-12 14:34:28 +00:00
Adam Joseph
29878259b3 feat(tvix/cli): add benchmark for bf286a54bc
cl/7558 used this expression as a benchmark to justify the introduction
of LightSpan::Delayed:

  builtins.length (builtins.attrNames (import ${pkgs.path} {}))

Let's add it as a benchmark case so it can be referenced easily.

Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
Change-Id: Idb6c69ddd284605dd3b5fd9ac5c79a69b9a470b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10253
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:28:53 +00:00
Florian Klink
27c07b72c6 refactor(tvix): use io::Result for EvalIO
This is just a alias for Result<_, io::Error>, but shorter.

Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 14:28:50 +00:00
Adam Joseph
ad566999ca fix(tvix/eval): preserve catchables in nix_cmp_ordering(), fix b/338
This commit fixes b/338 by properly propagating catchables through
comparison operations.

Change-Id: I6b0283a40f228ecf9a6398d24c060bdacb1077cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10221
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
ae28dc3ca6 test(tvix/eval): test for b/338 catchable hygiene problem
Commit 05f42519b5 fixed b/281 by
establishing a hygiene regimen to partition *catchable* errors
(i.e. those which tryEval can detect) from all other errors, like
internal VM failures or I/O errors (which Nix must not be allowed to
detect, since these errors are fundamentally impure).

Unfotunately there are still cases where tvix assumes that anything
other than Value::Bool means it should panic!().  I found another
one, and added a test case for it in:

  eval_okay_src_tests_tvix_tests_eval_okay_compare_ordering_catchable_nix

Not yet passing.

Change-Id: I69c62ed9ea5c8f81870e8de5c5fe12dcde849763
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10220
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
edbd5055a1 feat(tvix/eval): nonrecursive nix_cmp_ordering(), fixes b/339
This commit rewrites Value::nix_cmp_ordering() into an equivalent
nonrecursive form.  Except for calls to Thunk::force(), the new form
no longer uses generators, and is async only because of the fact
that it calls Thunk::force().

I originally believed that this commit would make evaluation faster.
In fact it is slightly slower.  I believe this is due to the added
vec![] allocation.  I am investigating.

Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460224","system-seconds":"0.67","user-seconds":"5.84"}
Change-Id: Ic627bc220d9c5aa3c5e68b9b8bf199837cd55af5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10212
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
8a40f75c2d fix(tvix/eval): never use partial_cmp() (partial fix b/338)
This is part of a fix for b/338.

We should never use PartialOrd::partial_cmp().

All Nix types except floats are obviously totally-ordered.  In
addition, it turns out that because Nix treats division by zero
rather than producing a NaN, and because it does not support
"negative zero", even floats are in fact totally ordered in Nix.

Therefore, every call to PartialOrd::partial_cmp() in tvix is an
error.  We have to *implement* this function, but we should never
call it on built-in types.

Moreover, nix_cmp_ordering() currently returns an Option<Ordering>.
I'm not sure what was going on there, since it's impossible for it
to return None.  This commit fixes it to return simply Ordering
rather than Option<Ordering>.

Change-Id: If5c084164cf19cfb38c5a15554c0422faa5f895d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10218
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
19d13eb070 test(tvix/eval): add test case for b/339
Not yet passing.

Change-Id: I1de3f72d8b3f46567fdba010fc3ab4bace3f1699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10219
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-12-12 14:26:46 +00:00
Adam Joseph
72ece2e518 feat(tvix/eval): nonrecursive nix_eq()
This commit rewrites Value::nix_eq() into an equivalent.  Except for
calls to Thunk::force(), the new form no longer uses generators, and
is async only because of the fact that it calls Thunk::force().

I believed that the nonrecursive form would be faster.  It is, in
fact, slightly slower.  I believe this is due to the vec![]
allocation; I am investigating.

Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"459068","system-seconds":"0.71","user-seconds":"5.39"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
Change-Id: I10f4868891e4b7475df13f0cbc41ec78dd985dd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10118
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-12 14:26:46 +00:00
Adam Joseph
0c15a09b15 feat(tvix/cli): add macrobenchmark
This commit adds a simple MVP benchmark, built on our nix
infrastructure instead of cargo.  It simply runs `tvix-eval` inside
of GNU time, and prints the three essential statistics in a short
JSON blob.

You can run the benchmark with a simple `nix run`, like:

  nix run -f . tvix.cli.benchmark-hello
  nix run -f . tvix.cli.benchmark-firefox
  nix run -f . tvix.cli.benchmark-cross-firefox

Currently these blobs are stored only in the CI logs, which I'm sure
get garbage-collected at some point.  We should be putting them in
the git trailers, but that can wait for a future CL.

I tried using `cargo bench` for this but found it incredibly
frustrating.  Maybe I'm doing it wrong.  It seems to be designed for
microbenchmarks only, and very hard to control.  It kept building
all sorts of unnecessary stuff (like the tests), and unlike
crate2nix it was doing all the builds on only a single machine
instead of using more than one machine.  Worse, for that single
machine it kept picking my laptop instead of my fast servers!  It
seems excessively cargo-flavored for such a straightforward task.

Benchmark: {"hello.outPath":{"kbytes":"244736","system":"0.36","user":"2.76"}}
Benchmark: {"firefox.outPath":{"kbytes":"1506736","system":"2.38","user":"32.01"}}
Benchmark: {"pkgsCross.aarch64-multiplatform.firefox.outPath":{"kbytes":"11334548","system":"10.70","user":"107.07"}}

Change-Id: I85bc046ec551360284d7ecfc81a03914f0085909
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10216
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-12 11:56:58 +00:00
Florian Klink
8018313b68 feat(tvix/glue): add nixpkgs eval benchmark
This adds a criterion.rs-based testbench into tvix-glue.

It can be invoked by running `cargo bench` from inside the `tvix-glue`
crate.
`target/criterion/report/index.html` contains nice graphs.
It's able to diff against the previous run, so you can invoke `cargo
bench` before and after a certain change to reason about the impact in
evaluation performance.

Currently, we need to create a bunch of Evaluator resources inside the
benchmark loop itself, which is a bit annoying, as it leaks into the
things we benchmark.

This should become better with b/262.

Fixes b/322.

Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 10:27:51 +00:00
Florian Klink
6c87ab960d chore(tvix/glue): allow unused_variables
cl/9364 did introduce a warning here, which is visible when building in
release mode - or invoking `cargo bench` in tvix-glue.

Change-Id: Ia82082a58543f0fdd32866fdfcd37d0a5fdfda9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10261
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 09:01:51 +00:00
Florian Klink
ccaf10b4a6 refactor(tvix/*store/sled): make ::new() more generic
We don't really require the Path to be a PathBuf, we don't even require
it to be a Path, we only need it to be AsRef<Path>>.

This removes some conversion in the from_addr cases, which can just
reuse `url.path()` (a `&str`).

Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 22:46:50 +00:00
Florian Klink
9748543f1c chore(tvix): drop sled compression support
It's been a while since the last sled release, and that one binds to a
pretty old version of zstd, requiring workarounds like cl/10090.

Upstream sled main branch currently has zstd halfway patched out (it's
a no-op, but the feature flag and options are still there), and it's in
that state for a year.

Rather than maintaining our own fork of sled, let's just stop using the
compression feature in sled, dropping the version pin to zstd that way,
removing the need for cl/10090.

This doesn't mean we won't reintroduce per-blob compression - but we
probably just won't let sled take care of the compression, but do it
ourselves - which is necessary for more chunked blob storage anyways.

Even though we do drop the feature flag, we still need to explicitly use
use_compression(false).

Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 22:46:50 +00:00
Florian Klink
6025be423a chore(tvix/proto): drop evaluator.proto
This only contained the (unused) evaluator.proto file.

Considering we're less likely to have the CLI talk to a long-running
evaluator, but instead embed the evaluator inside the CLI, remove this.

If we add a RPC to speak to an evaluator, we can resurrect this from
git history.

Change-Id: I2196aade55221660330dfd32dc3e52c39ec6ed43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10241
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
1cc3e9c1a8 refactor(tvix): use granular proto files
Only pass in the proto files that are actually needed to build that
crate. They are already constructed in depot.tvix.$crate.protos.protos.

Change-Id: If4381e6c3350e420ee4ddce1e0513bfe970678a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10240
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
43943e2623 refactor(tvix/*/protos): separate lint target
Break up the go-bindings derivation. Keep "protos" containing all proto
files (well, and the buf config), and use it for a check phase running
linter and formatter, as well as the existing "go-bindings" attribute

Change-Id: I52cb9d08570bb76452acb831eb711c5b6c0eacfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10239
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
8486f87e3c feat(tvix/build): add derivation_to_build_request
This function converts from a nix_compat::derivation::Derivation to
a BuildRequest.

In addition to the Derivation itself, it needs two lookup functions to
map input paths to their castore nodes.

Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 22:35:39 +00:00
Florian Klink
92bd69aef2 docs(tvix/*store-go): fix README invocation
The command is called `regenerate`, not `generate`.

Change-Id: I18075042ebd461e4dd0718a936e6bbe738a144d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10259
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-11 21:47:43 +00:00
Florian Klink
a1b2dc8aaf feat(tvix/build-go): init
This adds the generated golang bindings for tvix-build.

Change-Id: I2eb0d1cc38bc2fa34afd7c904eea05c5ee192cce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10242
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 21:43:38 +00:00
Florian Klink
fd27d8ddc3 feat(tvix/build): init
This adds the tvix-build crate, currently only containing a
`tvix_build::proto` module, exposing the data structures defined in
tvix/build/protos.

Change-Id: I75f5d9196969ed0877b1fe640cacfecba0fb2e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10235
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 21:43:38 +00:00
Florian Klink
3c0a9a949a chore(tvix/store/protof): buf format
Change-Id: I29b0fb62aa849ee01860dc7fe3dbba9b941ca172
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10238
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 10:29:50 +00:00
Florian Klink
459d9e106f chore(tvix/castore/protof): buf format
Change-Id: Idf11de78b0d6eca69fda34a89f2c57a00ed89ad5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10237
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-11 10:24:44 +00:00
Aspen Smith
d9ac36d307 fix(tvix): Fix cwd in benchmark script
Change-Id: I6d9435c6710ce086aaedb2c1dc634b73a19f476c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10249
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-11 04:13:41 +00:00
Florian Klink
01a3c87a3a chore(tvix): bump criterion to 0.5
Change-Id: I28904ca23437b4bb745c0eb1f4eb9ae33e09eb5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10244
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 03:59:30 +00:00
Florian Klink
b14053ae03 docs(tvix): fix crate2nix-generate invocation
This got renamed recently, but we forgot to update it here.

Change-Id: I7d713c8a0e6ccca57fe67985d9cb4e7f1eeef3b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10243
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-12-11 03:59:28 +00:00
Aspen Smith
b2c8c1ef94 feat(tvix): Benchmark nixpkgs eval with hyperfine
Add some hyperfine benchmarks to Tvix's windtunnel benchmark script for
evaluating the outPath of hello and a cross-compiled hello.

Change-Id: I9d76e5ce0a3fd7d9c125c36c5fced675b660a8a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10248
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
2023-12-11 03:59:27 +00:00
Adam Joseph
2b4920c355 fix(tvix/cli): panic on root cause of the fetchurl(non-boot) bug
Currently we produce wrong drvPaths for a large number of packages
that use fetchurl (but not fetchurlBoot, which is what stdenv uses).
A simple reproducer is `pkgs.perl538`.

I debugged this down to the root cause, which is the fact that tvix
doesn't realize that the mapping from FOD-paths to outputHash is
*NOT* a 1:1 mapping.  It is a many-to-one mapping.  You can have
lots of different FODs with the same outputHash or even the same
outPath.  For example, perl538.src and perldevel.src use the same
source tarball but a different `version`.

Anyways, I have found the root cause but have run out of time for a
while, so I've added a panic!() to in the spot where we have a logic
bug in order to call it out.

Change-Id: I9766b39cfe2fe7eafec84945b2ad6cc28f9c4b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9364
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-10 14:16:21 +00:00
Florian Klink
efaff79584 feat(tvix): make clippy stricter
Also run clippy on tests, and enable all features.

Change-Id: Ide9f1bc9f565333072afb918c391c7930b658f41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10234
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
22a669d27e refactor(tvix/castore): address clippy
We match to destructure a single pattern.

Change-Id: I564a3510b4860e90b3315a9639effc48ee88b483
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10233
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
Florian Klink
340242174d refactor(nix-compat/aterm): address clippy
This pushes to a Vec immediately after creation.

Change-Id: I2360b45810475d98ededc1d097fb4cbdeabc576b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10232
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-12-09 12:58:39 +00:00
Florian Klink
f9ff4fe969 refactor(store/fs): address clippy
Change-Id: I28d60263ef672942940656caa8e0f9d20cc6b7d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10231
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
f4c4e32040 refactor(nix-compat/nixbase32): address clippy
Change-Id: I07b53ead10b0efb31e88c6ae93c124c5f35261d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10230
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
06350445e9 refactor(nix-compat/narinfo): address clippy lints
Change-Id: I00f28624ec0469525e52f408f00caa20d3b701b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10229
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
1bdb90e56d refactor(nix-compat/derivation): address clippy
Change-Id: I5d7057347838f733d7dc1331c7b7ef8d27244412
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10228
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-12-09 12:58:39 +00:00
Florian Klink
d3ecef1a5b refactor(tvix/eval): address clippy lints
Change-Id: Ic2bd4e8291b30ceac9fa0e88a4f56e61ae99b603
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10227
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:39 +00:00
Florian Klink
f11c6d610c chore(tvix/nix_cli): drop crate
The only thing this was doing was invoking nix-store --add, which is not
gonna help us populate the tvix-store at all (and we now have
`tvix-store import`).

This is also (rightfully) causing clippy warnings, because of some
fields being unused.

It's more of a skeleton, and rather than shelling out to Nix for some
usecases, we might introduce a "compatible" Nix CLI frontend for a
subset of commands.

Drop this for now, to decrease the noise and confusion.

Change-Id: I2fd399e9320260f08893b685561755af9c7c961c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10226
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Florian Klink
0907420600 refactor(nix-compat/store_path): from_absolute_path to StorePathRef
The only non-test usage was only checking for the error case, and we can
still convert this to an owned StorePath by calling to_owned() on
StorePathRef.

Change-Id: I9f67a759e580c9c429c96896bcdd295392aa5a2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10225
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:58:35 +00:00
Florian Klink
5855959d87 refactor(nix-compat/derivation): rename input derivation components
Match the naming in parse_input_derivations, call the keys
"input_derivation", and the values "output_names".

Change-Id: I7d1974819028f8ea543dc3ad78afb803ff9db865
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10224
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 12:20:51 +00:00
Adam Joseph
5fd42a9e70 feat(tvix/eval): impl DoubleEndedIter for OwnedAttrsIterator
Change-Id: I4bd85dbe9c27047f4abbdeff4e2b796e9bcab3a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10211
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-12-09 11:35:19 +00:00
Adam Joseph
36c3e646e1 chore(store/src/tests): make clippy shut up
Change-Id: I6dfceaa32a8c01f8395b7889ae19847fabf95ed3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10214
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 10:51:10 +00:00
Adam Joseph
dfca879535 fix(tvix/glue): do not panic if PathInfoService returns Err
Nixpkgs tries to `import` the value returned by
`builtins.unsafeGetAttrPos`, which in our case is the file
`/deep/thought`.  Since that doesn't exist, tvix-glue panics, but
there's no interpreter backtrace to follow.

Let's return an Err instead of panicking.

------------------------------------------------------------------------------
Before:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at src/tvix_store_io.rs:276:58:
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(41580), ...)
Command exited with non-zero status 101

------------------------------------------------------------------------------
After:

thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27:
called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
note: while evaluating this Nix code
   --> [code]:1:1
    |
1   | (import /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src {}).pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:6:78
    |
6   |   runCommand "rocm-llvm-binutils-${llvm.version}" { preferLocalBuild = true; } ''
    |  ______________________________________________________________________________^
7   | |   mkdir -p $out/bin
8   | |
9   | |   for prog in ${lld}/bin/*; do
...   |
27  | |   ln -s ${lld}/bin/lld $out/bin/ld
28  | | ''
    | |__^
note: while evaluating this as native code (coerce_to_string)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:27:9
    |
27  |   ln -s ${lld}/bin/lld $out/bin/ld
    |         ^^^^^^
note: while evaluating this as native code (getAttr)
   --> <src-builtins/derivation.nix>:26:19
    |
26  |         outPath = builtins.getAttr outputName strict;
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (derivationStrict)
   --> <src-builtins/derivation.nix>:14:12
    |
14  |   strict = derivationStrict drvAttrs;
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:23
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:30
    |
249 |             outPath = assert condition; drv.${outputName}.outPath;
    |                              ^^^^^^^^^
note: while evaluating this as native code (throw)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:262:8
    |
262 |     in handler msg;
    |        ^^^^^^^^^^^
note: while evaluating this Nix code
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |________________________________________________________^
note: while evaluating this as native code (force)
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14
    |
254 |           else ''
    |  ______________^
255 | |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
256 | |
257 | |         '' + (builtins.getAttr reason remediation) attrs;
    | |__________^
error[E029]: I/O error: /deep/thought:42: task panicked
   --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:255:41
    |
255 |           Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate.
    |                                         ^^^^^^^^^^^^^^^

Command exited with non-zero status 1
Benchmark: {"pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath":{"kbytes":"26613180","system":"22.35","user":"140.62"}}

Change-Id: I587b57e9e49d1f3ecdc0fc9cf996d179a3548f34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10223
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-09 10:47:39 +00:00
Adam Joseph
f4460bd612 chore(nix_cli): make clippy shut up
Change-Id: I0c243baf4deecfd81bd939da86d685e2cbc59d69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10215
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-09 09:25:24 +00:00
Adam Joseph
2132bbcb47 fix(tvix/nar-bridge): fix FOD hash
The golang mothership seems to be monkeying with hashes again.

Change-Id: I7430b4cde84fa51be2b572fba02e3567864bb87a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10209
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 09:24:53 +00:00
Adam Joseph
0e2633048e feat(tvix): run crate2nix generate in CI
This runs `crate2nix generate` in CI and then runs `depotfmt` on the
result to ensure that our machine-generated code is really, really
readable and pretty.  Then it checks that the result of all that
is identical to the committed Cargo.nix.

A self-hashing FOD is used to allow network access.

No magic hashes are involved.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I68ec5003dbc6a40894a5a4d6e902f138c99f6719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10194
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-07 18:43:46 +00:00
Aspen Smith
0b524890e9 refactor(tvix): Make benchmark script directory-agnostic
Allow running the benchmark script from any directory - primarily so
Windtunnel can point to the /tvix josh workspace rather than the depot
overall

Change-Id: Ie5fc3ef995bf8114277298ae5c5010e6a0bf13ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10205
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2023-12-06 16:19:46 +00:00
Adam Joseph
49b34183e3 feat(tvix/eval): rewrite Thunk::force() in nonrecursive form
This commit rewrites Thunk::force() so that it is not (directly)
self-recursive.  It maintains a Vec of all the
previously-encountered thunks which point to the one it is currently
forcing, rather than recursively calling itself.

Benefits:

- Short term:

  This commit saves the cost of a round-trip through the generator
  machinery for the generators::request_force() which is removed by
  this commit.

- Medium term:

  Once a similar transformation has been applied to nix_cmp(),
  nix_add(), nix_eq(), and coerce_to_string(), those four functions,
  along with Thunk::force(), will make non-tail calls only to each
  other.  They can then be merged into a single tail-recursive
  function which does not use the generator machinery at all:

    enum Task { Cmp, Add, Eq, CoerceToString, Force};

    fn Value::walk(task:Task, v1:Value, v2:Value) {
      // ...

- Long term:

  The long-term goal here is to use generators **only for builtins**
  and [Marionette]-style remote control of the VM.  In other words:
  use `async` for things that actually involve concurrency.  Calls
  from the VM to builtins can then be blocking calls, because even
  cppnix will overflow the stack if you make a MAX_STACK_DEPTH-deep
  recursive call which passes through a builtin at every stack frame
  (e.g. `{ func = builtins.sort (a: b: ... func ...) ...}`).

  This way the inner "tight loop" of the interpreter doesn't pay the
  costs of `async` and generators.  These costs manifest in terms
  of: performance, complex nonlocal control flow, and language
  impediments (async Rust is a restricted subset of real Rust, and
  is missing things like traits).

[Marionette]: https://firefox-source-docs.mozilla.org/testing/marionette/Intro.html

Change-Id: I6179b8abb2ea0492180fcb347f37595a14665777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10039
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-12-06 06:53:01 +00:00
Aspen Smith
8135a8d38c fix(tvix/eval): Return error rather than panicking on bad substring
If builtins.substring is invoked with (byte!!) offsets that aren't at
codepoint boundaries, return an error rather than panicking. This is
still incorrect (see b/337) but pushes the incorrectness forward a step.

Change-Id: I5a4261f2ff250874cd36489ef598dcf886669d04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10199
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-05 23:12:23 +00:00
Florian Klink
41235bf908 refactor(tvix): move src into let binding
Change-Id: Ida2a3ac722fb2445745759323975884dfeef3e87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10193
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-05 21:55:09 +00:00
Aspen Smith
6edcb03cb6 feat(tvix): Add script for running benchmarks in Windtunnel
Currently this just uses a Docker container, which is gross but works
fine for now since we don't have the ability to build benchmarks in nix
as of cl/7538

Change-Id: I48e317f44bc2c73533d7663403786a3a37c7952f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10189
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: grfn <grfn@gws.fyi>
2023-12-05 17:46:26 +00:00
Florian Klink
dfb48dcade fix(tvix/nar-bridge): drop pathinfoservice
This now exists in tvix-store directly, as NixHTTPPathInfoService, and
contrary to this version, also validates signatures.

Change-Id: Ib6ca161e40d627b7d9741839fc849f2392f422da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10155
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-28 16:08:59 +00:00
Florian Klink
563886c3de feat(tvix/store/pathinfosvc/nix_http): allow configuring pubkeys
This allows setitng the trusted-public-keys URL parameter to a
(whitespace-separated) list of public keys.

NARInfo files retrieved need to contain a valid signature.

Change-Id: Ifd6580b723cbae3182e9cadfa54f1ca2b41d6599
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10153
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
4297e33d94 feat(tvix/store/pathinfosvc): add signature verification
Introduce an Option<Vec<narinfo::PubKey>>, configurable with a
`set_public_keys` method.

If set, this configures NixHTTPPathInfoService to validate signatures.

Change-Id: I157c5e13c41fc9bfd40b0655381fb4cf33900868
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10152
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-28 16:08:59 +00:00
Florian Klink
5fc737b02e feat(nix-compat/narinfo): add PubKey::verify(fp, sig)
This makes it easy for each PubKey to check if a given Signature is
correct for a given fingerprint.

Change-Id: I56e6211d133f74f390fd1ae3ae799eef12221904
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10151
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
6af67af76e feat(nix-compat/narinfo): add PubKey
This represents a ed25519 public key and "name".
These are normally passed in the `trusted-public-keys` Nix config option,
and consist of a name and base64-encoded ed25519 pubkey, separated by a `:`.

Change-Id: I9ab4b3e0e5821805ea6faf2499626630fc5a3f0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10150
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
e7a86273b1 refactor(nix-compat/narinfo/signature): rename Error type
Just call this Error, we can infer from the package what error this is.

Change-Id: I5df25d2873ec739c49c08804f35562c84c222e06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10149
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
7e8719be91 feat(tvix/store/pathinfosvc/nix_http): check Nar{Size,Hash} matches
Ensure the initially communicated NarHash/NarSize from the NarInfo
matches what we read, and don't return a PathInfo message if there's
a mismatch.

Also move the buffering layer around a bit.

Change-Id: I68c60ecfaf0f9cd5edacea648437ecb0c9729251
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10148
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-28 16:08:59 +00:00
Florian Klink
fce9c0e99e fix(tvix/store/fs): don't panic on PathInfoService error
An error in the PathInfoService request can appear in case the
underlying request returns an error.

We shouldn't panic and bork the fuse mount, but instead return an IO
error.

Change-Id: I2daeae629e1627d06adcd7b82ddb76c50c602212
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10154
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-28 14:54:56 +00:00
Vincent Ambo
360c98ce38 chore(3p/sources): bump channels & overlays (2023-11-27)
* picked avrdude from stable channel
* removed override for texlive, as the upstream fix is merged
* picked awscli2 from stable channel
* bump tdlib to 1.8.21 (new minimum for telega.el)
* tvix/turbofetch: switch to nixpkgs-native mechanism for
  CARGO_MANIFEST_LINKS (whatever that is)

Change-Id: Ic695721b5ca750b89d21cab7a257e1db682b23c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10083
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-27 21:16:46 +00:00
Florian Klink
0415bc6fd2 fix(nix-compat/narinfo/signature): validate name field
We should restrict this to alphanumeric mostly, and we definitely don't
want newlines.

Not entirely sure about the exact additionally allowed characters
outside of alphanumeric, but this can always be extended further.

Change-Id: I1357e79e553f2df2fa97792889f63f0f35d50ed5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10147
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-27 15:12:13 +00:00
Florian Klink
b7de931cc6 fix(nix-compat/narinfo): don't panic trying to parse signatures
BASE64.decode_mut panics if we're passing data that has the wrong size.
Do the size check first and error out there.

Also update the error, and talk about b64-encoded sizes.

Change-Id: I290f80a37d48526a30bf1df9d1d9fe34865008eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10146
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-11-27 15:12:13 +00:00
Florian Klink
dfaaf41cef refactor(nix-compat): use ed25519_dalek::SIGNATURE_LENGTH
No need to hardcode magic numbers here, we have a constant for that.

Change-Id: I67b671c0c4bb7c3bfb001e9c36499f31873ee717
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10145
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-27 15:11:41 +00:00
Vincent Ambo
e65fa82d74 refactor(tvix/eval): use or_default helper in entry API
This fixes a future clippy lint.

Change-Id: Ic830e94ef23595580c1037f10878c76bbb546dd9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10110
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-11-25 15:48:46 +00:00
Vincent Ambo
5ffb997864 fix(tvix): ensure PartialOrd/Ord agree for StorePath & NixString
This fixes a *future* clippy lint:
https://rust-lang.github.io/rust-clippy/master/index.html#/incorrect_partial_ord_impl_on_ord_type

In essence, because the implementation of *both* Ord and PartialOrd
implies that ordering is not partial, all results of PartialOrd should
simply be those of Ord. This is to avoid subtle bugs in future
refactorings.

Change-Id: I8fc6694010208752dd47746a2aaaeca0c788d574
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10109
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-25 15:48:46 +00:00
Adam Joseph
512346ba0b refactor(tvix/eval): add ThunkRepr::is_forced()
Change-Id: I4eab5c81fb82337da06327248845cd2f3a4490d3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10038
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-11-25 02:55:57 +00:00
Adam Joseph
b5a15989cd feat(tvix/eval): add Thunk::unwrap_or_clone()
This commit adds Thunk::unwrap_or_clone(), which uses
Rc::try_unwrap() to avoid cloning the Value out of a an Rc which has
only one strong reference.

Change-Id: Icacefe0c823dcddf046d90c0c5cd5ed59fe976d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10037
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2023-11-25 02:55:56 +00:00
sterni
875bb26fc3 fix(tvix/castore): correctly flag unreachable code
Change-Id: Id09afa4b77c3c70fb5695f253f6df4aa88b61e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10113
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-24 23:36:15 +00:00
sterni
c5961e7774 docs(tvix/eval): optimization potential for inherit (from) exprs
Change-Id: Ibddaa111a5b7a86c42dbe153ae8e53f9a5601a54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10112
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-11-24 23:36:15 +00:00
Florian Klink
6b685ec4a5 feat(tvix/store): add as_narinfo() for PathInfo
This allows seeing a PathInfo as a nix_compat::narinfo::NarInfo<'_>.

It doesn't allocate any new data, but the NarInfo<'_> view allows us to
access things like signature verification, or rendering out
(alternations of this) as strings.

Change-Id: Id0d8d7feeb626ee02c3d8a4932f24ace77022619
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10108
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
639cca3e22 feat(nix-compat/narinfo/signature): add new() constructor
This is useful when creating a new Signature struct where the individual
elements are already parsed.

Change-Id: Ie33c66287641951e7a030aaa1e7ff0a86b2628ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10111
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
c44cbc852a refactor(tvix/store): impl From<&nar_info::Ca> for nixhash::CAHash
Change-Id: I637a4cff5a5ca29c4d86e0b76a2f20f8741f5628
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10107
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:37:10 +00:00
Florian Klink
1e04f60800 refactor(tvix/store): impl From<nixhash::CAHash> for nar_info::Ca
Change-Id: Iaa68044d3b469f15a932aa3b59548505eaa6b8bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10106
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:11:49 +00:00
Florian Klink
671206a63f refactor(tvix/nix-compat): move from_name_and_digest to StorePathRef
We can simply use .to_owned() on that thing afterwards if we want to
construct an owned StorePath.

Change-Id: I0f3e2e4434b99ee522f2a7dbfa391e13a987479c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10105
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-22 20:11:49 +00:00
Florian Klink
eb84898c17 feat(nix-compat/narinfo): drop .drv from Narinfo.deriver field
We always know this needs to end with a .drv, and fail parsing if it
doesn't, so there's no need to hang onto these 4 bytes.

This will make it much easier to synthesize a NarInfo<'_> later on from
a PathInfo proto, because we don't have to make this ".drv" appear out
of thin air.

Change-Id: Id95e7fd937d7c9a420a39b5a4bab73985640ca3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10084
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-11-22 18:24:04 +00:00
Florian Klink
ef8a8af0bf refactor(tvix/nix-compat): cleanup parse_{ca,hash} and fmt structs
These were used to format to and parse from strings.

Move this to the CAHash and NixHash structs directly, and be explicit in
the name about which encoding for digests is used.

For output path calculation, nix encodes the nixpaths in hex, but for
writing out NARInfos, it's using nixbase32.

Change-Id: Ia585a76a3811b2609e7ce259fda66a29403b7e07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10079
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-11-22 17:54:59 +00:00
Florian Klink
a834966efd feat(tvix/nix-compat/narinfo): add fingerprint
This adds support to compute the fingerprint string, which is what's
ed25519-signed in binary caches.

Change-Id: I8947239c609896acfd7261f110450014bedf465a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10080
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 22:34:59 +00:00
Florian Klink
bb18556bf3 feat(tvix/nix-compat): add narinfo::Signature::verify
This adds support for verifying signatures found in NARInfo files.

This still needs to be hooked together with the nix+http[s] backend.

Change-Id: Ic1c8ddbdecfb05cefca2492808388b0f7f3f2637
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10081
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-19 22:06:07 +00:00
Florian Klink
a5749fada5 refactor(nix-compat/narinfo): move signature into separate file
Change-Id: Ic257475e2afebf059c5317c1cc5b04ba63d5d318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10078
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 21:58:02 +00:00