Commit graph

2160 commits

Author SHA1 Message Date
Florian Klink
f22d5b3d11 feat(tvix/castore/blobsvc/from_addr): support object_store
The object_store crate supports a ton of different stores, with different schemes.

For now, use a objectstore+ scheme prefix to enable these.

Change-Id: I946f76e32a0fb0867ef59060217894cda5b959b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11080
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-03-11 22:42:01 +00:00
Florian Klink
1c2db676a0 feat(tvix/castore/blobsvc): add object storage implementation
This uses the `object_store` crate to expose a tvix-castore BlobService
backed by object storage.

It's using FastCDC to chunk blobs into smaller chunks when writing to
it.

These are exposed at the .chunks() method.

Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-03-11 22:42:01 +00:00
Florian Klink
d327bf775d feat(tvix/store/bin): allow disabling OTLP at runtime
This was only possible by disabling without the otlp feature flag so
far.

Introduce the same --otlp=false mechanism that nar-bridge also supports
to be able to turn it off at runtime.

Change-Id: Ib22a364c35056ca9d8e327c0e2a79970a4cf4b2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11135
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-11 17:02:26 +00:00
Aspen Smith
54609e8c17 feat(tvix/glue): Add AsyncRead wrapper to decompress streams
Add a new AsyncRead wrapper, DecompressedReader, that wraps an
underlying AsyncRead, but sniffs the magic bytes at the start of the
stream to determine which compression format is being used out of the
three that are supported by builtins.fetchTarball, and switches to the
correct decompression algorithm adapter dynamically.

This will be used in the implementation of builtins.fetchTarball

Change-Id: I892a4683d5c93e67d4c173f3d21199bdc6605922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11019
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-11 11:35:18 +00:00
Aspen Smith
de727bccf9 feat(tvix/glue): Implement builtins.fetchurl
Implement the fetchurl builtin, and lay the groundwork for implementing
the fetchTarball builtin (which works very similarly, and is implemented
using almost the same code in C++ nix).

An overview of how this works:

1. First, we check if the store path that *would* result from the
   download already exists in the store - if it does, we just return
   that
2. If we need to download the URL, TvixStoreIO has an `http_client:
   reqwest::Client` field now which we use to make the request
3. As we're downloading the blob, we hash the data incrementally into a
   SHA256 hasher
4. We compare the hash against the expected hash (if any) and bail out
   if it doesn't match
5. Finally, we put the blob in the store and return the store path

Since the logic is very similar, this commit also implements a *chunk*
of `fetchTarball` (though the actual implementation will likely include
a refactor to some of the code reuse here).

The main thing that's missing here is caching of downloaded blobs when
fetchurl is called without a hash - I've opened b/381 to track the TODO
there.

Adding the `SSL_CERT_FILE` here is necessary to teach reqwest how to
load it during tests - see 1c16dee20 (feat(tvix/store): use reqwests'
rustls-native-roots feature, 2024-03-03) for  more info.

Change-Id: I83c4abbc7c0c3bfe92461917e23d6d3430fbf137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-03-11 02:21:54 +00:00
Florian Klink
37703c9e44 docs(tvix/README): update IRC channels
`#tvl` is the general TVL community channel, `#tvix-dev` is dedicated to
Tvix development discussion.

Change-Id: If899d9ae70dffb4acf6a41ded54f80a1f0551c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11110
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-10 17:14:41 +00:00
Florian Klink
bd1def3ec4 fix(tvix/store/grpc/pathinfo): skip_all fields, handle errors
request only contains the outer metadata wrapping, and that's not too
interesting:

> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }

Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.

Also, ensure we handle all error cases properly, and log them. We
don't use `err` from instrument, as that'd also log an error on
`Status::not_found`.

Change-Id: Id1b983cb8b059c148c8a376f8802a1d28c59ba97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11103
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-09 05:48:46 +00:00
Florian Klink
4e78de7393 fix(tvix/castore/grpc/directory): skip_all fields in instrument
This only contains the outer metadata wrapping, and that's not too interesting:

> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }

Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.

Clean up the error logging a bit.

Change-Id: Ife1090ed411766a61e1fa60fd4c9570f38de1e98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11102
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-09 05:47:41 +00:00
Florian Klink
05deb37f44 fix(tvix/castore/grpc/blob): skip_all fields in instrument
This only contains the outer metadata wrapping, and that's not too interesting:

> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }

Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.

Log errors in some places where we didn't so far.

Change-Id: Ia68d6c526987d3716be62a0809195401cf28512b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11101
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-09 05:47:36 +00:00
Florian Klink
8f804f5d9c chore(tvix/nix-compat): bump zstd dependency
Otherwise this causes conflicts when adding a more recent zstd version
to another crate.

Change-Id: I02d1b1a37f18711c0969cb04e761e967fc4655dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11100
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-09 05:47:03 +00:00
Florian Klink
4954a39de4 fix(tvix/castore): also set SSL_CERT_FILE for tests there
For everything using reqwest here during test cases, we also need to
set SSL_CERT_FILE.

Change-Id: If8aeda65f3d75cb9ac5c9bc64e37a0cb7dffc17c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11092
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 17:43:42 +00:00
Florian Klink
0b396553d6 refactor(tvix/store/protos): more granular nix-compat conversions
implement From<&nix_compat::nixhash::CAHash> for nar_info::ca::Hash
individually, and make
From<&nix_compat::nixhash::CAHash> for nar_info::Ca a small wrapper that
uses it, as well as the .hash().digest_as_bytes() for the digest.

Change-Id: I7e9b6edd1e3f149eb270faf2928cd846d74e77ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11071
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 17:12:51 +00:00
Florian Klink
ef3f8936cb refactor(tvix/*/from_addr): improve test debuggability
If there's an unexpected test failure, print it out, rather than just
saying something is false even though it should be true.

Use .expect() for this, which displays the error if it failed.
We can't use expect_err(), as our stores are not display'able, so use an
assertion with a message there.

Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 16:54:19 +00:00
Florian Klink
b0e5a932ec docs(tvix): remove differences-drv-paths.md
This was true while we didn't do string contexts, but drvPath should not
differ anymore.

Change-Id: I58e6ed439519bd4d68259d1577946f3c65110dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11074
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 16:53:41 +00:00
Florian Klink
c9514b7c0c feat(tvix/docs): ship custom highlight.js with Nix support
Selected languages:
 - bash
 - c
 - cpp
 - diff
 - go
 - ini
 - json
 - markdown
 - nix
 - protobuf
 - rust
 - shell
 - xml
 - yaml

Created with https://highlightjs.org/download

Change-Id: I76a87d2c1e9fa582d1474f8ffb9f3129e9a858bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11073
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-03-03 16:53:16 +00:00
Florian Klink
65a810fc0b feat(tvix/docs): switch to mdbook
Some of the docs are still outdated (like architecture and drv path
inconsistencies).

Change-Id: I7a6afceb008ef4cd19a764dd6c637b39fa842a2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11072
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-03-03 16:52:38 +00:00
edef
6bdaebcb55 fix(tvix/tools/weave): handle sliced arrays correctly
The start/end offsets are not necessarily coterminous with the underlying
values array, so even if the stride is fixed, we still we need to slice
the chunks down to match the start/end offsets.

This bug shouldn't affect the correctness of any existing code, since
we're always working with unsliced arrays read directly from Parquet.

Change-Id: I2f7ddc4e66d4d3b2317a44bd436a35bff36bac79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11081
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 16:45:19 +00:00
Florian Klink
1c16dee207 feat(tvix/store): use reqwests' rustls-native-roots feature
This makes reqwest honor `SSL_CERT_FILE` - previously it was using the
chain bundled in webpki-roots.

`object_store` pulls in `reqwest` with this feature, and the cargo
solver will enable that feature globally as soon as we pull it in, as it
assumes features are additive.

This requires setting `SSL_CERT_FILE` when running tests, otherwise
they'll fail with the unhelpful "NotFound" error.

This was quite some fun to debug, why adding `object_store` to
tvix-castore suddenly made tvix-store tests fail!

Change-Id: I64fc82b4d994715480efdb1ffecb279716456ab9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11090
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:32:38 +00:00
Florian Klink
4b4443240e feat(tvix/castore): add HashingReader, B3HashingReader
HashingReader wraps an existing AsyncRead, and allows querying for the
digest of all data read "through" it.
The hash function is configurable by type parameter, and we define
B3HashingReader.

Change-Id: Ic08142077566fc08836662218f5ec8c3aff80be5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11087
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:31:31 +00:00
Florian Klink
8383e9e02e feat(tvix/castore/digests): impl From digest::Output<_> for B3Digest
This allows calling .into() to get a B3Digest.

Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:18:19 +00:00
Florian Klink
442eabe9d0 feat(tvix/nix-compat): add HashMode and CAHash::mode()
This allows peeking at the mode without matching on all enum kinds of
CAHash directly.

Change-Id: Icca147a876f7f4cff0bbaa067ac4dae40d05bee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11069
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 13:17:50 +00:00
Brian Olsen
eff2cc4f68 fix(tvix/nix-compat): Make CAHash deserialize more formats
Currently CAHash only deserializes the hash in hex code while
the serializer outputs a nixbase32 hash. This means that you can't currently
deserialize what has been serialized.

This change makes deserialize support any digest format (so hex, nixbase32
and base64) as well as flattens the deserialize code and error handling.

It also implements serde methods of HashAlgo directly using Display and TryFrom
implementations because otherwise these would get serialized as eg. Sha256 instead
of sha256 which also broke CAHash serialize/deserialize.

Change-Id: I1941a72eaec741e4956292adaaf0115b97f260ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11082
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 13:07:18 +00:00
Florian Klink
260c2938d4 refactor(tvix/nix-compat): rename NixHash::{digest,hash}
This doesn't return the digest, but the internal NixHash.

To get the digest, you use `ca_hash.hash().digest_as_bytes()`
(as done in tvix/nix-compat/src/derivation/write.rs).

Change-Id: Ib07918dec63ae42ddf8c8d3f1d10510f9c4df255
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11070
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 11:56:18 +00:00
Florian Klink
7bebf492ec refactor(tvix/castore/blobsvc/chunked_reader): refactor, document
The public-consumable thing here is ChunkedReader, not ChunkedBlob.

ChunkedBlob is a helper that can be used to get a new AsyncRead, but
not AsyncSeek. It is used internally by ChunkedReader whenever the
client seeks.

Make this more obvious, by extending the documentation, and putting
ChunkedReader at the top of this file.

Also make ChunkedBlob and its methods private, and give ChunkedReader a
more useful constructor (from_chunks, instead of from_chunked_blob).

Change-Id: I2399867591df923faa73927b924e7c116ad98dc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11079
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-03 11:22:56 +00:00
Florian Klink
53fb9ff4c6 feat(tvix/castore/blobsvc): BlobReader for more trivial types
Change-Id: I80e4f26c41a504fa4c6a013c2a1e76de613ba294
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11078
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-02 17:05:23 +00:00
Florian Klink
982459d343 fix(tvix/castore/blobwriter): don't require Sync + 'static
There's no reason for these two.

Change-Id: Ie6f238bbb0b17971c9877b11b61ea7ebca573c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11075
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-02 05:56:21 +00:00
Padraic-O-Mhuiris
e0ffb4d47a feat(tvix/eval): implement builtins.hashFile
The primary addition is the `builtin_hashFile` function which attempts
value-path coercion and on success reads the file to a nix_string.
Similar to the `builtins_hashString` implementation, the hash is then
derived using the `hash_nix_string` function in the introduced hash
module which has extracted hashing functionality initially specified in
cl/11005.

Change-Id: I5e3ef0317223af99488ebb79efc2fb49b4cbc1b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11007
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-29 10:58:21 +00:00
edef
e3860689ba feat(tvix/tools/weave): init
Scalable tracing GC for the cache.nixos.org dataset.

Change-Id: I6c7852796f28e1a1c7607384ffb55f44407e1185
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10765
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-27 11:50:21 +00:00
Florian Klink
d5ec9a0c6f chore: update links to tvixbolt
Use bolt.tvix.dev everywhere.

Change-Id: Ifebbea60058418c12cde20a2a3879a8a7f5f830c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11022
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-24 10:17:56 +00:00
Padraic-O-Mhuiris
5c3065b43a feat(tvix/eval): implement builtins.hashString
Implements md5, sha1, sha256 and sha512 using the related crates from
the RustCrypto hashes project (https://github.com/RustCrypto/hashes)

Change-Id: I00730dea44ec9ef85309edc27addab0ae88814b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11005
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-02-23 16:04:35 +00:00
Aspen Smith
ffb134398d refactor(tvix/glue): Make a single errors.rs module
To pave the way for adding a new error type for builtins in this crate,
move DerivationError to a new builtins::errors module.

Change-Id: I65fcad63e43ed40ad39c2c6540a2ab80fdd90fd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11016
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-02-23 15:47:46 +00:00
Florian Klink
91d5745c3d refactor(nix-compat/store_path): simplify build_ca_path
Move the the `fixed:out:[r:]{}:` generation to a helper function,
use matches! for more clarity.

Change-Id: I4e930c42aacbf5c7451d1f8c8c80ccb4c45389f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11006
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2024-02-22 17:57:55 +00:00
Aspen Smith
5e31096154 feat(tvix/eval): Store string context alongside data
Previously, Nix strings were represented as a Box (within Value)
pointing to a tuple of an optional context, and another Box pointing to
the actual string allocation itself. This is pretty inefficient, both in
terms of memory usage (we use 48 whole bytes for a None context!) and in
terms of the extra indirection required to get at the actual data. It
was necessary, however, because with native Rust DSTs if we had
something like `struct NixString(Option<NixContext>, BStr)` we could
only pass around *fat* pointers to that value (with the length in the
pointer) and that'd make Value need to be bigger (which is a waste of
both memory and cache space, since that memory would be unused for all
other Values).

Instead, this commit implements *manual* allocation of a packed string
representation, with the length *in the allocation* as a field past the
context. This requires a big old pile of unsafe Rust, but the payoff is
clear:

    hello outpath  time:   [882.18 ms 897.16 ms 911.23 ms]
                   change: [-15.143% -13.819% -12.500%] (p = 0.00 < 0.05)
                   Performance has improved.

Fortunately this change can be localized entirely within
value/string.rs, since we were abstracting things out nicely.

Change-Id: Ibf56dd16c9c503884f64facbb7f0ac596463efb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10852
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: aspen <root@gws.fyi>
2024-02-21 20:53:04 +00:00
aspen
d74c68025b revert(tvix): don't use Rc::clone explicitly
This reverts commit d9565a4d0a.

Reason for revert: this was intentional - putting Rc::clone instead of
.clone is a common Rust idiom, and makes it explicit that we're cloning
a shared reference, not an underlying resource

Change-Id: I41a5f323ee35d7025dc7bb02f7d5d05d0051798d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10995
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 18:36:11 +00:00
Florian Klink
771200df7c fix(tvix/eval): allow reading non-UTF8 files
With our values using bstr now, we're not restricted to only reading
files that contain valid UTF-8.

Update our `read_to_string` function to `read_to_end`
(named like `std::io::Read::read_to_end`), and have it return a Vec<u8>.

Change-Id: I87f0291dc855a132689576559c891d66c30ddf2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11003
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Pádraic Ó Mhuiris <patrick.morris.310@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 13:55:41 +00:00
sterni
3e93efdc8c docs(tvix/eval): notes on the implementation of catchable errors
Change-Id: I3b2f039d2c2d40ee5d13d071915ced6f3deb18f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11000
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-21 13:32:38 +00:00
Peter Kolloch
fde488ec6d feat(tvix/nix-compat): Use StorePath in Output
https: //b.tvl.fyi/issues/264
Change-Id: Icb09be9643245cc68d09f01d7723af2d44d6bd1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11001
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-21 11:38:03 +00:00
Peter Kolloch
035f617b7f feat(tvix/nix-compat): input_sources as StorePath
https: //b.tvl.fyi/issues/264
Change-Id: I7a235734dc1f8e93e387a04ba369f3b702c6d5b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10992
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-21 11:34:24 +00:00
Peter Kolloch
c06fb01b3b feat(tvix/nix-compat): input_derivations with StorePaths
...in `Derivation`.

This is more type-safe and should consume less memory.

This also removes some allocations in the potentially hot path of output hash calculation.

https: //b.tvl.fyi/issues/264
Change-Id: I6ad7d3cb868dc9f750894d449a6065608ef06e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-21 11:34:24 +00:00
Peter Kolloch
a44a8985cc feat(tvix/nix-compat): generalize aterm writing for derivation
...so that we can also use `StorePath`s in
derivation.input_derivations.

Towards https://b.tvl.fyi/issues/264

Change-Id: I71d296ca273979c70f277a7f4f88a5f76de3d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10973
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 11:34:24 +00:00
Aspen Smith
5ced8e7292 fix(tvix/eval): Propagate catchables in builtins.groupBy
One last place where we needed to wrap the forcing of the element of a
list in `try_value!`. This fixes a previously `notyetpassing` test

Change-Id: I8827a3e39630e6959013b70bdfa9cbcb93f4e91c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10789
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-02-20 23:53:58 +00:00
Ryan Lahfa
7388078630 feat(tvix/eval): implement builtins.filterSource
We add a new set of builtins called `import_builtins`, which
will contain import-related builtins, such as `builtins.path` and
`builtins.filterSource`. Both can import paths into the store, with
various knobs to alter the result, e.g. filtering, renaming, expected
hashes.

We introduce `filtered_ingest` which will drive the filtered ingestion
via the Nix function via the generator machinery, and then we register
the root node to the path info service inside the store.

`builtins.filterSource` is very simple, `builtins.path` is a more
complicated model requiring the same logic albeit more sophisticated
with name customization, file ingestion method and expected SHA-256.

Change-Id: I1083f37808b35f7b37818c8ffb9543d9682b2de2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10654
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 14:16:36 +00:00
Vincent Ambo
20833656ae fix(tvix/eval): propagate catchable errors at the top of an eval
(Re-)Adds an error variant that wraps a catchable error kind, which is
used for returning the result of an evaluation.

Previously this would return the internal catchable value, which would
lead to panics if users tried to use these. Somehow this was missed; I
think we need error output tests.

Change-Id: Id6e24aa2ce4ea4358a29b2e1cf4a6749986baf8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10991
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 12:33:57 +00:00
Florian Klink
d9565a4d0a refactor(tvix): don't use Rc::clone explicitly
All these functions have an Rc of Rc<TvixStoreIO> or Rc<dyn StoreIO>, so
we can call io.clone() directly.

Change-Id: I96a5bcb9af4aca93892cb72bcfaf14540da52381
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10989
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-20 12:24:13 +00:00
Vincent Ambo
d0d1027a85 feat(tvix/serde): add an example using nixpkgs/lib
This displays how users can configure an impure evaluation for
tvix-serde, which makes it possible to use e.g. `nixpkgs/lib`.

We might want to add an example showing how the full Nix-glue
compatibility stuff can be added here, too.

Change-Id: I2224a3fc66e739969d4c723c3d9d8127a046b6fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10994
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2024-02-20 12:13:45 +00:00
Vincent Ambo
e839116e5c fix(tvix/eval): make path resolution work by default in impure mode
The previous behaviour (enabling `import`, but not allowing e.g.
`<nixpkgs/lib>` to resolve) was very confusing.

Now imports from NIX_PATH become enabled by default, unless the user
already overrode that behaviour with something else by setting
`Evaluation::nix_path` manually.

Change-Id: Iad970beb633d9887be4b185b01e6f5858d81bea3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10993
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 12:13:45 +00:00
Vincent Ambo
f40ff1737e fix(tvix/eval): fix accidental recursion in printing errors
There's some code path where fancy_format_str()->Error::Display, which
recurses forever and overflows the stack. This was introduced in a
previous commit today.

Change-Id: I87a59492099f6c138c752478901b9aa614bb57cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10990
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-20 11:54:03 +00:00
Vincent Ambo
94f582341e refactor(tvix/eval): use internal SourceCode field in error printers
Makes use of the SourceCode field now being stored directly in
errors (see parent CL). With this change, the default `Display`
implementation can now format errors correctly, and there is no need
to keep a `SourceCode` around just for error formatting.

Updates dependent crates (CLI, serde, tvixbolt) to use this correctly.

Change-Id: Iddc5d7a6b4bab391f30a999e4c68aca34304c059
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10987
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 09:18:08 +00:00
Vincent Ambo
3c87687798 refactor(tvix/eval): add SourceCode directly into error types
With this change it's no longer necessary to track the SourceCode
struct separately from the evaluation for error reporting: It's just
stored directly in the errors.

This also ends up resolving an issue in compiler::bindings, where we
cloned the Arc containing file references way too often. In fact those
clones probably compensate for all additional SourceCode clones during
error construction now.

Change-Id: Ice93bf161e61f8ea3d48103435e20c53e6aa8c3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10986
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-20 09:18:08 +00:00
Florian Klink
b38badf206 docs(tvix/castore/directorysvc): K/V is not necessarily flat
Some implementations of DirectoryService might not allow retrieval of
intermediate Directory nodes, that are not at the "root".

Think about an object store implementation. The client is doing a
get_recursive anyways to reduce the number of roundtrips.

By documenting the fact we don't need to support looking up intermediate
Directory messages, we can just batch all directories into the same
object, keyed by the root.

Change-Id: I019d720186d03c4125cec9191e93d20586a20963
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10988
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 09:17:38 +00:00