Commit graph

2150 commits

Author SHA1 Message Date
Florian Klink
4954a39de4 fix(tvix/castore): also set SSL_CERT_FILE for tests there
For everything using reqwest here during test cases, we also need to
set SSL_CERT_FILE.

Change-Id: If8aeda65f3d75cb9ac5c9bc64e37a0cb7dffc17c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11092
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 17:43:42 +00:00
Florian Klink
0b396553d6 refactor(tvix/store/protos): more granular nix-compat conversions
implement From<&nix_compat::nixhash::CAHash> for nar_info::ca::Hash
individually, and make
From<&nix_compat::nixhash::CAHash> for nar_info::Ca a small wrapper that
uses it, as well as the .hash().digest_as_bytes() for the digest.

Change-Id: I7e9b6edd1e3f149eb270faf2928cd846d74e77ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11071
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 17:12:51 +00:00
Florian Klink
ef3f8936cb refactor(tvix/*/from_addr): improve test debuggability
If there's an unexpected test failure, print it out, rather than just
saying something is false even though it should be true.

Use .expect() for this, which displays the error if it failed.
We can't use expect_err(), as our stores are not display'able, so use an
assertion with a message there.

Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 16:54:19 +00:00
Florian Klink
b0e5a932ec docs(tvix): remove differences-drv-paths.md
This was true while we didn't do string contexts, but drvPath should not
differ anymore.

Change-Id: I58e6ed439519bd4d68259d1577946f3c65110dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11074
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 16:53:41 +00:00
Florian Klink
c9514b7c0c feat(tvix/docs): ship custom highlight.js with Nix support
Selected languages:
 - bash
 - c
 - cpp
 - diff
 - go
 - ini
 - json
 - markdown
 - nix
 - protobuf
 - rust
 - shell
 - xml
 - yaml

Created with https://highlightjs.org/download

Change-Id: I76a87d2c1e9fa582d1474f8ffb9f3129e9a858bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11073
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-03-03 16:53:16 +00:00
Florian Klink
65a810fc0b feat(tvix/docs): switch to mdbook
Some of the docs are still outdated (like architecture and drv path
inconsistencies).

Change-Id: I7a6afceb008ef4cd19a764dd6c637b39fa842a2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11072
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-03-03 16:52:38 +00:00
edef
6bdaebcb55 fix(tvix/tools/weave): handle sliced arrays correctly
The start/end offsets are not necessarily coterminous with the underlying
values array, so even if the stride is fixed, we still we need to slice
the chunks down to match the start/end offsets.

This bug shouldn't affect the correctness of any existing code, since
we're always working with unsliced arrays read directly from Parquet.

Change-Id: I2f7ddc4e66d4d3b2317a44bd436a35bff36bac79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11081
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 16:45:19 +00:00
Florian Klink
1c16dee207 feat(tvix/store): use reqwests' rustls-native-roots feature
This makes reqwest honor `SSL_CERT_FILE` - previously it was using the
chain bundled in webpki-roots.

`object_store` pulls in `reqwest` with this feature, and the cargo
solver will enable that feature globally as soon as we pull it in, as it
assumes features are additive.

This requires setting `SSL_CERT_FILE` when running tests, otherwise
they'll fail with the unhelpful "NotFound" error.

This was quite some fun to debug, why adding `object_store` to
tvix-castore suddenly made tvix-store tests fail!

Change-Id: I64fc82b4d994715480efdb1ffecb279716456ab9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11090
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:32:38 +00:00
Florian Klink
4b4443240e feat(tvix/castore): add HashingReader, B3HashingReader
HashingReader wraps an existing AsyncRead, and allows querying for the
digest of all data read "through" it.
The hash function is configurable by type parameter, and we define
B3HashingReader.

Change-Id: Ic08142077566fc08836662218f5ec8c3aff80be5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11087
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:31:31 +00:00
Florian Klink
8383e9e02e feat(tvix/castore/digests): impl From digest::Output<_> for B3Digest
This allows calling .into() to get a B3Digest.

Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-03-03 15:18:19 +00:00
Florian Klink
442eabe9d0 feat(tvix/nix-compat): add HashMode and CAHash::mode()
This allows peeking at the mode without matching on all enum kinds of
CAHash directly.

Change-Id: Icca147a876f7f4cff0bbaa067ac4dae40d05bee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11069
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 13:17:50 +00:00
Brian Olsen
eff2cc4f68 fix(tvix/nix-compat): Make CAHash deserialize more formats
Currently CAHash only deserializes the hash in hex code while
the serializer outputs a nixbase32 hash. This means that you can't currently
deserialize what has been serialized.

This change makes deserialize support any digest format (so hex, nixbase32
and base64) as well as flattens the deserialize code and error handling.

It also implements serde methods of HashAlgo directly using Display and TryFrom
implementations because otherwise these would get serialized as eg. Sha256 instead
of sha256 which also broke CAHash serialize/deserialize.

Change-Id: I1941a72eaec741e4956292adaaf0115b97f260ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11082
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 13:07:18 +00:00
Florian Klink
260c2938d4 refactor(tvix/nix-compat): rename NixHash::{digest,hash}
This doesn't return the digest, but the internal NixHash.

To get the digest, you use `ca_hash.hash().digest_as_bytes()`
(as done in tvix/nix-compat/src/derivation/write.rs).

Change-Id: Ib07918dec63ae42ddf8c8d3f1d10510f9c4df255
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11070
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-03 11:56:18 +00:00
Florian Klink
7bebf492ec refactor(tvix/castore/blobsvc/chunked_reader): refactor, document
The public-consumable thing here is ChunkedReader, not ChunkedBlob.

ChunkedBlob is a helper that can be used to get a new AsyncRead, but
not AsyncSeek. It is used internally by ChunkedReader whenever the
client seeks.

Make this more obvious, by extending the documentation, and putting
ChunkedReader at the top of this file.

Also make ChunkedBlob and its methods private, and give ChunkedReader a
more useful constructor (from_chunks, instead of from_chunked_blob).

Change-Id: I2399867591df923faa73927b924e7c116ad98dc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11079
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-03 11:22:56 +00:00
Florian Klink
53fb9ff4c6 feat(tvix/castore/blobsvc): BlobReader for more trivial types
Change-Id: I80e4f26c41a504fa4c6a013c2a1e76de613ba294
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11078
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-02 17:05:23 +00:00
Florian Klink
982459d343 fix(tvix/castore/blobwriter): don't require Sync + 'static
There's no reason for these two.

Change-Id: Ie6f238bbb0b17971c9877b11b61ea7ebca573c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11075
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-02 05:56:21 +00:00
Padraic-O-Mhuiris
e0ffb4d47a feat(tvix/eval): implement builtins.hashFile
The primary addition is the `builtin_hashFile` function which attempts
value-path coercion and on success reads the file to a nix_string.
Similar to the `builtins_hashString` implementation, the hash is then
derived using the `hash_nix_string` function in the introduced hash
module which has extracted hashing functionality initially specified in
cl/11005.

Change-Id: I5e3ef0317223af99488ebb79efc2fb49b4cbc1b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11007
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-29 10:58:21 +00:00
edef
e3860689ba feat(tvix/tools/weave): init
Scalable tracing GC for the cache.nixos.org dataset.

Change-Id: I6c7852796f28e1a1c7607384ffb55f44407e1185
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10765
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-27 11:50:21 +00:00
Florian Klink
d5ec9a0c6f chore: update links to tvixbolt
Use bolt.tvix.dev everywhere.

Change-Id: Ifebbea60058418c12cde20a2a3879a8a7f5f830c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11022
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-24 10:17:56 +00:00
Padraic-O-Mhuiris
5c3065b43a feat(tvix/eval): implement builtins.hashString
Implements md5, sha1, sha256 and sha512 using the related crates from
the RustCrypto hashes project (https://github.com/RustCrypto/hashes)

Change-Id: I00730dea44ec9ef85309edc27addab0ae88814b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11005
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-02-23 16:04:35 +00:00
Aspen Smith
ffb134398d refactor(tvix/glue): Make a single errors.rs module
To pave the way for adding a new error type for builtins in this crate,
move DerivationError to a new builtins::errors module.

Change-Id: I65fcad63e43ed40ad39c2c6540a2ab80fdd90fd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11016
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-02-23 15:47:46 +00:00
Florian Klink
91d5745c3d refactor(nix-compat/store_path): simplify build_ca_path
Move the the `fixed:out:[r:]{}:` generation to a helper function,
use matches! for more clarity.

Change-Id: I4e930c42aacbf5c7451d1f8c8c80ccb4c45389f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11006
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
2024-02-22 17:57:55 +00:00
Aspen Smith
5e31096154 feat(tvix/eval): Store string context alongside data
Previously, Nix strings were represented as a Box (within Value)
pointing to a tuple of an optional context, and another Box pointing to
the actual string allocation itself. This is pretty inefficient, both in
terms of memory usage (we use 48 whole bytes for a None context!) and in
terms of the extra indirection required to get at the actual data. It
was necessary, however, because with native Rust DSTs if we had
something like `struct NixString(Option<NixContext>, BStr)` we could
only pass around *fat* pointers to that value (with the length in the
pointer) and that'd make Value need to be bigger (which is a waste of
both memory and cache space, since that memory would be unused for all
other Values).

Instead, this commit implements *manual* allocation of a packed string
representation, with the length *in the allocation* as a field past the
context. This requires a big old pile of unsafe Rust, but the payoff is
clear:

    hello outpath  time:   [882.18 ms 897.16 ms 911.23 ms]
                   change: [-15.143% -13.819% -12.500%] (p = 0.00 < 0.05)
                   Performance has improved.

Fortunately this change can be localized entirely within
value/string.rs, since we were abstracting things out nicely.

Change-Id: Ibf56dd16c9c503884f64facbb7f0ac596463efb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10852
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: aspen <root@gws.fyi>
2024-02-21 20:53:04 +00:00
aspen
d74c68025b revert(tvix): don't use Rc::clone explicitly
This reverts commit d9565a4d0a.

Reason for revert: this was intentional - putting Rc::clone instead of
.clone is a common Rust idiom, and makes it explicit that we're cloning
a shared reference, not an underlying resource

Change-Id: I41a5f323ee35d7025dc7bb02f7d5d05d0051798d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10995
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 18:36:11 +00:00
Florian Klink
771200df7c fix(tvix/eval): allow reading non-UTF8 files
With our values using bstr now, we're not restricted to only reading
files that contain valid UTF-8.

Update our `read_to_string` function to `read_to_end`
(named like `std::io::Read::read_to_end`), and have it return a Vec<u8>.

Change-Id: I87f0291dc855a132689576559c891d66c30ddf2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11003
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Pádraic Ó Mhuiris <patrick.morris.310@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 13:55:41 +00:00
sterni
3e93efdc8c docs(tvix/eval): notes on the implementation of catchable errors
Change-Id: I3b2f039d2c2d40ee5d13d071915ced6f3deb18f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11000
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-21 13:32:38 +00:00
Peter Kolloch
fde488ec6d feat(tvix/nix-compat): Use StorePath in Output
https: //b.tvl.fyi/issues/264
Change-Id: Icb09be9643245cc68d09f01d7723af2d44d6bd1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11001
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-21 11:38:03 +00:00
Peter Kolloch
035f617b7f feat(tvix/nix-compat): input_sources as StorePath
https: //b.tvl.fyi/issues/264
Change-Id: I7a235734dc1f8e93e387a04ba369f3b702c6d5b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10992
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-21 11:34:24 +00:00
Peter Kolloch
c06fb01b3b feat(tvix/nix-compat): input_derivations with StorePaths
...in `Derivation`.

This is more type-safe and should consume less memory.

This also removes some allocations in the potentially hot path of output hash calculation.

https: //b.tvl.fyi/issues/264
Change-Id: I6ad7d3cb868dc9f750894d449a6065608ef06e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-21 11:34:24 +00:00
Peter Kolloch
a44a8985cc feat(tvix/nix-compat): generalize aterm writing for derivation
...so that we can also use `StorePath`s in
derivation.input_derivations.

Towards https://b.tvl.fyi/issues/264

Change-Id: I71d296ca273979c70f277a7f4f88a5f76de3d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10973
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-21 11:34:24 +00:00
Aspen Smith
5ced8e7292 fix(tvix/eval): Propagate catchables in builtins.groupBy
One last place where we needed to wrap the forcing of the element of a
list in `try_value!`. This fixes a previously `notyetpassing` test

Change-Id: I8827a3e39630e6959013b70bdfa9cbcb93f4e91c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10789
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-02-20 23:53:58 +00:00
Ryan Lahfa
7388078630 feat(tvix/eval): implement builtins.filterSource
We add a new set of builtins called `import_builtins`, which
will contain import-related builtins, such as `builtins.path` and
`builtins.filterSource`. Both can import paths into the store, with
various knobs to alter the result, e.g. filtering, renaming, expected
hashes.

We introduce `filtered_ingest` which will drive the filtered ingestion
via the Nix function via the generator machinery, and then we register
the root node to the path info service inside the store.

`builtins.filterSource` is very simple, `builtins.path` is a more
complicated model requiring the same logic albeit more sophisticated
with name customization, file ingestion method and expected SHA-256.

Change-Id: I1083f37808b35f7b37818c8ffb9543d9682b2de2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10654
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 14:16:36 +00:00
Vincent Ambo
20833656ae fix(tvix/eval): propagate catchable errors at the top of an eval
(Re-)Adds an error variant that wraps a catchable error kind, which is
used for returning the result of an evaluation.

Previously this would return the internal catchable value, which would
lead to panics if users tried to use these. Somehow this was missed; I
think we need error output tests.

Change-Id: Id6e24aa2ce4ea4358a29b2e1cf4a6749986baf8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10991
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 12:33:57 +00:00
Florian Klink
d9565a4d0a refactor(tvix): don't use Rc::clone explicitly
All these functions have an Rc of Rc<TvixStoreIO> or Rc<dyn StoreIO>, so
we can call io.clone() directly.

Change-Id: I96a5bcb9af4aca93892cb72bcfaf14540da52381
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10989
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-20 12:24:13 +00:00
Vincent Ambo
d0d1027a85 feat(tvix/serde): add an example using nixpkgs/lib
This displays how users can configure an impure evaluation for
tvix-serde, which makes it possible to use e.g. `nixpkgs/lib`.

We might want to add an example showing how the full Nix-glue
compatibility stuff can be added here, too.

Change-Id: I2224a3fc66e739969d4c723c3d9d8127a046b6fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10994
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2024-02-20 12:13:45 +00:00
Vincent Ambo
e839116e5c fix(tvix/eval): make path resolution work by default in impure mode
The previous behaviour (enabling `import`, but not allowing e.g.
`<nixpkgs/lib>` to resolve) was very confusing.

Now imports from NIX_PATH become enabled by default, unless the user
already overrode that behaviour with something else by setting
`Evaluation::nix_path` manually.

Change-Id: Iad970beb633d9887be4b185b01e6f5858d81bea3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10993
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 12:13:45 +00:00
Vincent Ambo
f40ff1737e fix(tvix/eval): fix accidental recursion in printing errors
There's some code path where fancy_format_str()->Error::Display, which
recurses forever and overflows the stack. This was introduced in a
previous commit today.

Change-Id: I87a59492099f6c138c752478901b9aa614bb57cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10990
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
2024-02-20 11:54:03 +00:00
Vincent Ambo
94f582341e refactor(tvix/eval): use internal SourceCode field in error printers
Makes use of the SourceCode field now being stored directly in
errors (see parent CL). With this change, the default `Display`
implementation can now format errors correctly, and there is no need
to keep a `SourceCode` around just for error formatting.

Updates dependent crates (CLI, serde, tvixbolt) to use this correctly.

Change-Id: Iddc5d7a6b4bab391f30a999e4c68aca34304c059
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10987
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 09:18:08 +00:00
Vincent Ambo
3c87687798 refactor(tvix/eval): add SourceCode directly into error types
With this change it's no longer necessary to track the SourceCode
struct separately from the evaluation for error reporting: It's just
stored directly in the errors.

This also ends up resolving an issue in compiler::bindings, where we
cloned the Arc containing file references way too often. In fact those
clones probably compensate for all additional SourceCode clones during
error construction now.

Change-Id: Ice93bf161e61f8ea3d48103435e20c53e6aa8c3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10986
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-20 09:18:08 +00:00
Florian Klink
b38badf206 docs(tvix/castore/directorysvc): K/V is not necessarily flat
Some implementations of DirectoryService might not allow retrieval of
intermediate Directory nodes, that are not at the "root".

Think about an object store implementation. The client is doing a
get_recursive anyways to reduce the number of roundtrips.

By documenting the fact we don't need to support looking up intermediate
Directory messages, we can just batch all directories into the same
object, keyed by the root.

Change-Id: I019d720186d03c4125cec9191e93d20586a20963
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10988
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 09:17:38 +00:00
Vincent Ambo
2277e62c2d refactor(tvix/eval): add API for enabling impure evaluation features
This makes some of the work of configuring an arbitrary I/O handler
easier.

Change-Id: I158db3235fe83df6e709578ed515e0e028c20086
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10959
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-20 08:46:51 +00:00
Florian Klink
43570bae11 refactor(tvix/store/proto): use bstr::ByteSlice
String::from_utf8_lossy simply discards invalid bytes, while bstr
replaces them with their replacement character.

Change-Id: Ib78ff36ca5faacc1ad60bc4ddde7b62773848c07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10981
Tested-by: BuildkiteCI
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 05:29:14 +00:00
Vincent Ambo
98a17dbdf9 chore(tvix/glue): remove unnecessary assignment
Change-Id: I747307317c45085f9f7762d659870c5bd75b908b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10958
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 04:58:31 +00:00
Aspen Smith
0631b870b0 fix(tvix/store): Improve error message for InvalidNodeName
Rather than debug-formatting a Vec<u8>, which gives just a sequence of
numbers (not very helpful!) use String::from_utf8_lossy to give a much
more human-readable error message

Change-Id: I6cae795ebe2ab9a656c087b82a143411e6c7c81a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10980
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-20 03:48:40 +00:00
Peter Kolloch
46d89f899f feat(tvix/nix-compat): Extract to_plain_hex_string
Towards https://b.tvl.fyi/issues/264

Change-Id: Ibde971bfb6baa97b5c678d84ce1941189bc59f6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10969
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-19 17:32:10 +00:00
Peter Kolloch
df73d5f242 refactor(tvix/nix-compat): Restrict visibility
Towards https://b.tvl.fyi/issues/264

Change-Id: I323907cb842ed10ddd05b72c285c598b60d213e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10972
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-19 17:15:31 +00:00
Peter Kolloch
5777050821 feat(tvix/castore): Compile fix for Darwin
Towards https://b.tvl.fyi/issues/264

Change-Id: If8fa912ae3fb2987b761f649ab738529ebf3b2e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10970
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-19 17:14:24 +00:00
Aspen Smith
0db46dacea feat(tvix/glue): Init fetcher builtins
Initialize a new empty builtins module `fetcher_builtins`, which will
contain the builtins which fetch URLs from the internet:

* fetchurl
* fetchGit
* fetchTarball
* fetchTree (maybe? this is experimental)

These builtins are all implemented in CPP nix at:
https://github.com/NixOS/nix/blob/2.20.2/src/libexpr/primops/fetchTree.cc

These builtins are added to the evaluation context using a similar
mechanism to the derivation builtins, and have been added everywhere
derivation builtins were previously being added.

Change-Id: I133b91cc9560f23028621414537f712e7bd8a825
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10974
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-19 16:51:10 +00:00
Peter Kolloch
b147452948 feat(tvix/nix-compat): Extend parse errors
...so that we can parse (and at the same time
validate) store paths directly to `StorePath`s.

https: //b.tvl.fyi/issues/264
Change-Id: Ide4cb5403f8e3400f7d2a5838c652ac35dbd1a4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10968
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
2024-02-19 16:36:31 +00:00
edef
f4963cdfd8 chore(tvix/tools/narinfo2parquet): polars 0.34.2 -> 0.36.2
Change-Id: Ib17a7bbe560000c3fef3ba6182819f710d707909
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10967
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-19 16:03:14 +00:00