Commit graph

1222 commits

Author SHA1 Message Date
Florian Klink
62ecfc7001 chore(tvix/store): allow(unused_mut)
When building tvix-store without default features, this variable doesn't
need to be mutable. Silence the warning.

Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-12 16:45:37 +00:00
Florian Klink
c9c8b10370 feat(tvix/eval): don't warn twice about dead code
We currently send two warnings in case of detecting dead code - W008
inside compile_dead_code, and a more detailed warning in all places that
invoke compile_dead_code:

```
warning[W007]: useless operation on boolean: this expression is always false
     --> /nix/store/qz3gjn95gazab4fkb7s8lm6hz17rdzzy-414z9nnj1wy66ymq6vgb693x9xjz6hf2-nixpkgs-src/pkgs/top-level/perl-packages.nix:12079:15
      |
12079 |     doCheck = false && !stdenv.isDarwin;
      |               ^^^^^^^^^^^^^^^^^^^^^^^^^

warning[W008]: this code will never be executed
     --> /nix/store/qz3gjn95gazab4fkb7s8lm6hz17rdzzy-414z9nnj1wy66ymq6vgb693x9xjz6hf2-nixpkgs-src/pkgs/top-level/perl-packages.nix:12079:24
      |
12079 |     doCheck = false && !stdenv.isDarwin;
      |                        ^^^^^^^^^^^^^^^^
```

The place invoking `compile_dead_code` has more context to why the code
is unused, so it's error message is much more useful.

Stop emitting the less informative warning inside compile_dead_code
(W008), and update the comment that we expect the caller to emit a
warning.

I kept W008 itself still around, in case we end up having places this
will get used again.

Change-Id: I2c5d84fc0cb4035872cd4b71cc3e9e34e120eb37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8024
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:26:49 +00:00
Florian Klink
e4687765d7 refactor(tvix/store): bump fastcdc dep
This removes the use of Box::new, switching fastcdc to version 3.0.2
with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed.

Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-11 14:12:46 +00:00
Florian Klink
b049b88d2d refactor(tvix/store): factor out hash update into function
We're using this in a bunch of places. Let's move it into a helper
function.

Change-Id: I118fba35f6d343704520ba37280e4ca52a61da44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8251
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
2dc93f8de2 docs(tvix/store): make upload_chunk docs a docstring
Change-Id: I6794e2a69b4907ab8d9ed2d55335c11db2c30c02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8250
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-11 14:12:45 +00:00
Florian Klink
ead113cdfc feat(tvix/store/import): use StreamCDC instead of blobwriter
This seems to be way faster.

Change-Id: Ica7cee95d108c51fe67365f07366634ddbbfa060
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8246
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-11 12:54:07 +00:00
Florian Klink
ceb9d670bf refactor(tvix/store): rename C to DS, client to directory_service
This matches the semantics in other gRPC wrappers.

Change-Id: I505c901aa77a58deab216316d08542877b0b2c14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8247
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 16:15:48 +00:00
Florian Klink
3506d3bd0e refactor(tvix/store): move upload_chunk out of blobwriter
This is useful not only in blobwriter contexts.

Change-Id: I4c584b5264ff7b4bb3b1a9671affc39e18bf4ccf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8245
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 15:45:45 +00:00
Florian Klink
ac3e470b15 feat(tvix/store): clippy
warning: the following explicit lifetimes could be elided: 'set
   --> store/src/proto/mod.rs:201:1
    |
201 | / fn update_if_lt_prev<'set, 'n>(
202 | |     prev_name: &'set mut &'n str,
203 | |     name: &'n str,
204 | | ) -> Result<(), ValidateDirectoryError> {
    | |_______________________________________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes
    = note: `#[warn(clippy::needless_lifetimes)]` on by default

Change-Id: I053d370a34e5c6721ef86502c7f225c3fbfb8746
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8244
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:23:04 +00:00
Florian Klink
e1783c45a4 refactor(tvix/store/blobwriter): don't clone
This already is a &CS, so we don't need to clone().

Change-Id: I5397a5948ae7fe4781f18df760a79047f83dca01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8243
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 12:23:04 +00:00
Florian Klink
d2b353e516 refactor(tvix/store): clippy
Change-Id: Ib41c766a45e7d5467a4e5e90e1c00b009121959a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8238
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:15:02 +00:00
Florian Klink
6d3f5b7eb4 refactor(tvix/store): impl Default for MemoryBlobService
Change-Id: I78a4cd86cb364b970e99393579808e773db5ceb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8237
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:15:02 +00:00
Florian Klink
b3d6e70909 refactor(tvix/store): impl Default for MemoryChunkService
Change-Id: If1ceb870e9e5d03b651923a3a786350a8663e990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 12:13:01 +00:00
Florian Klink
e716bd0957 refactor(tvix/store): impl Default for MemoryDirectoryService
Change-Id: Ie75a1e67c8c0499ff10f60d90251b768937dad3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8235
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-10 12:13:01 +00:00
Florian Klink
8e97c0deb0 refactor(tvix/store): impl Default for MemoryPathInfoService
Change-Id: I07f83f8aaa6312cc37ce7e7ee2b713e60df23782
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8234
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 12:13:00 +00:00
Florian Klink
510927e43a feat(tvix/store): use rayon to upload chunks concurrently
Look at the data that's written to us, and upload all chunks but the
rest in parallel, using rayon. This required moving `upload_chunk`
outside the struct, and accepting a ChunkService to use for upload
(which it was previously getting from `self.chunk_service`).

This doesn't speed up things too much for now, because things are still
mostly linear.

Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 11:53:29 +00:00
Florian Klink
2ef60282b6 feat(tvix/store/import): make sure entries are sorted
The Directory service does already reject inserting invalid (wrongly
sorted) Directory messages, but our test case didn't provoke it.

Change-Id: I228e201925e8999186659a2d8da0118db184d9ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8167
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
a4d06b68d8 feat(tvix/store): move to daemon subcommand, add import subcommand
This exposes the previous default behavior at the `tvix-store daemon`
subcommand.

It also adds a `tvix-store import` command, which will ingest a given
path into the store.

Change-Id: Ide14f1d409b9364e7f98090690c744326486e470
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8166
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
6f6ddccc92 refactor(tvix/store): move entrypoint to src/bin/tvix-store.rs
Change-Id: Ibb83be75a2be27debd9e85b43c1b824f59e54dab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8165
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
b4a89461f5 feat(tvix/store): use Memory*Service instead of temporary sled
Change-Id: I65d2030aa643b6937dd92baa85db033bdce38b02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8208
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
a4f6c4181a feat(tvix/store): add new_temporary for all Sled services
This provides a service using /dev/shm, that's deleted once the
reference is dropped.

Refactor all tests to use these, which allows getting rid of most
TempDir usage in the tests.

The only place where we still use TempDir is in the importer tests,
which work on a filesystem path.

Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
535e1b15ab fix(tvix/store/proto/grpc_blobservice_wrapper): buffer recv data
While we don't want to keep all of the data in memory, we want to
feed a reasonably-enough buffer to the chunking function, to prevent
unnecessarily trying to chunk over and over again.

Change-Id: I5bbe2d55e8c1c63f8f7ce343889d374b528b559e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8160
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
b29d1ae372 feat(tvix/store): add import::import_path
This imports the contents at a given Path into the tvix store.

It doesn't register the contents at a Path in the store itself, that's up
to the PathInfoService.

Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
52a5181eba refactor(tvix/store/tests/nar_renderer): only upload EMPTY_BLOB
This also uploaded HELLOWORLD_BLOB_CONTENTS before, but it's not
referred from anywhere in the fixture.

Change-Id: I823133afe0f08d18a59e2ac4e4d4bb7d34ce8a2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8158
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
092f8a7df8 refactor(tvix/store/tests): move nar contents to fixtures
Change-Id: I89a700240e7250a4d93eb8c4235e79c29ecb7f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8157
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
e7279b6063 refactor(tvix/store/tests): move fixtures into separate module
Change-Id: I362dbf0899e4dc42114fd2e6a8fa7f537e9ea138
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8156
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
28a862976b refactor(tvix/store/tests): move gen_*_service() into helper
This allows hiding to tests what exact implementation we're using, when
testing things that do something with a store, but don't care what's
used for underlying storage.

Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
0baaabc43e refactor(tvix/store): move blob splitting into a BlobWriter struct
This will moves the chunking-as-we-receive logic that so far only lived
in grpc_blobservice_wrapper.rs into a generic BlobWriter.

Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
d8ab140d25 feat(tvix/store): do not buffer blob data
Use the FastCDC::cut function to ask fastcd for cutting points as we
receive the data. Make sure to keep the last chunk in the temporary
buffer, as we might not actually cut at the end.

Also, use rayon to calculate the blake3 hash if the input data is
> 128KiB.

Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
a40d2dcdcd feat(tvix/store): bump fastcdc, use v2020 version
This switches away from the less canonical "ronomon" version to the
implementation as described in the
[paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et
al., in 2020.

That version uses 64-bit hash values and tends to be faster than both
the ronomon and v2016 versions, and produces the same chunking as the
2016 version.

As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's
the recommended choice.

The crate also gained support for streaming version of chunkers:
https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be
useful.

Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
fbabcf0420 feat(tvix/store): add --json arg
This configures logging as JSON.

Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
7fe7e03df3 chore(tvix/store): remove old implementations
These were implementing against the (more complicated) gRPC interface,
for which we now have a wrapper.

Change-Id: I0a8284493718ab99618a1d21a76df4d173edb899
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8100
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
6b91efa5cf refactor(tvix/store): use GRPC Wrappers and new services
This swaps out the implementation used in the main entrypoint.
We now use the non-gRPC aware Sled*Service, and wrap it with
GRPC*ServiceWrapper to spin up the gRPC server.

Change-Id: I5a1236c0612eee85e49891062040127c8bd95058
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8106
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
cd22a8aa6d feat(tvix/store): add tests for GRPCPathInfoServiceWrapper
Change-Id: I22de33c28c132fc9710ddecd0e9158ee3b3ef94b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8108
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
dd1ee7a2e7 feat(tvix/store): add GRPCPathInfoServiceWrapper
This exposes a proto::pathinfo_service_server::PathInfoService for a
directoryservice::PathInfoService and a way to calculate NARs.

Change-Id: I30cd058562f83d063c78b84976ec97190de49400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8097
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
9b3228959a feat(tvix/store): add pathinfoservice
This adds a PathInfoService trait, and an implementation for it using
sled, and one using a HashMap.

Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
419f7d0f0c feat(tvix/store): add NARCalculationService trait and impl
This adds a NARCalculationService trait, which will take a root node,
and provide a proto::CalculateNarResponse in return.

It also adds a NonCachingNARCalculationService implementation, that will
simply always render the NAR in memory to calculate the size and sha256,
without any caching.

Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
ab02fc668c feat(tvix/store): validate blob size in NARRenderer
Make sure the blob size in the current proto node matches what we get
back from the blob backend.

Change-Id: I939fa18f37c7bc86ada8a495c7be622e69ec47f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8129
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
bc3f71838f refactor(tvix/store): don't use anyhow::Result<()> in tests
It's much easier to just unwrap the TempDir::new() result.

Change-Id: I7b05bc18f3146401e30e1cc2bb412503c5171a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8128
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
df3223fd68 chore(tvix/store): move NAR rendering logic into Renderer struct
This moves the logic rendering NARs to a struct using the
previously introduced, more granular BlobService, ChunkService and
DirectoryService.

Instead of passing them around to the helper functions, they're kept as
members of a struct.

Remove the async invocations in the nar_renderer tests, there's nothing
async in here.

Change-Id: Ic6d24aaad68a1fda46ce29f2cdb5f7b87f481d5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8095
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
cdb9458310 feat(tvix/store): add tests for GRPCDirectoryServiceWrapper
Change-Id: Iccf70664cd5dd8ea22e0b43002d2ff1a9680870a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8102
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
1540c7e276 feat(tvix/store): add GRPCDirectoryServiceWrapper
This exposes a proto::directory_service_server::DirectoryService for a
directoryservice::DirectoryService.

Change-Id: I191a6e44e5bce10d904b5f10f0073d6adaf1f572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8094
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
3af467d7ee feat(tvix/store): add directoryservice
This adds a DirectoryService trait, and an implementation for it using
sled, and one using a HashMap.

Change-Id: Ida61524b2ca949e1b3a78089a5aa7d9f9800c8d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8093
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
35ea0b0d2e feat(tvix/store): add tests for GRPCBlobServiceWrapper
Change-Id: I0c9a2d887b5b94c43afee97a8a1e701cc9f74f23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8098
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
69516f1f68 feat(tvix/store/blobservice): add GRPCBlobServiceWrapper
This takes a BlobService and ChunkService in the constructor, and
provides a [proto::blob_service_server::BlobService] trait for it.

Implementing proto::blob_service_server::BlobService is a lot of surface
to cover, and providing this wrapper will make individual
implementations taking care of how to store chunks or chunking
information much simpler.

Change-Id: Ia7b46484fb3ac9104354d496ff2922dca96ff7b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8092
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
bdf0725def refactor(tvix/store): remove anyhow::Result in directory it test
There's no `?` being used.

Change-Id: Ifbfd88f6056e0064d11458cb335d05b333ec7b6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8127
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
95aec51b2c refactor(tvix/store): remove useless clone in test
Change-Id: If62887cbc3338110cb7c815ec2ee2160fae04353
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8111
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
80f68bf828 chore(tvix/store): move protos into separate mod.rs
This allows adding more stuff into this namespace, from different files.

Also move tests on proto-related code from src/tests to src/proto/tests.

Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00
Florian Klink
60abca1d8e feat(tvix/store): add blobreader
A BlobReader can be used to read a blob, which might consist out of
multiple chunks. Chunks are fetched from a ChunkService.

Change-Id: I1806225b0052adaa4a2320b79b744e554e524ee3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8088
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 10:58:55 +00:00
Vincent Ambo
2696839770 feat(tvix/eval): introduce generators module
This module contains the request/response types for generators
requesting actions from the VM.

For most of these, an async helper function is added that will be used
inside of generator functions to make use of these requests/responses
instead of constructing them directly.

Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-07 22:04:59 +00:00
Vincent Ambo
9cebae9b56 refactor(tvix/eval): merge OpCall & OpTailCall
As applies are thunked, there was no situation where OpCall could be
emitted. In practice, all calls were already tail calls.

Change-Id: Id0d441dcdd86f804d7cddd0cc14f589bbfc75e5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8147
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-07 22:04:59 +00:00
Florian Klink
b74ffda583 feat(tvix/store): add blobservice
This adds a BlobService trait, and an implementation for it using sled,
and one using a HashMap.

Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 15:32:47 +00:00
Florian Klink
119aa43171 feat(tvix/store): add chunkservice
This adds the simpler ChunkService trait, and an implementation for it
using sled, and one using a HashMap.

Change-Id: Icb0fdc41b37b44e9e9e4f548d0f4acae1d83b71e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8086
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-06 15:32:47 +00:00
Florian Klink
3f27fe3484 feat(tvix/store): impl From<PoisonError> for Error
Change-Id: Ib61e276b45c0102e383a7e7e641172b151369b03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8207
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 15:32:47 +00:00
Florian Klink
786854713c feat(tvix/store): add errors
Change-Id: I7be0a4eaa29840e1a3329bccb791975de22828ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8085
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-06 15:32:47 +00:00
Florian Klink
6019c75deb docs(tvix/store): add document describing why we don't use git trees
This came up recently again, it makes sense to document the reasoning
behind the decision.

Change-Id: Ic51d5bc7998c70e8b070b6f42877d8e88613935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8223
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-05 18:33:54 +00:00
Florian Klink
db8cebe0b4 docs(tvix/store): move link hrefs to bottom
Change-Id: Id8e32ee4de98fbe48266d417279194cd93e968cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8222
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-05 14:42:05 +00:00
Vincent Ambo
bfb787a6c5 refactor(tvix/eval): remove VM argument from suspended native thunks
Because they do not use it, and it can not be passed with the coming
generator refactoring.

Change-Id: I0d96f2357a7ee79cd8a0f401583d4286230d4a6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8146
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-04 15:53:36 +00:00
Vincent Ambo
38fd3cb292 refactor(tvix/eval): insert storeDir "builtin" in eval startup
Instead of using a suspended native thunk, calculate and optionally
insert the storeDir builtin when the VM is constructed.

We already have the IO handle available at this point and can just
check whether a storeDir is present, and insert its absolute value as
a builtin.

Change-Id: If966eee6ff26dc888b6e888e7c46170c0c346b05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8145
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-04 15:53:36 +00:00
Vincent Ambo
e1f082a3ab feat(tvix/eval): add SharedThunkSet
This is a ThunkSet wrapped to be shareable, which will be required
once ThunkSets are embedded in futures.

Change-Id: I5a067b7972ac86e4d354c75ef05c86b2284c1137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8144
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-04 15:18:37 +00:00
Vincent Ambo
dccbda5960 fix(tvix/eval): ThunkSet does not need mutable pointers
Change-Id: Iea248870a0ea5d38cb02ff059c968fbd563570b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8143
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-04 15:18:37 +00:00
Vincent Ambo
8c7d5b4f87 chore(tvix/eval): implement From<OrdMap<..>> for NixAttrs
Again simplifying some code down the line, where bits of code that
construct attribute sets already have the final structure available.

Change-Id: I0bb7a1daa63298122b51be73d35d695a4f73f8b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8140
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-04 15:18:37 +00:00
Vincent Ambo
a11a3e2c59 refactor(tvix/eval): implement From<Span> for LightSpan
This simplifies some code down the line.

Change-Id: I58dd71e796e11479f44516cf24932f8061843d23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8139
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-04 15:18:37 +00:00
Florian Klink
1f9b582239 feat(tvix/nix-compat/derivation): make use of NixPath in some places
Some of these strings are actually just the nix hash representation of
the hash calculated earlier.

There's another one passed around via calculate_drv_replacement_str, but
that's left for a followup.

Change-Id: Id99a2a926a980d679eb49c34ee6a36bf224699b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8218
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-04 12:26:00 +00:00
Florian Klink
c2a681eaff feat(tvix/nix-compat): add NixHash::{new, to_nix_hash_string}
This provides a way to construct a NixHash struct without parsing
strings.

Change-Id: I947d96e15e51e72d5b02929cda8c5fc31d81253a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8217
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-04 12:26:00 +00:00
Florian Klink
67c9e2c770 feat(tvix/nix-compat): introduce parsing for Nix Hash strings
We ironically didn't add support parsing for the "native" format that
Nix uses under the hood.

This extends the from_str method to peek at the prefix of the string to
determine whether to try decoding as SRI, Nix string, or whether it
should be a bare digest.

Change-Id: I33efd24968b16f86eff18305b4ca8f112c7131d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-04 12:26:00 +00:00
Florian Klink
4cc4b2a814 refactor(tvix/nix-compat): move digest decoding into helper function
This will be used for both Nix hash strings and hash strings without the
algo specified.

Change-Id: Iedfe5494fba5f2be00614ba0fc38bf659eafd447
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8215
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-04 12:26:00 +00:00
Florian Klink
daba4cc0e5 refactor(tvix/nix-compat): address clippy in nixbase32.rs tests
Change-Id: If8820cba4cf19bf0f7aa27e0d93b70eb7b4ee81f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8221
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-04 12:21:28 +00:00
Florian Klink
93b0766485 refactor(tvix/nix-compat): address clippy in derivation/tests/mod.rs
Change-Id: I3f3f7cb590c900abf8b39533ed73fe9135d58f0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8220
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-04 12:19:24 +00:00
Florian Klink
f9dcf54b04 refactor(tvix/nix-compat): address clippy in store_path.rs
Change-Id: Ib10bd93bbb23696d7048e7ed8e405953db94693f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8219
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-04 12:19:24 +00:00
Vincent Ambo
248f1c2151 chore(tvix/nix-compat): include invalid hashes in SRI hash error
Change-Id: I379b18cb07da0988f8ce4934976ae7d6566d6bb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8210
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-03-03 23:35:47 +00:00
Vincent Ambo
344c119370 chore(tvix/eval): fix clippy warnings
Change-Id: I4c02f0104c455ac00a3f299c1fbf75cbb08e8972
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8142
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-03-03 11:01:52 +00:00
Vincent Ambo
7e9a65dcdb refactor(tvix/eval): remove useless map call
Change-Id: Ifb59ef148ea4fab613f2e4efb133c04baafa3a98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8141
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-03-03 10:57:50 +00:00
Vincent Ambo
fb4c197b39 refactor(tvix/eval): enhance debug output for bytecode dumps
This adds addresses of thunk and closure chunks to the debug output
displayed when dumping bytecode.

This makes it possible to see in the dump which thunks are referenced
by constants in other thunks.

Change-Id: I2c98de5227e7cb415666cd3134c947a56979dc80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8137
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-03 10:52:48 +00:00
Aaqa Ishtyaq
0dcbf5dfb9 chore(tvix/cli): iter instead of into_iter for references
This CL removes calling into_iter on a reference, as it will
not move out it's content into resulting iterator.

Change-Id: Ifcc10b7cf33b98453570cbcec3eb82ffaba2ffcb
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8126
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-16 22:42:14 +00:00
Aaqa Ishtyaq
faffb2a4cb refactor(tvix/eval): remove redundant clone
This CL removes redundant clone from value which is
going to be dropped without further use.

Change-Id: Ibd2a724853c5cfbf8ca40bf0b3adf0fab89b9be5
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8125
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-16 22:42:14 +00:00
Vincent Ambo
bfe6cace5e docs(tvix/eval): add proposal for VM loop restructuring
Change-Id: Ib991d68724a73886a8343d7f785b5b3aedd637ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8103
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2023-02-16 22:23:51 +00:00
Florian Klink
f2624f1028 refactor(tvix/store): remove needless borrow in build.rs
Change-Id: I2c2a70e5ffbb080702f57218aa9dcfe53d08870f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8110
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-15 18:40:36 +00:00
Florian Klink
cb8466d417 refactor(tvix/store/proto): use .cloned()
Instead of using an explicit closure to clone elements, use .cloned().

Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-15 18:40:34 +00:00
Florian Klink
cfa0755fc4 refactor(tvix/store): avoid casting to the same type
The size field already is u32, we don't need to convert here.

Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-15 18:40:33 +00:00
Vincent Ambo
3c311ffab2 fix(tvix/eval): correctly print lambda address in observer
We want the address that the Rc is pointing to, not the address of the
Rc.

Change-Id: I8eba21677f242bbe4166c74d4aa4269c316076e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8045
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-14 08:02:40 +00:00
Aaqa Ishtyaq
fc61dff037 chore(tvix/eval): use writeln for newline string
This CL address clippy warning which expects to
use `writeln` instead of `write` for strings with
new line.

Change-Id: Ia72a07502c60cfd489ecf1e3833b9d42d44a8b17
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8030
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-13 19:45:58 +00:00
Aaqa Ishtyaq
105069302b chore(tvix/eval): clippy warn is length zero
This CL address clippy warning about finding the zero
length of something using `is_empty()` instead of `len() == 0`.

Change-Id: I2b36c7c7b65b733609fc0dcd33be06f9d772bc9b
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8029
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-13 19:45:58 +00:00
Vincent Ambo
c98c5399b9 fix(tvix/eval): skip runtime completely on compiler errors
This branch was missing, and an assumption elsewhere just executed the
returned (broken) bytecode.

This fixes b/253.

Change-Id: I015023ba921bc08ea03882167f1f560feca25e50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8090
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-13 16:21:47 +00:00
Vincent Ambo
91a366af46 fix(tvix/eval): make fields of eval's Error type public
These should be inspectable by callers.

Change-Id: Ia9ef871aa63958d06066aaea61b2aecbd217369b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8089
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-13 16:21:46 +00:00
Florian Klink
d4cbdc2beb feat(tvix/nix-compat/nixhash): support BASE64_NOPAD SRI
Nix accepts SRI hashes that are missing their padding characters
in base64, as seen in
7e49471316/pkgs/development/libraries/kerberos/krb5.nix .

It only seems to work in the SRI case, not with `sha256` being set to a
(nopad) base64 string.

Add regression tests for this, and document why we don't want to support
*additional* characters afterwards.

Reported in https://b.tvl.fyi/issues/252

Change-Id: I9ffc2b417501b426ced1894a9cbf95ff5f0e5159
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8037
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
2023-02-08 10:39:26 +00:00
Florian Klink
5fa35d9d49 refactor(tvix/nix-compat/nixhash): fix test function name
Change-Id: I3b089758ce8c01df544064422e56a97a8402513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8036
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2023-02-08 10:39:26 +00:00
Alyssa Ross
5da1f4161c fix(tvix): fix InvalidEncodedDigestLength message
Before:
	error[E997]: invalid output hash: invalid encoded digest length '43' for algo 43

After: error[E997]: invalid output hash: invalid encoded digest length '43' for algo sha256

Change-Id: Icb533839586b47f90b4e8d22083cce397f9f3655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8035
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-07 17:40:54 +00:00
Florian Klink
87e1934f2d docs(store/protos): update comment on blobstore Read and Put
Further emphasize Read() can be used to ask for blobs OR chunks, and
that clients usually want to stat and then request (smaller) chunks,
rather than reading whole blobs.

Also clarify that the chunking used to send BlobChunks over has nothing
to do with the chunk sizes communicated in a Stat() request.

Change-Id: Ia615d190aae570611de2655b11342a14d0b75976
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-07 14:14:35 +00:00
Vincent Ambo
720e80c1f9 fix(tvix/eval): fix the default case for path parsing
Plain paths like `foo/bar.nix` are also allowed, so we can not
determine this based on the prefix.

The upstream PR that is referenced in a comment here has a
significantly different interface than we expected, so I'm not
touching that comment yet in this CL before I've had more time to
digest it.

Change-Id: Iea33bbb35de9c00a7d7fedf64d02253c75c1cc9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8032
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-04 12:44:59 +00:00
Vincent Ambo
26b55f8cda fix(tvix/cli): use tvlfyi/wu-manber fork for refscanner
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1)
but it's not clear whether upstream will accept patches, so for now
lets point this directly at our fork.

Change-Id: Iccdcedae3e9a8b783241431787c952561d032694
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-04 12:44:59 +00:00
Florian Klink
d50044d570 chore(tvix/eval): only use Rc with impure feature
We only use Rc in `impl EvalIO for StdIO`, which is only included when
building with the "impure" feature.

Change-Id: Id29d647c899cbfcdda11abfb9fabd5aa7e24299f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8025
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-03 18:58:17 +00:00
Vincent Ambo
f16b0f24e2 refactor(tvix/eval): wrap Builtin type in a Box
This reduces the size of `Builtin` from 88 (!) bytes to 8, and as the
largest variant of `Value`, the size of that type from 96 to 64.

The next largest type is NixList, clocking in at 64 bytes.

This has noticeable performance impact. In an implementation without
disk I/O, evaluating nixpkgs.stdenv looks like this:

Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath'
  Time (mean ± σ):      1.151 s ±  0.003 s    [User: 1.041 s, System: 0.109 s]
  Range (min … max):    1.147 s …  1.155 s    10 runs

After this change, it looks like this:

Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath'
  Time (mean ± σ):      1.046 s ±  0.004 s    [User: 0.954 s, System: 0.092 s]
  Range (min … max):    1.041 s …  1.053 s    10 runs

Change-Id: I5ab7cc02a9a450c0227daf1f1f72966358311ebb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8027
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 18:47:33 +00:00
Vincent Ambo
32698766ef refactor(tvix/eval): statically resolve select from constant attrs
When resolving a select expression (`attrs.name` or `attrs.name or
default`), if the set compiles to a constant attribute set (as is most
notably the case with `builtins`) we can backtrack and replace that
attribute set directly with the compiled value.

For something like `builtins.length`, this will directly emit an
`OpConstant` that leaves the `length` builtin on the stack.

Change-Id: I639654e065a06e8cfcbcacb528c6da7ec9e513ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 18:47:33 +00:00
Florian Klink
f2afd38f2d docs(tvix): move mg shell command into one line
This was reflowed in a funny way. Move the whole command into its own
line, to prevent it from happening.

Change-Id: Ifba4daf418487ca4c32586820071930d29020f42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8026
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-03 12:07:07 +00:00
Vincent Ambo
0dd07440b4 test(tvix/cli): eval nixpkgs.stdenv in CI and assert that it matches
Change-Id: If80194b5fdbf69512217bd4780e416e678045323
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8023
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-03 10:47:18 +00:00
Vincent Ambo
3419f63575 fix(tvix/eval): ensure all evaluated thunks are correctly memoized
This fixes a very complicated bug (b/246). Evaluation
progresses *much* further after this, leading to several less
complicated bugs likely being uncovered by this

What was the problem?
=====================

Previously, when evaluating a thunk, we had a code path that looked
like this:

    match *thunk {
        ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => {
            let inner_repr = inner_thunk.0.borrow().clone();
            drop(thunk);
            self.0.replace(inner_repr);
        }
        /* ... */
    }

This code path created a copy of the inner `ThunkRepr` of a nested
thunk, and moved that copy into the `ThunkRepr` of the parent.

The effect of this was that the original `ThunkRepr` (unforced!) lived
on in the original thunk, without the memoization of the subsequent
forcing applying to it.

This had the result that Tvix would repeatedly evaluate these thunks
without ever memoizing them, if they occured repeatedly as shared
inner thunks. Most notably, this would *always* occur when
builtins.import was used.

What's the solution?
====================

I have completely rewritten `Thunk::force_trampoline_self` to make all
flows that can occur in it explicit. I have also removed the outer
loop inside of that function, and resorted to more use of trampolining
instead.

The function is now well-commented and it should be possible to read
it from top-to-bottom and get a general sense of what is going on,
though the trampolining itself (which is implemented in the VM) needs
to be at least partially understood for this.

What's the new problem(s)?
==========================

One new (known) problem is that we have to construct `Error` instances
in all error types here, but we do not have spans available in some
thunk-related situations. Due to b/238 we cannot ask the VM for an
arbitrary span from the callsite leading to the force. This means that
there are now code paths where, under certain conditions, causing an
evaluation error during thunk forcing will panic.

To fix this we will need to investigate and fix b/238, and/or add a
span tracking mechanism to thunks themselves.

What other impacts does this have?
==================================

With this commit, eval of nixpkgs mostly succeeds (things like stdenv
evaluate to the same hashes for us and C++ Nix, meaning we now
construct identical derivations without eval breaking).

Due to this we progress much further into nixpkgs, which lets us
uncover more additional bugs. For example, after this commit we can
quickly see that cl/7949 introduces some kind of behavioural issue and
should not be merged as-is (this was not apparent before).

Additionally, tvix-eval is now seemingly very fast. When doing
performance analysis of a nixpkgs eval, we now mostly see the code
path for shelling out to C++ Nix to add things to the store in there.
We still need those code paths, so we can not (yet) do a performance
analysis beyond that.

Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 10:47:18 +00:00
Vincent Ambo
38e8c2e959 fix(tvix/cli): keep tracking full paths in known_paths
We need to distinguish explicitly between the paths used for the
scanner, and the paths that populate the derivation inputs. The full
paths must be accessible from the result of the refscanner to populate
drv fields correctly.

This was previously hidden by debug changes that masked actual IO
operations with no-ops.

Change-Id: I037af6e6bbe2b573034d695f8779bee1b56bc125
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8022
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-02 23:37:34 +00:00
Vincent Ambo
e6235e2932 feat(tvix/cli): cache imported paths in NixCompatIO
Creates a cache of imported literal files (e.g.
`./default-builder.sh`) which avoids shelling out to Nix for each
instance of the same file.

Note that a better way to tackle this is to create memoizable thunks
for these expressions in the compiler, but we are lacking a little bit
of infrastructure for that at the moment.

Change-Id: Ibc062b20d81e97dd3986e734d225a744e1779fe7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8015
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 17:50:44 +00:00
Vincent Ambo
9d6f29a72b refactor(tvix/cli): use Wu-Manber string scanning for drv references
Switch out the string-scanning algorithm used in the reference scanner.

The construction of aho-corasick automata made up the vast majority of
runtime when evaluating nixpkgs previously. While the actual scanning
with a constructed automaton is relatively fast, we almost never scan
for the same set of strings twice and the cost is not worth it.

An algorithm that better matches our needs is the Wu-Manber multiple
string match algorithm, which works efficiently on *long* and *random*
strings of the *same length*, which describes store paths (up to their
hash component).

This switches the refscanner crate to a Rust implementation[0][1] of
this algorithm.

This has several implications:

1. This crate does not provide a way to scan streams. I'm not sure if
   this is an inherent problem with the algorithm (probably not, but
   it would need buffering). Either way, related functions and
   tests (which were actually unused) have been removed.

2. All strings need to be of the same length. For this reason, we
   truncate the known paths after their hash part (they are still
   unique, of course).

3. Passing an empty set of matches, or a match that is shorter than
   the length of a store path, causes the crate to panic. We safeguard
   against this by completely skipping the refscanning if there are no
   known paths (i.e. when evaluating the first derivation of an eval),
   and by bailing out of scanning a string that is shorter than a
   store path.

On the upside, this reduces overall runtime to less 1/5 of what it was
before when evaluating `pkgs.stdenv.drvPath`.

[0]: Frankly, it's a random, research-grade MIT-licensed
     crate that I found on Github:

     https://github.com/jneem/wu-manber

[1]: We probably want to rewrite or at least fork the above crate, and
     add things like a three-byte wide scanner. Evaluating large
     portions of nixpkgs can easily lead to more than 65k derivations
     being scanned for.

Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 17:50:44 +00:00
Vincent Ambo
2c07ff0f8c docs(tvix): add more information to README
The README was very sparse before and we've actually had people email
us (as it says to contact us) just to ask what Tvix *is*. This should
answer some questions!

Change-Id: I0f248cb060eccfe086468afed1d648652b35dfd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8018
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 16:25:52 +00:00
Vincent Ambo
503e6810e7 fix(tvix/eval): unsafeDiscardStringContext is a no-op
... not just a TODO.

Most use-cases of unsafeDiscardStringContext are for cases where a
string is processed in some ways and no longer contains a "physical"
reference, but still has its context attached in C++ Nix.

We don't need to do this. This does diverge in behaviour in use-cases
related to build scheduling, but that whole behaviour will be
different in Tvix.

Change-Id: I4056d4c09f62d44d6bd52b791db03fe5556672b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8016
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-02 14:30:32 +00:00
Vincent Ambo
75d1b72b7c refactor(tvix/eval): import_cache can be a HashMap
... instead of a BTreeMap, as we do not need ordering guarantees here.
HashMaps are noticeably faster here (especially as we've been sorting
essentially random data!).

Change-Id: Ie92d74286df9f763c04c9b226ef1066ee8484c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8014
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 14:25:30 +00:00
Vincent Ambo
98ea614c6e chore(tvix/eval): elaborate on internal types in Value::type_of
This aids in debugging quite substantially.

Change-Id: Ic43232aa6165ae1c3db7ac2701938e1dfeeb418c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8013
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 09:29:43 +00:00
Florian Klink
6abd1e2dae chore(tvix): update Cargo.lock
Change-Id: I08b2dc2393819c4f0d3871fe05bdbd29e89a6d75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8010
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 23:06:02 +00:00
Vincent Ambo
dc1e86001c fix(tvix): add dummy target to attach extra-step to
Change-Id: I594a6652e2efe7aa6e35c7cdd84fc3097660614f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8009
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-01 17:34:08 +00:00
Vincent Ambo
87de3fff52 chore(tvix): mirror exported repository
Pushes a mirror of tvix to github/tvlfyi/tvix

Change-Id: I86e7c1b1dc1fcc1012fa72234588b26ac2554056
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8007
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 17:21:05 +00:00
Florian Klink
5ac33a227f docs(tvix): update README to document usage inside tvix view
If the tvix view is cloned through josh, you don't use mg, but a
`shell.nix` is provided.

Also, add the `git clone` command, so people browsing tvix source code
in the browser know where to clone from.

Change-Id: I18483d6a52953f9f4eafd1533ea69afb0e329b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Florian Klink
f5ccb65f26 feat(tvix/nix-compat/derivation): Display -> to_aterm_string()
Instead of implementing `std::fmt::Display for Derivation` and relying
on the `to_string` method, introduce a `to_aterm_string()` method, which
does the same thing, but makes it clearer what we're producing, rather
than just calling `to_string()``.

Change-Id: I21823de9096a0f2c2eb6f4591e48c1aa9fd94161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7998
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 16:31:56 +00:00
Florian Klink
99d5cf822a refactor(tvix/cli): use nixhash module for output hash calculation
This covers all the weird corner cases.

Change-Id: I85637e82e8929828064ab562dc8a1c8bf161fffa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7991
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 15:29:48 +00:00
Florian Klink
265dd36b98 feat(tvix/nix-compat): add nixhash module
This module takes care of parsing various hashes and algorithms.

It will get used to modify derivation output hashes in the next CL.

Change-Id: Idc07c401dbb7510f49883ac02b8379b9a5d930c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7990
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 15:29:48 +00:00
Florian Klink
366d2c5e92 refactor(tvix/nix-compat/derivation): remove module inception
The module `src/derivation/derivation.rs` is a sign of module inception.

Move the Derivation struct definiton up into `src/derivation/mod.rs`,
and some of the helpers in a `util.rs`.

Change-Id: Ib24a5f8a27bdd45df8b1fa2b3482a79b33cab8d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7997
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:43:25 +00:00
Florian Klink
2551adb4f1 refactor(tvix/nix-compat): operator precedence can trip the unwary
warning: operator precedence can trip the unwary
  --> nix-compat/src/nixbase32.rs:41:23
   |
41 |                 c |= ((input[i + 1] as u16) << 8 - j as u16) as u8
   |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: consider parenthesizing your expression: `(input[i + 1] as u16) << (8 - j as u16)`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#precedence
   = note: `#[warn(clippy::precedence)]` on by default

Change-Id: I091071d649abf4ed38f5f4e39a0c5d21a0459bff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7996
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:43:24 +00:00
Florian Klink
361ffd7903 refactor(tvix/derivation): don't create deref'd immediately ref
clippy says:

> This expression creates a reference which is immediately dereferenced
> by the compiler

Change-Id: Ic2c093b043ebee9ae80912075083107e4d216cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7995
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:21 +00:00
Florian Klink
ce4d87b129 refactor(tvix/nix-compat): remove unneeded return
Change-Id: I97b9fb3ad33d2f51baf29486d4eff11f8dedcbab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7994
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:38:19 +00:00
Florian Klink
894d17b29b refactor(tvix/nix-compat): const is always 'static
Change-Id: If33bbf47a6cc376f2c0935ea4331a819223b2f00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7993
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:18 +00:00
Vincent Ambo
759f9dbf39 feat(tvix/cli): implement builtins.placeholder
This doesn't require any other corresponding handling *yet*, as the
actual replacements happen in the builder logic (which we delegate to
cppnix at the moment).

Change-Id: I034147c933f05ae427c7a8794647132d108d0ede
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7972
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-01 10:01:40 +00:00
Florian Klink
5f62cd1ad5 refactor(tvix/serde): allow dead_code in struct
This is an example, which uses the debug trait to print all field
values.

Silence the compiler warning about unused fields.

Change-Id: I5f1216c77819003302e83ba1af1ff13c924f3b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7971
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:35:46 +00:00
Florian Klink
a7e20bbef3 docs(tvix/nix-compat/store_path): fix docstrings
Change-Id: I19261233e338b406a87fe680675eb4dfd853fe11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7970
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:35:45 +00:00
Vincent Ambo
ddf6e6cf47 fix(tvix/eval): allow builtins.toXML to serialise any function
This adds a fake argument name to builtins.toXML which allows toXML to
serialise any value instead of panicking on functions. We do still
have to fix the value itself, eventually, though.

Change-Id: I2e330ecddcd80442b4fac5eced64431ac86123ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 15:31:30 +00:00
Florian Klink
0db73cb2bd feat(tvix/store): add write_nar function
This adds a function that consumes a [proto::node::Node] pointing to the
root of a (store) path, and writes the contents in NAR serialization to
the passed [std::io::Write].

We need this in various places:

 - tvix-store's calculate_nar() RPC method needs to render a NAR stream
   to get the nar hash, which is necessary to give things imported in
   the store a "NAR-based" store path.

 - communication with (remote) Nix (via daemon protocol) needs a NAR
   representation.

 - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a
   NAR representation.

Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:28:22 +00:00
Florian Klink
a23b7e17c0 refactor(tvix/nix-compat): absorb nar writer
Expose it at `nix_compat::nar::writer`.

Change-Id: I98a8acfa6277074f24608d64fb6c0082b41d10f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7969
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:18:39 +00:00
Florian Klink
1e28cc7ede docs(tvix/nix-compat/derivation): fix docstrings
Change-Id: I3fd02e62c8a24fa7f27461b17cdd824b060d9bdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7968
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:40 +00:00
Florian Klink
2d24c5f260 refactor(tvix/nix-compat): absorb //tvix/derivation
Put this in its src/derivation.

Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:39 +00:00
Florian Klink
9e809e21cc feat(tvix/store): implement iteration over Directory nodes
This provides a `Directory.nodes()` function, returning an iterator over
all three node types in an ordered fashion.

Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
1ee6bd06e3 feat(tvix/store/proto): implement get_name for node::Node, pub trait
Make the trait public, so consumers can use it.

Also, implement it for node::Node, so we can later use this to access
the name from all three if we don't care about the enum type.

Change-Id: Iae530a16b705493645e61947852c03273876cc55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
c27bacd905 refactor(tvix): introduce nix-compat crate
Move nixbase32 and store_path into this.

This allows //tvix/cli to not pull in //tvix/store for now.

Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 13:26:18 +00:00
Florian Klink
8ea93bb646 refactor(tvix/cli/derivation): use if let to destructure
We only do logic here if algo and hash_mode are Some(_)
(and there's an `out` output).

The fact we don't do anything in all in other cases is a bit hidden at
the bottom. Use if let for the destructuring, and drop the other case.

Change-Id: Icc0e38e62947d52d48ef610f754749737977fca9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7966
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:26:11 +00:00
Florian Klink
6fa91349a9 refactor(tvix/cli): remove unneeded clone
Change-Id: I6f4cb24bdd636af8918a2ade44075af92161c97d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7965
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:26:11 +00:00
Vincent Ambo
4da1e64c03 feat(tvix/serde): add an example application to the project
This shows how people can use tvix_serde to deserialise configuration
structs for their programs from Nix code.

Change-Id: I71bf4e03dce19dddafe67dd729b4e4b10719a739
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7945
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-31 13:11:28 +00:00
Vincent Ambo
7702941dd9 test(tvix/eval): add tests for internal formals dependencies
Formals can depend on each other when using another formal as a
default value.

This test ensures that the compiler's declaration and initialisation
order of formals is consistent with what actually happens in the VM.

Change-Id: Ibdabe262554e8066d67fac1ebc3b5a48ef626e18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7948
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 10:25:02 +00:00
Florian Klink
4b3ccd205a refactor(tvix/cli): absorb construct_output_hash
This helper function only was created because
populate_output_configuration was hard to test before cl/7939.

With that out of the way, we can pull it in.

Change-Id: I64b36c0eb34343290a8d84a03b0d29392a821fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 08:52:00 +00:00
Vincent Ambo
e4bb750b3b refactor(tvix/cli): force outside of output configuration helper
Change-Id: I28357fe131cefedcef9761b08a72f675f4a10789
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7939
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 08:51:03 +00:00
Florian Klink
8ec8d03175 fix(tvix/store/nixbase32): fix encoder/decoder
Replace our data_encoding usage with the implementation taken from
https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32

Also uncomment some of the unit tests, and add a regression test for a
NIXBASE32.encode with a 32 bytes sequence.

The previous implementation of NIXBASE32.encode is wrong in that case:

```
❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA=
0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk
❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C
00000000  b3 a2 4d e9 7a 8f db c8  35 b9 83 31 69 50 10 30  |..M.z...5..1iP.0|
00000010  b8 97 70 31 bc b5 4b 3b  3a c1 37 40 f8 46 ab 30  |..p1..K;:.7@.F.0|
00000020
```

Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 11:46:30 +00:00
Florian Klink
85e563c554 refactor(tvix/store/tests): move Directory tests into directory.rs
There's gonna be more tests coming, let's move them into a separate file
everything Directory-related into a separate module.

Change-Id: I78a0f263925528907a22724b028f75cce644b329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-30 09:45:11 +00:00
Florian Klink
a2c33f517c feat(tvix/store/proto): implement get_name for all nodes
Also add a `NamedNode` trait. We'll later use this to access names from
all three individually.

Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 09:45:11 +00:00
Florian Klink
9c18888715 docs(tvix/store/protos): add docstring for Directory::digest()
Change-Id: I361dbca444a267fea28cd212d563ee2d03497c16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7952
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 09:45:11 +00:00
Florian Klink
42d3c06988 docs(tvix/store/protos): fix docstring
Change-Id: Idf835e9648ab6c95c29960f3f8176153d4888b27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7951
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-30 09:45:11 +00:00
Vincent Ambo
124af9e5d5 refactor(tvix/cli): add helper method for strong string coercion
This is repetitive and error prone (e.g. switching around
to_string/as_str has drastic consequences) due to the ToString
overloads.

Change-Id: I9b16a2e0e05e4c21e83f43e9f603746eb42e53f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7947
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-29 17:39:36 +00:00
Florian Klink
e3c2b3650a refactor(tvix/cli): describe errors with thiserror
This is much less code, and makes it much easier to read.

Change-Id: I9028f226105f905c2cc2cabd33907ff493e26225
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7938
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-27 14:08:15 +00:00
Florian Klink
a94a1434cc fix(tvix/cli): handle SRI hashes in outputHash
Instead of being called with `md5`, `sha1`, `sha256` or `sha512`,
`fetchurl.nix` (from corepkgs / `<nix`) can also be called with a `hash`
attribute, being an SRI hash.

In that case, `builtin.derivation` is called with `outputHashAlgo` being
an empty string, and `outputHash` being an SRI hash string.

In other cases, an SRI hash is passed as outputHash, but outputHashAlgo
is set too.

Nix does modify these values in (single, fixed) output specification it
serializes to ATerm, but keeps it unharmed in `env`.

Move this into a construct_output_hash helper function, that can be
tested better in isolation.

Change-Id: Id9d716a119664c44ea7747540399966752e20187
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7933
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 14:06:13 +00:00
Florian Klink
bda5fc58d0 feat(tvix/derivation): derive PartialEq for error
Allows easier to use these errors in test cases.

Change-Id: I8abee3b522909ed4aa4b066499cc48eececbc036
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7937
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-27 14:06:12 +00:00
Vincent Ambo
f22b9cb0d7 feat(tvix/cli): faux implementation of builtins.toFile
This adds an implementation of this builtin which correctly calculates
paths, but does not actually write anything to the store or verify
references.

Change-Id: Ie9764cbc1d13a73d8dc9350910304e2b7cad3fe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7910
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
de10a924f2 feat(tvix/cli): implement builtins.derivation
This uses the actual upstream Nix code for
`builtins.derivation` (which is not a primop in C++ Nix) to implement
`builtins.derivation` as a wrapper around `builtins.derivationStrict`.

We're doing it this way to ensure that our thunking logic is correct.
An initial Rust-native rewrite (see e.g. cl/7363) is pretty difficult
to debug while there are still other issues to root out, but
eventually we might want to turn this into native code.

Change-Id: I5845e18073e103b8670e40648bd7fd9b511058e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7902
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
8a9aa018dc feat(tvix/cli): implement builtins.derivationStrict
Implements the logic for converting an evaluator value supplied as
arguments to builtins.derivationStrict into an actual,
fully-functional derivation struct.

This skips the implementation of structuredAttrs, which are left for a
subsequent commit.

Note: We will need to port some eval tests over to CLI to test this
correct, which will be done in a separate commit later on.

Change-Id: I0db69dcf12716180de0eb0b126e3da4683712966
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7756
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
3d7c371e22 feat(tvix/cli): add helper for handling special drv parameters
Adds a helper function which handles special parameters to
`builtins.derivation` that are not just blindly passed through to the
builder environment, but populate other specific fields of the
derivation (outside of the ones handled by other, more complex helpers
from previous commits).

Change-Id: I82d1edf9af714fc4591e9071c0b83ece83be7eee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7901
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
dfc50c9ef5 feat(tvix/cli): add helper for populating drv output configuration
This threads through the fields that control whether a derivation is a
fixed-output derivation or not.

Change-Id: I49739de178fed9f258291174ca1a2c15a7cf5c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7900
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
fdca93d6ed feat(tvix/cli): add helper for populating derivation inputs
This adds a helper function which takes the output of the reference
scanner used on derivation inputs and populates the `input_sources`
and `input_derivations` field of the derivation accordingly.

Note that we have a divergence from C++ Nix here, as we do not
populate the entire FS closure of a literally referred derivation (and
our standing theory is that this is unnecessary for nixpkgs).

Change-Id: Id0f605dd8c0a82973c56605c2b8f478fc17777d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7899
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
c6811f0cea feat(tvix/cli): add helper for populating derivation outputs
Adds a small helper function which uses a Nix value supplied to
`builtins.derivation{Strict}` to populate the `outputs` field of the
`Derivation` struct.

Change-Id: Iccc7a4f293b3d913140aed576a573a8992241e46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7898
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
1028aff105 chore(tvix/eval): remove dead comment
Change-Id: Icedb7f272e5067569b8dbf1c2d8b0fdd352b8e12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7936
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-26 23:30:43 +00:00