Commit graph

531 commits

Author SHA1 Message Date
Florian Klink
357c4d4836 feat(tvix/store): add nixbase32 mod
This implements the nix-specific base32 encoding and decoding, exposing
a subset of the API that the data-encoding crate provides.

Nix uses a custom alphabet, no padding, and encodes bytes in reverse
order. The latter one is the reason we can't just use the data-encoding
crate directly.

Three odd corner case tests ported over from go-nix failed. We opened
b/235 to further investigate.

Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-12-30 13:31:46 +00:00
Florian Klink
5ba47a2bc3 feat(tvix/store): parameterise validate_node_name error
We'll be using validate_node_name in other places in a bit, where
returning a ValidateDirectoryError is not appropriate.

Accept a function mapping a string to error as a second argument, and
pass ValidateDirectoryError::InvalidName at the current call sites.

Change-Id: I45cbb0deb4763061ad912c6b18a112c727795a17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7682
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2022-12-29 21:55:45 +00:00
Florian Klink
e97a429e73 fix(tvix/store): fix typo
Change-Id: I351a2bd4e007443aef1b97f50aecffc095ac60b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7681
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 21:51:43 +00:00
Florian Klink
6ab8320f07 chore(tvix/store): address clippy warnings
The only warnings left are unused warnings, but that'll change once we
have a real implementation, and not just tests.

Change-Id: I28912281b5e66735be37e999cc8ef4b8b09028fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7669
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-29 12:07:57 +00:00
Florian Klink
b0879917d4 chore(tvix/store): trim Cargo.toml
lazy_static is only used in tests, and anyhow isn't used at all (yet).

This can be dropped.

Change-Id: Ic41ff3f9bb93cfa600c3485e85464f78a3976504
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7668
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-29 12:07:57 +00:00
Florian Klink
ff71366f41 chore(tvix/store): move tests into separate file
Move them from the bottom of src/proto.rs to its own src/tests/mod.rs.

Also drop the test_ prefix, this is not golang.

Change-Id: I2e0b6b9812264f3d9721c0766936f08157fadc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7667
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 12:07:57 +00:00
Florian Klink
c4ee942b1c feat(tvix/store/protos): rename Get to Read, add Stat method
Stat exposes metadata about a given blob,
such as more granular chunking, baos.
It implicitly allows checking for existence too, as asking this for a
non-existing Blob will return a Status::not_found grpc error.

The previous version returned a Status::not_found error on the Get
request too, but there was no chance to prevent the server from starting
to stream (except sending an immediate cancellation).

Being able to check whether something exists in a BlobStore helps to
prevent from uploading in first place.

The granular chunking bits are an optional optimization - if the
BlobStore implements no more granular chunking, the Stat response can
simply contain a single chunk.

Read returns a stream of BlobChunk, which is just a stream of bytes -
not necessarily using the chunking that's returned in the reply of a
Stat() call. It can be used to read blobs or chunks.

Change-Id: I4b6030ef184ace5484c84ca273b49d710433731d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7652
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
1c15154b83 feat(tvix/store): make blobstore stream chunks
This changes the RPC methods to return/consume a stream of chunks, instead of a
very big message containing the whole blob, to keep message sizes in manageable
sizes (less than 4MiB).

Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
d973c9772d chore(tvix/store/protos): add PathInfoService::CalculateNAR()
Expose the NAR calculation to a separate `CalculateNAR` method, which
responds with the NAR size and sha256 hash.

Contrary to what cl/7618 and cl/7620 initially did, don't add different
other request types.

In the CalculateNARResponse message, there's now some duplication in the
(optional) `narinfo` field of a PathInfo, but I'm not entirely sure if
we want to drop the fields from there yet.

Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
ce7be00994 chore(tvix/store/protos): drop non-sha256 NAR hashes
Nix actually doesn't support anything else than sha256 NAR hashes, so
there's little reason to communicate anything else in here.

Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I760370bd6cabd02028e001a74c454ef9296eb600
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7619
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 13:55:18 +00:00
Florian Klink
27bfc8cfcd chore(tvix/store/protos): move (Root)Node into separate message
This is gonna get used in another place in a second.

Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I347c11c8d24379628b7ed09d2c90670c576e686a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7617
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 13:55:18 +00:00
Florian Klink
7163d3ad37 feat(tvix/store): implement Directory::validate()
Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 10:58:50 +00:00
Florian Klink
0c7e545fd0 chore(tvix/store): use DUMMY_DIGEST, too
Missed from cl/7649.

Change-Id: Id2d382567f91cba9cdd792fe44bc0a83d040ec69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7656
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 10:58:50 +00:00
Florian Klink
f930c91086 chore(tvix/store): use Directory::default in some more places
Change-Id: Iebad5e0b38f9506933fa855e1bfb2611d27ff572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7655
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 10:58:50 +00:00
Florian Klink
35b18b2fdb chore(tvix/store): use lazy_static and dummy digests
While there's currently nothing in here checking the size of the digest,
we should use something that passes the to-be-introduced validate()
function.

Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 21:31:33 +00:00
Florian Klink
dfd9286f68 feat(tvix/store/protos): implement Directory.Validate()
Validate thecks the Directory message for invalid data, such as:
- violations of name restrictions
- invalid digest lengths
- not properly sorted lists
- duplicate names in the three lists

Change-Id: I8d43a13797793c64097e526ef3bd482c9606c87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7648
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 21:31:33 +00:00
Florian Klink
c3fb6d2218 feat(tvix/store/protos): implement Size() and Digest() for Directory
This adds Size() and Digest() functions for the golang version.

Change-Id: If71445a9bb26100bb4076ac4f5c96945b33919f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7325
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-27 21:31:33 +00:00
Florian Klink
a91e6334ff feat(txix/store/protos): init go.mod, generate .pb.go files
This allows importing the generated .pb.go files into other go projects.

I initially looked at buildGo.protos, but it doesn't work for multi-.proto
files, and actually having LSP support for the generated structs is nice, too.

Change-Id: Idbd448008010790a10a0ea42e4059dbb609eaf1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7322
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
2b9330911a feat(tvix/store/protos): add go_package option
Change-Id: I0898b8a0a78e704219da38e5acaabef1e640d4e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7321
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2022-12-26 11:38:52 +00:00
Florian Klink
e05e8bdd03 docs(tvix/store): add README, document services and store model
These are intended to help digest the protocol definitions for tvix-
store, and how they tie into the whole concept.

Co-Authored-By: Vincent Ambo <mail@tazj.in>
Change-Id: Ic1ba3ba41ef599209453f15d0ac2e07a6144bcca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7439
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
3676ce8d6d feat(tvix/store): use tonic-build for protos
While prost-build already exposes protobuf message types as structs, we
actually need tonic-build too, to be able to get traits for all the RPC
services defined in the proto files.

Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
2e04f68df8 feat(tvix/proto): add rpc_directory.proto
This provides an interface to retrieve and upload single Directory
messages, or a DAG of them.

Change-Id: Id9e7084bd697d843649a122da2c992a3e36d808c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7137
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 14:35:32 +00:00
Florian Klink
870cd533dd feat(tvix/proto): add rpc_pathinfo
This defines a service that can be used to upload and retrieve metadata
of nix paths.

Change-Id: Id86eb531ce4ae316adb15934b0d1386a14ba2132
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 14:35:32 +00:00
Florian Klink
5f9796cf94 feat(tvix/proto): add rpc_blobstore
This defines a service that can be used to get and put content-addressed
chunks of data.

Change-Id: I36cf2278ed1daf71848c04fdfd14450b2268c5de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-15 19:10:26 +00:00
Vincent Ambo
3aca3d3bba refactor(tvix): build Rust projects using crate2nix
Introduces granular dependency builds using crate2nix, bootstrapped
off the generated configuration from the newly introduced
workspace (see cl/7533).

This commit checks in the generated Cargo.nix file which can be
regenerated with a parameterless invocation of `crate2nix generate` in
`//tvix`. I tried generating this in IFD, but it turned out to be
harder than what seemed worthwhile for now.

In this setup, the various build targets for Rust projects end up
being attributes of the imported `Cargo.nix` file at the `tvix.crates`
attribute. These still lack configuration, however, which has been
fixed in the various `default.nix` files of individual projects.

Note that we (temporarily) lose the ability to build tvix-eval's
benchmarks in CI. I haven't figured out what magic incantation summons
them from the void again ...

The `eval-okay-readDir` tests from both test suites have been disabled
because they fail for unknown reasons when run in this new derivation.
Somebody will have to debug it!

Change-Id: I2014614ccb9c8951aedbd71df7966ca191a13695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7538
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-12-15 17:26:45 +00:00
Vincent Ambo
d9d627cdf0 refactor(tvix): share a Cargo.lock file between Rust projects
This relates to the (abandoned) cl/7256.

Introduces a Cargo workspace at //tvix that is primarily intended to
be used as a workaround for the annoying Nix+Rust tooling while having
a consistent set of dependencies.

This is driven in part by a desire to adopt crate2nix and get more
granular Nix builds for Tvix's Rust projects, and in part by a need to
split //tvix/eval into something providing the CLI (REPL etc.), and a
library providing eval, without significantly altering the structure
of build targets.

To accomplish this the workspace has been designed to allow projects
to remain independent build targets. I want to avoid lumping all the
projects together - something like //tvix/eval should always be
independent of other parts of tvix.

A helper function in //tvix/default.nix lets downstream naersk
projects construct a sparse root for the project which combines the
workspace's `Cargo.lock` with the project's own `Cargo.toml`.

Note that cargo commands in the workspace itself require the build
dependencies of _all_ projects to be present, which is currently a bit
annoying to accomplish.

This introduces some breakage:

1. It breaks usage of rust-analyser without being in a shell with the
   dependencies of *all* Tvix projects, as it is not capable of
   respecting only the subset of dependencies for a part of the
   workspace.

2. It is no longer possible to run tests using `cargo test`, as the
   test generation crate we use does not work with workspaces:

   https://github.com/frehberg/test-generator/issues/6

   This still works in the Nix build as we construct a Cargo project
   that looks like it's not in a workspace there. Until somebody fixes
   that crate / writes a new macro / does something else with the test
   suite, the way to run the tests is through the Nix build.

Long-term we'll probably want to get rid of cargo completely, it's
just a big wart and most tooling works without it if correctly
configured, but we don't have time for that now.

Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-15 17:26:45 +00:00
Florian Klink
0a4a0d1c6b feat(tvix/store): add size() and digest() functions for Directory
Change-Id: I3a9e3c77da1d5d7bfb93da0cd7bce8d48dbc8cd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7254
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-04 10:41:39 +00:00
Florian Klink
a97e3726df feat(tvix/store): build rust proto files
This uses prost-build to build the proto files.

Change-Id: Ia99fcfa4a19c741683cf28a45202e1d9211f9131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7286
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-04 10:41:39 +00:00
Florian Klink
c432aae9b3 feat(tvix/store): init
Change-Id: Id846647fa456ff8b187ed4f16d035ce030f11d25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7264
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-04 10:41:39 +00:00
Florian Klink
12413a669f feat(tvix/store): add pathinfo.proto
This adds the PathInfo message, which hosts information about a Nix
Store path, mapping to either of a {Directory,File,Symlink}Node.

Change-Id: I79d871b6fad450d6a4ae4101fb72c51f9a83471f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7132
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-04 10:41:39 +00:00
Florian Klink
ff5e041fdb chore(tvix/store): move castore.proto
This moves the castore.proto file into the //tvix/store/protos directory. Per-
component protos might make more sense, than a "tvix-wide" proto directory.

Change-Id: Ie728210174b041e9285a0e2ac605d715d0f0cbda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7285
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-04 10:41:39 +00:00