Commit graph

23 commits

Author SHA1 Message Date
Florian Klink
72e82ffcb1 refactor(tvix/store): use bytes for node names and symlink targets
Some paths might use names that are not valid UTF-8. We should be able
to represent them.

We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.

While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?

Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-21 19:01:49 +00:00
sterni
88b0545908 chore(3p/sources): Bump channels & overlays
* //tvix/store: update code generated from proto files

Change-Id: Ie6aa53e71f07a0a0e637673ba4005ed983db315d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8929
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-08 10:46:31 +00:00
Florian Klink
103d2eb909 refactor(tvix/store): bump CalculateNARResponse.nar_size to u64
Change-Id: I5f41981820363a5eb9982c3d19830916207c62cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8591
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 19:29:53 +00:00
Florian Klink
616fa4476f refactor(tvix/store): remove ChunkService
Whether chunking is involved or not, is an implementation detail of each
Blobstore. Consumers of a whole blob shouldn't need to worry about that.
It currently is not visible in the gRPC interface either. It
shouldn't bleed into everything.

Let the BlobService trait provide `open_read` and `open_write` methods,
which return handles providing io::Read or io::Write, and leave the
details up to the implementation.

This means, our custom BlobReader module can go away, and all the
chunking bits in there, too.

In the future, we might still want to add more chunking-aware syncing,
but as a syncing strategy some stores can expose, not as a fundamental
protocol component.

This currently needs "SyncReadIntoAsyncRead", taken and vendored in from
https://github.com/tokio-rs/tokio/pull/5669.
It provides a AsyncRead for a sync Read, which is necessary to connect
our (sync) BlobReader interface to a GRPC server implementation.

As an alternative, we could also make the BlobReader itself async, and
let consumers of the trait (EvalIO) deal with the async-ness, but this
is less of a change for now.

In terms of vendoring, I initially tried to move our tokio crate to
these commits, but ended up in version incompatibilities, so let's
vendor it in for now.

Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-11 14:27:57 +00:00
sterni
3a74268b6e chore(3p/sources): Bump channels & overlays
* //users/wpcarro/emacs: use top level (ELPA) version of eglot, as it
  was removed from MELPA:

  dc2ead17a8

* //3p/nixpkgs:tdlib: 1.8.11 -> 1.8.12

* //tvix/store: regenerate protos after buf update

Change-Id: I782a8d91fda5ed461788055a3721104e8c032207
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8327
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-21 01:31:48 +00:00
Florian Klink
87e1934f2d docs(store/protos): update comment on blobstore Read and Put
Further emphasize Read() can be used to ask for blobs OR chunks, and
that clients usually want to stat and then request (smaller) chunks,
rather than reading whole blobs.

Also clarify that the chunking used to send BlobChunks over has nothing
to do with the chunk sizes communicated in a Stat() request.

Change-Id: Ia615d190aae570611de2655b11342a14d0b75976
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-07 14:14:35 +00:00
Vincent Ambo
3e03e59893 refactor(tvix/store): move protobuf build config one level up
This embeds the build config directly at the point where `Cargo.nix`
is imported, making it transparent to library consumers.

Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 17:57:06 +00:00
Florian Klink
d22a9c8610 chore(tvix/store/protos): more idiomatic go
Pointed out by edef in
https://cl.tvl.fyi/c/depot/+/7648/comment/4551ba4b_e89ade36/#, thanks!

Change-Id: I6b0d317bb0210521622483cdf4cb557bc637a100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7709
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 15:18:02 +00:00
Florian Klink
c4ee942b1c feat(tvix/store/protos): rename Get to Read, add Stat method
Stat exposes metadata about a given blob,
such as more granular chunking, baos.
It implicitly allows checking for existence too, as asking this for a
non-existing Blob will return a Status::not_found grpc error.

The previous version returned a Status::not_found error on the Get
request too, but there was no chance to prevent the server from starting
to stream (except sending an immediate cancellation).

Being able to check whether something exists in a BlobStore helps to
prevent from uploading in first place.

The granular chunking bits are an optional optimization - if the
BlobStore implements no more granular chunking, the Stat response can
simply contain a single chunk.

Read returns a stream of BlobChunk, which is just a stream of bytes -
not necessarily using the chunking that's returned in the reply of a
Stat() call. It can be used to read blobs or chunks.

Change-Id: I4b6030ef184ace5484c84ca273b49d710433731d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7652
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
1c15154b83 feat(tvix/store): make blobstore stream chunks
This changes the RPC methods to return/consume a stream of chunks, instead of a
very big message containing the whole blob, to keep message sizes in manageable
sizes (less than 4MiB).

Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
d973c9772d chore(tvix/store/protos): add PathInfoService::CalculateNAR()
Expose the NAR calculation to a separate `CalculateNAR` method, which
responds with the NAR size and sha256 hash.

Contrary to what cl/7618 and cl/7620 initially did, don't add different
other request types.

In the CalculateNARResponse message, there's now some duplication in the
(optional) `narinfo` field of a PathInfo, but I'm not entirely sure if
we want to drop the fields from there yet.

Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
ce7be00994 chore(tvix/store/protos): drop non-sha256 NAR hashes
Nix actually doesn't support anything else than sha256 NAR hashes, so
there's little reason to communicate anything else in here.

Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I760370bd6cabd02028e001a74c454ef9296eb600
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7619
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 13:55:18 +00:00
Florian Klink
27bfc8cfcd chore(tvix/store/protos): move (Root)Node into separate message
This is gonna get used in another place in a second.

Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I347c11c8d24379628b7ed09d2c90670c576e686a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7617
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 13:55:18 +00:00
Florian Klink
dfd9286f68 feat(tvix/store/protos): implement Directory.Validate()
Validate thecks the Directory message for invalid data, such as:
- violations of name restrictions
- invalid digest lengths
- not properly sorted lists
- duplicate names in the three lists

Change-Id: I8d43a13797793c64097e526ef3bd482c9606c87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7648
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 21:31:33 +00:00
Florian Klink
c3fb6d2218 feat(tvix/store/protos): implement Size() and Digest() for Directory
This adds Size() and Digest() functions for the golang version.

Change-Id: If71445a9bb26100bb4076ac4f5c96945b33919f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7325
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-27 21:31:33 +00:00
Florian Klink
a91e6334ff feat(txix/store/protos): init go.mod, generate .pb.go files
This allows importing the generated .pb.go files into other go projects.

I initially looked at buildGo.protos, but it doesn't work for multi-.proto
files, and actually having LSP support for the generated structs is nice, too.

Change-Id: Idbd448008010790a10a0ea42e4059dbb609eaf1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7322
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
2b9330911a feat(tvix/store/protos): add go_package option
Change-Id: I0898b8a0a78e704219da38e5acaabef1e640d4e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7321
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2022-12-26 11:38:52 +00:00
Florian Klink
e05e8bdd03 docs(tvix/store): add README, document services and store model
These are intended to help digest the protocol definitions for tvix-
store, and how they tie into the whole concept.

Co-Authored-By: Vincent Ambo <mail@tazj.in>
Change-Id: Ic1ba3ba41ef599209453f15d0ac2e07a6144bcca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7439
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
2e04f68df8 feat(tvix/proto): add rpc_directory.proto
This provides an interface to retrieve and upload single Directory
messages, or a DAG of them.

Change-Id: Id9e7084bd697d843649a122da2c992a3e36d808c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7137
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 14:35:32 +00:00
Florian Klink
870cd533dd feat(tvix/proto): add rpc_pathinfo
This defines a service that can be used to upload and retrieve metadata
of nix paths.

Change-Id: Id86eb531ce4ae316adb15934b0d1386a14ba2132
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 14:35:32 +00:00
Florian Klink
5f9796cf94 feat(tvix/proto): add rpc_blobstore
This defines a service that can be used to get and put content-addressed
chunks of data.

Change-Id: I36cf2278ed1daf71848c04fdfd14450b2268c5de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-15 19:10:26 +00:00
Florian Klink
12413a669f feat(tvix/store): add pathinfo.proto
This adds the PathInfo message, which hosts information about a Nix
Store path, mapping to either of a {Directory,File,Symlink}Node.

Change-Id: I79d871b6fad450d6a4ae4101fb72c51f9a83471f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7132
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-04 10:41:39 +00:00
Florian Klink
ff5e041fdb chore(tvix/store): move castore.proto
This moves the castore.proto file into the //tvix/store/protos directory. Per-
component protos might make more sense, than a "tvix-wide" proto directory.

Change-Id: Ie728210174b041e9285a0e2ac605d715d0f0cbda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7285
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-04 10:41:39 +00:00