Previous CLs ensured BlobService and DirectoryService is implemented on
AsRef<dyn BlobService> and AsRef<dyn DirectoryService> respectively, so
there's no need to require AsRef in here.
Change-Id: I1e8b93188ca784e1b9a592f3d35fcf6b9df75c75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11275
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This stops using the gen_{blob,directory}_service() functions from
tvix_castore::utils in favor of the rstest-based fixtures.
Change-Id: I4c2fff2926f34af63d3bd5c29c8e939dbdf7e062
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11274
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This introduces rstest-based tests. We also add fixtures for creating
some BlobService / DirectoryService out of thin air.
To test a PathInfoService, we don't really care too much about its
internal storage - ensuring they work is up to the castore tests.
Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This makes the fixture more use-able when interacting with the trait,
the Bytes are only useful for the gRPC version.
Change-Id: Iaaea1adc6df18491f236a28c4343f5b4ee5fcfd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11271
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This allows StorePathRef as keys in a hashmap, by deriving Hash. The same is already done for StorePath.
Change-Id: I3fc54c45787948116dcb27dfb5dc806b9b505835
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11269
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
At least terraform wants all parameters passed via JSON to be strings.
It can't accept maps.
This means, allowing to pass a `argstr` JSON dict isn't really possible.
However, terraform is perfectly able to JSON-encode a map. So accept a
`argstr_json` argument instead, which `jq` will JSON-decode before
further processing it.
I dropped `argstr` support again so the jq expression still fits on my
screen, if anyone else (started) using this, I'm happy to review a CL
adding this. We should probably move the jq expression to some multiline
format then, though.
Change-Id: I5ab7a1169ab7305d3ab02db31c27732d9d1ab4e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11228
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
If this is set to true (and only then), also invoke `nix-build` on the
previously-instantiated .drv to cause builds/substitutions on the local
machine.
There's no terraform example for this in here, but this is useful if you
want to perform builds locally, for example to upload nix-built blobs
elsewhere through terraform.
Change-Id: Idcf7b8527aa9c27f6f9ca60ca607c29d82e1cce9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11215
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Implementing the full connection handshake.
The integration test is a bit naive, but there's not much to test yet.
Tested this against cpp nix. We reach the stage where cppnix sends the
opcode.
Change-Id: I98322832848ee5b048f22105731b0adeb44b2ce0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11227
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We already did all the instantiation, grabbing the calculated output
path too is cheap.
Change-Id: Id591865c65159409da739f706a9de29a9f50456a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11214
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This allows us to use containers around BlobServices as BlobServices too.
Change-Id: I3c7feb074f42b4e07c550fb8dfa63cf81d448ab5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11249
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This creates test scenarios (using the DirectoryService trait) that we
want all DirectoryService implementations to pass.
Some of these tests are ported from proto::tests::grpc_directoryservice,
which tested this on the gRPC interface (rather than the trait),
some others ensure certain behaviour for which we only recently
introduced general checking logic (through ClosureValidator).
We also borrow some code related to setting up a gRPC DirectoryService
client (connecting to a server exposing a in-memory DiretoryService)
from castore::utils, this will be deleted once it's all ported over.
Change-Id: I6810215a76101f908e2aaecafa803c70d85bc552
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11247
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This allows us to use containers around DirectoryServices as DirectoryServices too.
Change-Id: I56cca27b3212858db8b12b874df0e567dd868711
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11248
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This uses DirectoryClosureValidator for validation and the sled batch
API to insert multiple directories at once.
Change-Id: I2d6dc513ccbc02e638f8d22173da5463e73182ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11222
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This greatly simplifies the code in this function, replacing it with a
much better tested (and more capable!) version of the validation logic.
It also enables the gRPC server frontend to make use of the
DirectoryPutter interface. While this might not be too visible in terms
of latency thanks to gRPC streams bursting, it also enables further
optimizations later (such as bucketing of directory closures).
Change-Id: I21f805aa72377dd5266de3b525905d9f445337d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11221
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This simplifies a bunch of code, and gets rid of some TODOs.
Also, move it out of castore/utils, and into its own file.
Change-Id: Ie63e05a6cdfb2a73e878cf7107f9172aed1cdf13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11224
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This can be used to validate a Directory closure (connected DAG of
Directories), and their insertion order.
Directories need to be inserted (via `add`), in an order from the leaves
to the root. During insertion, we validate as much as we can at that
time:
- individual validation of Directory messages
- validation of insertion order (no upload of not-yet-known Directories)
- validation of size fields of referred Directories
Internally it keeps all received Directories (and their sizes) in a HashMap,
keyed by digest.
Once all Directories have been inserted, a drain() function can be
called to get a (deduplicated and) validated list of directories, in
from-leaves-to-root order (to be stored somewhere).
While assembling that list, a check for graph connectivity is performed
too, to ensure there's no separate components being sent (and only one
root).
It adds a test suite for these cases, which is much nicer to test than
where we previously had these checks (only in the gRPC server wrapper).
Followup CLs will move the existing putters to use this.
Change-Id: Ie88c832924c170a24626e9e3e91d868497b5d7a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11220
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
First stab at factoring out handlers into a generalized handler
function.
This is still kind of confusing, but can be simplified later.
Change-Id: I42da047de83f6d489337d57059f85f793313443a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11245
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
We need to ensure the Directories are successfully uploaded before doing
any testing with them.
Change-Id: Iafa8deb86b3d5eb302ebfba3ced34385f67a7229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11244
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Move the http calls into their own module, so we can trace the request
and provide a simple copy-to-replay command.
We have to work around a bug in the otel library, which would limit
our attribute value length to 128 bytes because it uses the wrong
option value.
~~~
`ifExists` is finally made more useful for dealing with optional
attributes in e.g. lists.
Change-Id: Iafab523e9ec4b00136db43f31fdc12aeefb7f77c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11241
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
We tried to be more strict than Nix, actually detecting if multiple
hashes were specified, or other garbage at the end.
However, Nix seems to just chop off at the end, so happily accepts
anything afterwards.
Example: https://github.com/NixOS/nixpkgs/pull/298041
Example: https://github.com/NixOS/nixpkgs/pull/298052
Change-Id: I2c1a49f51c8f8589a84df2fbf148e67e7380b550
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11234
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Move the functionality into two coarse modules.
There’s still the question about whether functions that change the
database tables should be in their own storage module, but let’s see
if it gets too confusing.
Change-Id: Ied1d47b353dd4597ffea35f111f440aad22e981d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11238
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
This documents the input and output format, and also removes some
references to Terraform and evaluating NixOS system configurations.
It can be used to evaluate anything.
Change-Id: I8492cc3e386f89b299469c78e586644ee82a708f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11213
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This allows these messages to be put in HashSets.
Change-Id: Ia58094cafe53eb624578821d3d8d969c5d21a1d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11219
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Log the entire span with "trace" level, not just its `ret` level.
The level of the error value event defaults to ERROR, so we don't loose
these.
B3Digest implements Debug and Display the same way, so we can omit the
`(Display)` part in `ret(Display)` for them.
Change-Id: Id00d123a5798e5bdc9820dd97ae2b4d4eb5455f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11218
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This is no public API to construct this, there's exactly one caller,
and it's perfectly fine to directly populate the struct there.
Change-Id: Idae43a0162ee9bc687d21c550e0c9df33f12d263
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11217
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Turns out there's some fancy extra features like syncing the shell and
the buffer directory, which ezemtsov discovered recently.
Change-Id: Ibb6f222d277e2cb3725387364d43e7b916b6df35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11216
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This makes it easier to see what's going wrong when uploading multiple
Directories.
Change-Id: Ieb71424b9761777c5f719b2f365962644de82baf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11209
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We had max_level_debug, not max_level_trace, so it was not possible to
have `RUST_LOG` emit trace-level values (at least for our crates).
release_max_level_info still controls the level for release builds, so
that's not affected.
Change-Id: I1f127ab63f13b1622d6616e06759a9414d9bb201
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11208
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This functionality is provided by the object store backend too
(using `objectstore+file://$some_path`).
This backend also supports content-defined chunking and compresses
chunks with zstd.
Change-Id: I5968c713112c400d23897c59db06b6c713c9d8cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11205
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This controls whether tvix-castore has support for various cloud
backends or not.
Use this to control the set of feature flags for the object_store
backend, and only enable the aws, azure and gcp ones if it's set.
In the future this can be used to enable/disable other cloud backends
too.
Without feature flags, `object_store` already supports the `InMemory`
and `LocalFilesystem` backends, and we also want to unconditionally
enable the `http` one. Make sure at least the construction of these
services is covered in the tests.
Similarly, the tvix-store crate, which provides the tvix-store CLI has a
`cloud` feature flag too (defaulting to enabled).
Change-Id: I9fb9c87b740e7dc83f8ff7a0862905d036d513f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11204
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
The rust trait was missing to document the order of the elements in the
stream. Document that, and also the reasoning behind this.
Change-Id: I27ef0b2020082783fc41c2015233175e2b8e716d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11203
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This will allow feature-flagging some of the backends.
Change-Id: Ie92914c3e2ad870eee87e73b3b5abe605fb56fe7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11202
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This will allow feature-flagging some of the backends.
Change-Id: Iddbdb89d3cf9c966a2c25b06b03e6917b284cae5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11201
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
This will allow feature-flagging some of the backends.
Change-Id: Idffbf8b3fd154f5a3d938225c3871feffea8ff8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11200
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Adding a VM integration test setup for tvix-daemon. This test acts as
our first milestone: implementing enough daemon operations to be able
to add a new store path to tvix-store.
The test is expected to fail for now. We don't want to run it on CI
yet.
Change-Id: I2bd8eb9a07c5de2ef91099e10fcac23c087b880b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11199
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This daemon is a re-implementation of the Nix daemon except it uses
tvix-store as a remote store.
For now, it's very barebones, this is just a quick and dirty setup to
get started with the project. We bind to the unix socket provided by
systemd, wait for cpp Nix to send the magic hello bytes, respond with
the magic hello bytes and call it a day.
Storing this under my username for now, the project is mostly
irrelevant as it is. We'll move it to Tvix if it gets complete and
relevant at some point.
Change-Id: Ifc5dce2df37413504f9de1942c5b7d425eddf759
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11198
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
