Commit graph

117 commits

Author SHA1 Message Date
Florian Klink
72bc4e0270 fix(tvix): use tracing::field::Empty for all indicatif.pb_show
This prevents the field from getting printed to stderr, apparently.

Change-Id: Ia9860e4ff37224003154db88ee5f83103060e626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12756
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-11-12 00:28:32 +00:00
Florian Klink
a218f421b2 refactor(tvix/store): move Cache to cache mod
This being in combinators makes it harder to find.

Change-Id: If7984bdbd43f164c670548639bb4846d859f6695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12750
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-11-11 18:46:33 +00:00
Florian Klink
a9f453f6da docs(tvix/[ca]store): improve docstrings, remove wildcard imports
Extend the docstrings of `add_default_services`, and add one for
`addrs_to_configs` as well as the module-wide one at
`tvix_store::composition`.

Change-Id: Ie9b449988eb210cd65b19b174094bbe0c4af2fd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12748
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-11-11 18:46:33 +00:00
Florian Klink
d505f03e00 refactor(tvix/store/composition): rename 'default' to 'root'
This becomes the root of the composition. `default` implies we can
directly access anything else, which we cannot. `root` makes this more
understandable, and it's all internal only anyways.

Change-Id: I297511bc05a7c32c59510b9d192b40d1bd937b5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12746
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-11-11 18:46:33 +00:00
Bob van der Linden
cfa4154131 feat(tvix): add instance_name to instrumentation of *Services
Currently it is not possible to distinguish between tracing of the same
*Service type whenever there are multiple of them. Now the instance_name
of ServiceBuilder is passed into the *Service and used in the existing
instrument as the `instance_name` field.

Places that did not already have a instance_name in its context use
`"default"`. In tests I used `"test"`.

Change-Id: Ia20bf2a7bb849a781e370d087ba7ddb3be79f654
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12739
Tested-by: BuildkiteCI
Autosubmit: Bob van der Linden <bobvanderlinden@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-11-08 20:16:49 +00:00
Florian Klink
849966d614 refactor(tvix/store/pathinfo/signing_wrapper): remove clone
Construct the owned signature in a separate scope, so all borrows to the
original PathInfo are already dropped again, and we can modify the
PathInfo without having to clone it.

Change-Id: I03e7390540c2cfe7a2c61850bdbe8a33d213a5d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12663
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-10-19 09:41:58 +00:00
Florian Klink
9c22345019 refactor(tvix/[ca]store): use auto_impl
This implements BS, DS, PS for Box'ed or Arc'ed variants of it with less
code, and less potential to accidentially forget to proxy default trait
methods for blanked impls, as fixed in cl/12658.

Change-Id: If2cdbb563a73792038ebe7bff45d6f880214855b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12661
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-10-18 21:45:55 +00:00
Florian Klink
bd6894d265 fix(tvix/store/pathinfo): use gRPC NarCalculationService
If the PathInfoService is using gRPC, expose it via the
nar_calculation_service() method in the PathInfoService.

Also pass nar_calculation_service through in tha blanket impl.

This now causes a `tvix-store import` using the default config to not
fall back to `SimpleRenderer`, which will calculate the NAR hash and
size by downloading the uploaded blobs (and blobs it didn't need to
upload) locally, making such imports faster.

Change-Id: If2c3fe6584e9093cba322d2360f355a3923904ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12658
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2024-10-18 21:45:55 +00:00
Florian Klink
1c80bc4b5b refactor(tvix/store): remove use of lazy_static
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.

Change-Id: Ifca792f4d2dbc36b703de4a4dfa406015ab86da7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12614
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-10-17 22:15:48 +00:00
Florian Klink
8b7b85359b test(tvix/store/signing_wrapper): restructure
Move things around a bit to make it easier to understand what's going on:

 - We first validate our fixture invariants
 - We then insert into the PathInfoService
 - Do all comparisons and checks we can on the returned PathInfo struct
 - Only convert to the NarInfo variant to calculate the fingerprint,
   and don't keep intermediate let bindings for this

Before cl/12588, this was arguably much harder to do that way, as we
relied on some of the conversions done in the to_narinfo() function.

Change-Id: Iaddbf1079f73ce566ef6d56f69a823e080b2e006
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12595
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sinavir <tvix@sinavir.fr>
2024-10-11 17:19:12 +00:00
Marijan Petričević
e8040ec61f refactor(tvix/store): use strictly typed PathInfo struct
This switches the PathInfoService trait from using the proto-derived
PathInfo struct to a more restrictive struct, and updates all
implementations to use it.

It removes a lot of the previous conversion and checks, as invalid
states became nonrepresentable, and validations are expressed on the
type level.

PathInfoService implementations consuming protobuf need to convert and
do the verification internally, and can only return the strongly typed
variant.

The nix_compat::narinfo::NarInfo conversions for the proto PathInfo
are removed, we only keep a version showing a NarInfo representation for
the strong struct.

Converting back to a PathInfo requires the root node now, but is
otherwise trivial, so left to the users.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-11 17:18:20 +00:00
Yureka
52bb3c6d02 feat(tvix/composition): allow urls as anonymous stores
This allows specifying an url in place of a named reference to another
composition entry, if the castore crate has been compiled with the
 xp-store-composition feature.

Example: `--directory-service-addr cache://?near=memory://&far=memory://`

This would be equivalent to the instantiation via toml file:

```toml
[memory1]
type = "memory"

[memory2]
type = "memory"

[default]
type = "cache"
near = "memory1"
far = "memory2"
```

Note that each anonymous url causes a distinct instance to be created.

Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-10 12:53:08 +00:00
Yureka
07bf8a0b6d feat(tvix/pathinfo/nixhttp): use ingest stores from url
This still defaults to the "default" services, but allows users to tell the
nix+http pathinfoservice to ingest the castore nodes into a non-default
blob-/directoryservice when used with the experimental store composition.

Change-Id: I5c0f683ce95d888eadf3f302520a47f42f1a481d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12148
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-10 12:53:08 +00:00
Ilan Joselevich
ab6e8d28aa chore(tvix/[ca]store): Drop sled support completely in favor of redb
Over the past couple of months we've been using redb instead of sled as
the default filesystem-based database in PS and DS. I am confident that
we can get rid of sled completely now, and just keep redb.

Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-10-03 18:16:03 +00:00
sinavir
9f36632509 feat(tvix/store): Add a signing PathInfoService
- Add a new PathInfoService implementation that wraps transparently
around another except that it dynamically signs all the incoming
path-infos with the provided signer.

- Add a ServiceBuilder for this PathInfoService that provides a
SigningPathInfoService with a keyfile signer

Change-Id: I845ddfdf01d14c503c796b2b80c720dab98be091
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12032
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sinavir <tvix@sinavir.fr>
Tested-by: BuildkiteCI
2024-09-26 12:29:03 +00:00
Florian Klink
2beabe968c refactor(nix-compat/store_path): make StorePath generic on S
Similar to how cl/12253 already did this for `Signature`, we apply the
same logic to `StorePath`.

`StorePathRef<'a>'` is now a `StorePath<&'a str>`, and there's less
redundant code for the two different implementation.

`.as_ref()` returns a `StorePathRef<'_>`, `.to_owned()` gives a
`StorePath<String>` (for now).

I briefly thought about only publicly exporting `StorePath<String>`
as `StorePath`, but the diff is not too large and this will make it
easier to gradually introduce more flexibility in which store paths to
accept.

Also, remove some silliness in `StorePath::from_absolute_path_full`,
which now doesn't allocate anymore.

Change-Id: Ife8843857a1a0a3a99177ca997649fd45b8198e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12258
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-20 15:14:27 +00:00
Florian Klink
5ec93b57e6 refactor(tvix/castore): add PathComponent type for checked components
This encodes a verified component on the type level. Internally, it
contains a bytes::Bytes.

The castore Path/PathBuf component() and file_name() methods now
return this type, the old ones returning bytes were renamed to
component_bytes() and component_file_name() respectively.

We can drop the directory_reject_invalid_name test - it's not possible
anymore to pass an invalid name to Directories::add.
Invalid names in the Directory proto are still being tested to be
rejected in the validate_invalid_names tests.

Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-17 15:59:10 +00:00
Florian Klink
49b173786c refactor(tvix/castore): remove name from Nodes
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.

This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.

It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).

The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.

The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.

The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.

Some further cleanups:

The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)

There's some leftover cleanups to do:
 - There should be a from_(sorted_)iter and into_iter in Directory, so
   we can construct and deconstruct in one go.
   That should also enable us to implement conversions from and to the
   proto representation that moves, rather than clones.

 - The BuildRequest and PathInfo structs are still proto-based, so we
   still do a bunch of conversions back and forth there (and have some
   ugly expect there). There's not much point for error handling here,
   this will be moved to stricter types in a followup CL.

Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 09:45:58 +00:00
Florian Klink
c7845f3c88 refactor(tvix/castore): move *Node and Directory to crate root
*Node and Directory are types of the tvix-castore model, not the tvix
DirectoryService model. A DirectoryService only happens to send
Directories.

Move types into individual files in a nodes/ subdirectory, as it's
gotten too cluttered in a single file, and (re-)export all types from
the crate root.

This has the effect that we now cannot poke at private fields directly
from other files inside `crate::directoryservice` (as it's not all in
the same file anymore), but that's a good thing, it now forces us to go
through the proper accessors.

For the same reasons, we currently also need to introduce the `rename`
functions on each *Node directly.

A followup is gonna move the names out of the individual enum kinds, so
we can better represent "unnamed nodes".

Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-13 18:39:49 +00:00
Yureka
3ca0b53840 refactor(tvix/castore): use Directory struct separate from proto one
This uses our own data type to deal with Directories in the castore model.

It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:

 - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
 - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.

Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-13 12:17:01 +00:00
Ilan Joselevich
3f6cd7aebc fix(tvix/store): Immediately return an error when using sled on /
We already do this for redb and for sled in SledDirectoryService.

Change-Id: I34c7178257a6a04e9f12ed4037a4ef585d7b0d54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12060
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-29 16:43:04 +00:00
Ilan Joselevich
9cbebfea27 fix(tvix/store) RedbPathInfoService: improve logs and errors
Add more logging and remove context from errors because that's already
provided by the logs (Errors also need to be refactored anyway, there's
also confusion about StorageError vs InvalidRequest, there's no
consistency)

Change-Id: Ia43c0d237d9075152490c635b05fb3fb343abcc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12058
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-29 15:50:11 +00:00
Ilan Joselevich
a982f73477 fix(tvix/store): adjust from_addr redb test to do what it says
Change-Id: If15f161d5c7aba05ac1d8e2a4f6fac5a7053943a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12040
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-28 17:47:41 +00:00
Ilan Joselevich
d17f0592da docs(tvix/store): Document redb in from_addr
Change-Id: Id2ef4ee1b22c20e5b79156f40821578979105ddc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12039
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-28 16:40:40 +00:00
Ilan Joselevich
9f10a71ec5 feat(tvix/store): add redb PathInfoService
This provides a PathInfoService implementation using redb
(https://github.com/cberner/redb) as the underlying storage engine.

Both an in-memory variant, as well as a filesystem one is provided,
similar how it's done with the sled implementation.

Supersedes: https://cl.tvl.fyi/c/depot/+/11692
Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc
Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-22 18:49:04 +00:00
sinavir
5d82e1e62f fix(tvix/store): Fix narinfo compression selection
Parsing of the narinfo file sets the compression field to None instead
of Some("none"). The mapping selecting the decompression reader expected
the former in //tvix/store/src/pathinfoservice/nix_http.rs.

Change-Id: I254a825b88a4016aab087446bdc0c7b6286de40c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12007
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 22:11:20 +00:00
Florian Klink
05b4e805ee refactor(tvix/nix-compat): rename PubKey to VerifyingKey
Align these with the way it's called in the ed25519 crates.

Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-07-21 21:41:54 +00:00
Yureka
ca8e2b9fbf chore(tvix): upgrade to tonic 0.12 / hyper 1.0
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 11:01:38 +00:00
Yureka
8b77c7fcd7 refactor(tvix/store): use composition in tvix_store crate
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-20 19:37:27 +00:00
Florian Klink
9ed967791c test(tvix/store): test listing endpoint too
Change-Id: Ia4035aca43cf9d3f7de982dd154715120ba25496
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11927
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-03 12:35:07 +00:00
Florian Klink
eb9a74d966 fix(tvix/store/pathinfo/bigtable): fix listing endpoint
We were wrongly comparing the raw row_key with the store path
digest, but we hexlower-encode the digest in the row key (via
derive_pathinfo_key).

Update the logic to fix that.

Change-Id: I8916d8de9fb8b25a6986d4158faa91ec97c57347
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11926
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-03 12:35:07 +00:00
Simon Hauser
618aacaa61 feat(tvix/tracing): http trace propagation
Introduces a helper function within tvix-tracing that returns a reqwest
tracing middleware that will ingest the traceparent if otlp is enabled.

It is feature flagged in tvix-tracing so not every consumer of that
library automatically has reqwest in its dependencies.

Tested using netcat to verify that the `traceparent` header is there if
otlp is enabled and missing if otlp feature is disabled.

Change-Id: I5abccae777b725f5ff7382e3686165383c477a39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-02 13:43:09 +00:00
Florian Klink
7f625e2fc1 refactor(tvix/*store): remove some trait bounds
We don't need to require these things for these impl blocks yet.

Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
2024-06-20 21:07:03 +00:00
Simon Hauser
639a00e2ab feat(tvix/tracing): gRPC trace context propagation
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.

The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.

This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.

Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-20 19:21:01 +00:00
Florian Klink
cfab953094 feat(tvix/store): display progress for NAR calculation
This is currently still taking a noticeable amount of time, so make sure
we show it is happening.

Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 09:47:54 +00:00
Florian Klink
ff40332864 refactor(tvix/store/nar/import): add ingest_nar_and_hash
This wraps ingest_nar, but also keeps track of the number of bytes read,
and calculates the sha256 digest of it.

Make use of it in the NixHTTPPathInfoService, where this code is coming
from.

Change-Id: I9c54e93d3ec8ed9ede87aed43e04d114fb06897b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-11 22:47:31 +00:00
Simon Hauser
825d498908 feat(tvix/tracing): introduce common tvix-tracing crate
Introduce a new common crate that contains tracing boilerplate which then
can be used in the cli, tvix-store and tvix-build crates.
It has otlp as an optional feature, which is currently only used by
tvix-store.

Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-10 16:35:08 +00:00
Florian Klink
bc42c355cf refactor(tvix/store/pathinfo): test with PathInfoService directly
Since cl/…, a PathInfoService doesn't need to implement `calculate_nar`
anymore, so most of them don't actually have a handle to a
{Blob,Directory}Service anymore.

This means, we can simplify the construction of them for test cases
a lot.

Change-Id: I100e9e1c9b00a049b4d6136c57aad4cdb04461c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11691
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-20 15:03:10 +00:00
Florian Klink
adb42959a3 feat(tvix/store/pathinfo): add a Cache combinator
This allows querying two PathInfoService implementations sequentially,
and inserts into the "near" one if it's not there yet.

There is no negative cache, and put / listing is not implemented (for
now).

Change-Id: I24c3d0e0c3c2f0524a6cc7b2f3cbc33eb20cf92b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11636
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 22:05:59 +00:00
Florian Klink
ed2c0be920 feat(tvix/store/pathinfo/*): add more instrumentation
Add instrumentation to the get() and put() implementations of all
PathInfoService.

Use the nixbase32 representation of the digest, not the base64 one.

Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-12 22:03:15 +00:00
Florian Klink
96b8b1a205 refactor(tvix/store/pathinfo/memory): tokio RwLock, improve list()
We don't want to use the std::sync::RwLock here, as it blocks.

This also means we don't need to deal with the error cases anymore.

The list() implementation is updated to use try_stream, which means we
can now avoid collecting everything into a Vec before returning from it.

Change-Id: I9057dcc410dc553e6b1be3f20d5ee830569e8218
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11611
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 15:07:31 +00:00
Florian Klink
fe20ba5ffc feat(tvix/store): add LruPathInfoService
This provides an implementation of PathInfoService storing PathInfo in
memory up to a certain capacity, then evicting these that have been used
the least recently.

Change-Id: I9d738687caf4f181a957f72245f26b92832313cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11622
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:55:21 +00:00
Florian Klink
a49c32ef42 refactor(tvix/store/pathinfo/sled): drop {blob,directory}_service
These are not used anymore.

Change-Id: I9c348391c9600e9319f171faf3eda7175ebf7076
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11621
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 13:55:21 +00:00
Florian Klink
03af6ab725 refactor(tvix/store/pathinfo/memory): drop {blob,directory}_service
These are not used anymore.

Change-Id: I6c16b4d80ddaabcb75fec3ea3e32b923b7719485
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11620
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:55:21 +00:00
Florian Klink
14766cfe1d refactor(tvix/store): drop calculate_nar from PathInfoService
This shouldn't be part of the PathInfoService trait.

Pretty much none of the PathInfoServices do implement it, and requiring
them to implement it means they also cannot make use of this calculation
already being done by other PathInfoServices.

Move it out into its own NarCalculationService trait, defined somewhere
at tvix_store::nar, and have everyone who wants to trigger nar
calculation use nar_calculation_service directly, which now is an
additional field in TvixStoreIO for example.

It being moved outside the PathInfoService trait doesn't prohibit
specific implementations to implement it (like the GRPC client for the
`PathInfoService` does.

This is currently wired together in a bit of a hacky fashion - as of
now, everything uses the naive implementation that traverses blob and
directoryservice, rather than composing it properly. I want to leave
that up to a later CL, dealing with other parts of store composition
too.

Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 13:33:59 +00:00
Florian Klink
944a781354 fix(tvix/store/pathinfo/sled): use spawn_blocking
This does IO, which might take a longer amount of time than what we want
to be blocking the normal executor.

Use spawn_blocking instead. I didn't add it for the constructors, as we
only call these once.

Change-Id: I9a1063099bac9582ca9681043c58c1edc780c5ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11618
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-11 13:33:59 +00:00
Florian Klink
73c848a18f refactor(tvix/store/pathinfo/grpc): simplify pingpong
We already have the same code in make_grpc_path_info_service_client.

Change-Id: Ibcd60831af8a061a2c3bb2f960f52a43d06cf6fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11615
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 13:28:51 +00:00
Florian Klink
01a4a2399c fix(tvix/store/pathinfo/nix_http): update error message
Autosubmit was too fast and submitted this before my push went through.

Flagged in https://cl.tvl.fyi/c/depot/+/11580/comment/40a56824_7be73334/

Change-Id: I1f835ae60c2c6fd2db6654c1b1c71d90bee141af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11595
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-06 15:35:24 +00:00
Florian Klink
aaf258f61e feat(tvix/store): use async nar reader for ingest_nar
Rename read_nar to ingest_nar, and have it use the async nar reader
version, and the ingest_entries machinery.

This means we can now drop all code dealing with manually assembling
castore nodes.

Update our consumer, NixHTTPPathInfoService to use the new API.
As we now accept an AsyncRead, we don't need to do any blocking here
anymore, and can use the same async-compression crate as in the fetching
logic (and support some more compression formats out of the box).

Change-Id: I8646d20bd8603f8da47b5c84bc9e4ac236eb7f1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11580
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:24:17 +00:00
Florian Klink
1c7d319164 refactor(tvix/store/pathinfo/sled): cleanup, add instrumentation
Write this a bit more compact, by using map_err(|e| …) and ?.

Ideally we'd get rid of the error mapping entirely, by using proper
error types, but that's left for a followup.

Change-Id: I68dc72b162ac89c5ff82d8c2bc26e1c808a0affd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11584
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-03 19:49:51 +00:00