2023-02-12 14:24:43 +01:00
|
|
|
use crate::{proto, Error};
|
feat(tvix/store/directorysvc): add gRPC client
This provides a GRPCDirectoryService struct implementing
DirectoryService, allowing a client to Directory objects from a (remote)
tvix-store.
Remote in this case is anything outside the current process, be it
another process, or an endpoint on the network.
To keep the sync interface in the `DirectoryService` trait, a handle to
some tokio runtime needs to be passed into the constructor, and the two
methods use `self.tokio_handle.spawn` to start an async function, and
`self.tokio_handle.block_on` to wait for its completion.
The client handle, called `grpc_client` itself is easy to clone, and
treats concurrent requests internally. This means, even though we keep
the `DirectoryService` trait sync, there's nothing preventing it from
being used concurrently, let's say from multiple threads.
There's still two limitations for now:
1) The trait doesn't make use of the `recursive` request, which
currently leads to a N+1 query problem. This can be fixed
by `GRPCDirectoryService` having a reference to another
`DirectoryService` acting as the local side.
I want to wait for general store composition code to pop up before
manually coding this here.
2) It's currently only possible to put() leaf directory nodes, as the
request normally requires uploading a whole closure. We might want
to add another batch function to upload a whole closure, and/or do
this batching in certain cases. This still needs some more thinking.
Change-Id: I7ffec791610b72c0960cf5307cefbb12ec946dc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8336
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-23 13:49:57 +01:00
|
|
|
mod grpc;
|
2023-02-12 14:24:43 +01:00
|
|
|
mod memory;
|
|
|
|
mod sled;
|
2023-03-27 14:47:57 +02:00
|
|
|
mod utils;
|
2023-02-12 14:24:43 +01:00
|
|
|
|
feat(tvix/store/directorysvc): add gRPC client
This provides a GRPCDirectoryService struct implementing
DirectoryService, allowing a client to Directory objects from a (remote)
tvix-store.
Remote in this case is anything outside the current process, be it
another process, or an endpoint on the network.
To keep the sync interface in the `DirectoryService` trait, a handle to
some tokio runtime needs to be passed into the constructor, and the two
methods use `self.tokio_handle.spawn` to start an async function, and
`self.tokio_handle.block_on` to wait for its completion.
The client handle, called `grpc_client` itself is easy to clone, and
treats concurrent requests internally. This means, even though we keep
the `DirectoryService` trait sync, there's nothing preventing it from
being used concurrently, let's say from multiple threads.
There's still two limitations for now:
1) The trait doesn't make use of the `recursive` request, which
currently leads to a N+1 query problem. This can be fixed
by `GRPCDirectoryService` having a reference to another
`DirectoryService` acting as the local side.
I want to wait for general store composition code to pop up before
manually coding this here.
2) It's currently only possible to put() leaf directory nodes, as the
request normally requires uploading a whole closure. We might want
to add another batch function to upload a whole closure, and/or do
this batching in certain cases. This still needs some more thinking.
Change-Id: I7ffec791610b72c0960cf5307cefbb12ec946dc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8336
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-23 13:49:57 +01:00
|
|
|
pub use self::grpc::GRPCDirectoryService;
|
2023-02-12 14:24:43 +01:00
|
|
|
pub use self::memory::MemoryDirectoryService;
|
|
|
|
pub use self::sled::SledDirectoryService;
|
2023-03-27 14:47:57 +02:00
|
|
|
pub use self::utils::DirectoryTraverser;
|
2023-02-12 14:24:43 +01:00
|
|
|
|
|
|
|
/// The base trait all Directory services need to implement.
|
|
|
|
/// This is a simple get and put of [crate::proto::Directory], returning their
|
|
|
|
/// digest.
|
|
|
|
pub trait DirectoryService {
|
2023-03-26 13:51:48 +02:00
|
|
|
type DirectoriesIterator: Iterator<Item = Result<proto::Directory, Error>> + Send;
|
2023-03-27 17:08:16 +02:00
|
|
|
type DirectoryPutter: DirectoryPutter;
|
2023-03-26 13:51:48 +02:00
|
|
|
|
2023-02-12 14:24:43 +01:00
|
|
|
/// Get looks up a single Directory message by its digest.
|
|
|
|
/// In case the directory is not found, Ok(None) is returned.
|
2023-03-16 00:01:30 +01:00
|
|
|
fn get(&self, digest: &[u8; 32]) -> Result<Option<proto::Directory>, Error>;
|
2023-02-12 14:24:43 +01:00
|
|
|
/// Get uploads a single Directory message, and returns the calculated
|
|
|
|
/// digest, or an error.
|
2023-03-16 00:01:30 +01:00
|
|
|
fn put(&self, directory: proto::Directory) -> Result<[u8; 32], Error>;
|
2023-03-26 13:51:48 +02:00
|
|
|
|
|
|
|
/// Looks up a closure of [proto::Directory].
|
|
|
|
/// Ideally this would be a `impl Iterator<Item = Result<proto::Directory, Error>>`,
|
|
|
|
/// and we'd be able to add a default implementation for it here, but
|
|
|
|
/// we can't have that yet.
|
|
|
|
fn get_recursive(&self, root_directory_digest: &[u8; 32]) -> Self::DirectoriesIterator;
|
2023-03-27 17:08:16 +02:00
|
|
|
|
|
|
|
/// Allows persisting a closure of [proto::Directory], which is a graph of
|
|
|
|
/// connected Directory messages.
|
|
|
|
fn put_multiple_start(&self) -> Self::DirectoryPutter;
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Provides a handle to put a closure of connected [proto::Directory] elements.
|
|
|
|
///
|
|
|
|
/// The consumer can periodically call [put], starting from the leaves. Once
|
|
|
|
/// the root is reached, [close] can be called to retrieve the root digest (or
|
|
|
|
/// an error).
|
|
|
|
pub trait DirectoryPutter {
|
|
|
|
/// Put a individual [proto::Directory] into the store.
|
|
|
|
/// Error semantics and behaviour is up to the specific implementation of
|
|
|
|
/// this trait.
|
|
|
|
/// Due to bursting, the returned error might refer to an object previously
|
|
|
|
/// sent via `put`.
|
|
|
|
fn put(&mut self, directory: proto::Directory) -> Result<(), Error>;
|
|
|
|
|
|
|
|
/// Close the stream, and wait for any errors.
|
|
|
|
fn close(&mut self) -> Result<[u8; 32], Error>;
|
2023-03-26 13:51:48 +02:00
|
|
|
}
|