refactor(tvix/castore): use Directory struct separate from proto one
This uses our own data type to deal with Directories in the castore model.
It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:
- In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
- In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.
Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-29 14:34:50 +02:00
|
|
|
use bstr::ByteSlice;
|
2023-02-12 11:39:31 +01:00
|
|
|
use thiserror::Error;
|
2023-03-25 22:17:23 +01:00
|
|
|
use tokio::task::JoinError;
|
2023-02-12 12:34:15 +01:00
|
|
|
use tonic::Status;
|
2023-02-12 11:39:31 +01:00
|
|
|
|
2024-08-17 21:00:50 +02:00
|
|
|
use crate::{
|
|
|
|
path::{PathComponent, PathComponentError},
|
|
|
|
SymlinkTargetError,
|
|
|
|
};
|
2024-08-16 16:32:20 +02:00
|
|
|
|
2023-02-12 11:39:31 +01:00
|
|
|
/// Errors related to communication with the store.
|
2024-03-23 21:49:49 +01:00
|
|
|
#[derive(Debug, Error, PartialEq)]
|
2023-02-12 11:39:31 +01:00
|
|
|
pub enum Error {
|
|
|
|
#[error("invalid request: {0}")]
|
|
|
|
InvalidRequest(String),
|
|
|
|
|
|
|
|
#[error("internal storage error: {0}")]
|
|
|
|
StorageError(String),
|
|
|
|
}
|
refactor(tvix/castore): use Directory struct separate from proto one
This uses our own data type to deal with Directories in the castore model.
It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:
- In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
- In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.
Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-29 14:34:50 +02:00
|
|
|
|
refactor(tvix/castore): remove `name` from Nodes
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.
This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.
It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).
The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.
The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.
The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.
Some further cleanups:
The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)
There's some leftover cleanups to do:
- There should be a from_(sorted_)iter and into_iter in Directory, so
we can construct and deconstruct in one go.
That should also enable us to implement conversions from and to the
proto representation that moves, rather than clones.
- The BuildRequest and PathInfo structs are still proto-based, so we
still do a bunch of conversions back and forth there (and have some
ugly expect there). There's not much point for error handling here,
this will be moved to stricter types in a followup CL.
Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-14 21:00:12 +02:00
|
|
|
/// Errors that occur during construction of [crate::Node]
|
refactor(tvix/castore): use Directory struct separate from proto one
This uses our own data type to deal with Directories in the castore model.
It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:
- In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
- In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.
Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-29 14:34:50 +02:00
|
|
|
#[derive(Debug, thiserror::Error, PartialEq)]
|
|
|
|
pub enum ValidateNodeError {
|
|
|
|
/// Invalid digest length encountered
|
|
|
|
#[error("invalid digest length: {0}")]
|
|
|
|
InvalidDigestLen(usize),
|
|
|
|
/// Invalid symlink target
|
2024-08-17 21:00:50 +02:00
|
|
|
#[error("Invalid symlink target: {0}")]
|
|
|
|
InvalidSymlinkTarget(SymlinkTargetError),
|
refactor(tvix/castore): use Directory struct separate from proto one
This uses our own data type to deal with Directories in the castore model.
It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:
- In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
- In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.
Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-29 14:34:50 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
impl From<crate::digests::Error> for ValidateNodeError {
|
|
|
|
fn from(e: crate::digests::Error) -> Self {
|
|
|
|
match e {
|
|
|
|
crate::digests::Error::InvalidDigestLen(n) => ValidateNodeError::InvalidDigestLen(n),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
refactor(tvix/castore): remove `name` from Nodes
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.
This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.
It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).
The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.
The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.
The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.
Some further cleanups:
The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)
There's some leftover cleanups to do:
- There should be a from_(sorted_)iter and into_iter in Directory, so
we can construct and deconstruct in one go.
That should also enable us to implement conversions from and to the
proto representation that moves, rather than clones.
- The BuildRequest and PathInfo structs are still proto-based, so we
still do a bunch of conversions back and forth there (and have some
ugly expect there). There's not much point for error handling here,
this will be moved to stricter types in a followup CL.
Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-14 21:00:12 +02:00
|
|
|
|
|
|
|
/// Errors that can occur when populating [crate::Directory] messages,
|
|
|
|
/// or parsing [crate::proto::Directory]
|
|
|
|
#[derive(Debug, thiserror::Error, PartialEq)]
|
|
|
|
pub enum DirectoryError {
|
|
|
|
/// Multiple elements with the same name encountered
|
2024-08-16 16:32:20 +02:00
|
|
|
#[error("{:?} is a duplicate name", .0)]
|
|
|
|
DuplicateName(PathComponent),
|
refactor(tvix/castore): remove `name` from Nodes
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.
This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.
It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).
The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.
The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.
The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.
Some further cleanups:
The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)
There's some leftover cleanups to do:
- There should be a from_(sorted_)iter and into_iter in Directory, so
we can construct and deconstruct in one go.
That should also enable us to implement conversions from and to the
proto representation that moves, rather than clones.
- The BuildRequest and PathInfo structs are still proto-based, so we
still do a bunch of conversions back and forth there (and have some
ugly expect there). There's not much point for error handling here,
this will be moved to stricter types in a followup CL.
Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-14 21:00:12 +02:00
|
|
|
/// Node failed validation
|
2024-09-22 21:28:39 +02:00
|
|
|
#[error("invalid node with name {}: {:?}", .0.as_bstr(), .1.to_string())]
|
|
|
|
InvalidNode(bytes::Bytes, ValidateNodeError),
|
2024-08-21 00:19:26 +02:00
|
|
|
#[error("Total size exceeds u64::MAX")]
|
refactor(tvix/castore): remove `name` from Nodes
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.
This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.
It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).
The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.
The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.
The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.
Some further cleanups:
The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)
There's some leftover cleanups to do:
- There should be a from_(sorted_)iter and into_iter in Directory, so
we can construct and deconstruct in one go.
That should also enable us to implement conversions from and to the
proto representation that moves, rather than clones.
- The BuildRequest and PathInfo structs are still proto-based, so we
still do a bunch of conversions back and forth there (and have some
ugly expect there). There's not much point for error handling here,
this will be moved to stricter types in a followup CL.
Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-14 21:00:12 +02:00
|
|
|
SizeOverflow,
|
|
|
|
/// Invalid name encountered
|
2024-08-17 21:00:06 +02:00
|
|
|
#[error("Invalid name: {0}")]
|
|
|
|
InvalidName(PathComponentError),
|
2024-10-18 21:42:04 +02:00
|
|
|
/// This can occur if a protobuf node with a name is passed where we expect
|
|
|
|
/// it to be anonymous.
|
|
|
|
#[error("Name is set when it shouldn't")]
|
|
|
|
NameInAnonymousNode,
|
refactor(tvix/castore): remove `name` from Nodes
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.
This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.
It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).
The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.
The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.
The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.
Some further cleanups:
The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)
There's some leftover cleanups to do:
- There should be a from_(sorted_)iter and into_iter in Directory, so
we can construct and deconstruct in one go.
That should also enable us to implement conversions from and to the
proto representation that moves, rather than clones.
- The BuildRequest and PathInfo structs are still proto-based, so we
still do a bunch of conversions back and forth there (and have some
ugly expect there). There's not much point for error handling here,
this will be moved to stricter types in a followup CL.
Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-14 21:00:12 +02:00
|
|
|
/// Elements are not in sorted order. Can only happen on protos
|
|
|
|
#[error("{:?} is not sorted", .0.as_bstr())]
|
|
|
|
WrongSorting(bytes::Bytes),
|
|
|
|
/// This can only happen if there's an unknown node type (on protos)
|
|
|
|
#[error("No node set")]
|
|
|
|
NoNodeSet,
|
|
|
|
}
|
2023-03-03 22:32:07 +01:00
|
|
|
|
2023-03-25 22:17:23 +01:00
|
|
|
impl From<JoinError> for Error {
|
|
|
|
fn from(value: JoinError) -> Self {
|
|
|
|
Error::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2023-02-12 12:34:15 +01:00
|
|
|
impl From<Error> for Status {
|
|
|
|
fn from(value: Error) -> Self {
|
|
|
|
match value {
|
|
|
|
Error::InvalidRequest(msg) => Status::invalid_argument(msg),
|
|
|
|
Error::StorageError(msg) => Status::data_loss(format!("storage error: {}", msg)),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2023-02-26 22:31:50 +01:00
|
|
|
|
2023-11-13 13:32:24 +01:00
|
|
|
impl From<crate::tonic::Error> for Error {
|
|
|
|
fn from(value: crate::tonic::Error) -> Self {
|
2023-10-12 19:26:52 +02:00
|
|
|
Self::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2024-07-21 00:36:19 +02:00
|
|
|
impl From<redb::Error> for Error {
|
|
|
|
fn from(value: redb::Error) -> Self {
|
|
|
|
Error::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
impl From<redb::DatabaseError> for Error {
|
|
|
|
fn from(value: redb::DatabaseError) -> Self {
|
|
|
|
Error::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
impl From<redb::TableError> for Error {
|
|
|
|
fn from(value: redb::TableError) -> Self {
|
|
|
|
Error::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
impl From<redb::TransactionError> for Error {
|
|
|
|
fn from(value: redb::TransactionError) -> Self {
|
|
|
|
Error::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
impl From<redb::StorageError> for Error {
|
|
|
|
fn from(value: redb::StorageError) -> Self {
|
|
|
|
Error::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
impl From<redb::CommitError> for Error {
|
|
|
|
fn from(value: redb::CommitError) -> Self {
|
|
|
|
Error::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2023-11-18 17:30:55 +01:00
|
|
|
impl From<std::io::Error> for Error {
|
|
|
|
fn from(value: std::io::Error) -> Self {
|
|
|
|
if value.kind() == std::io::ErrorKind::InvalidInput {
|
|
|
|
Error::InvalidRequest(value.to_string())
|
|
|
|
} else {
|
|
|
|
Error::StorageError(value.to_string())
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2023-02-26 22:31:50 +01:00
|
|
|
// TODO: this should probably go somewhere else?
|
|
|
|
impl From<Error> for std::io::Error {
|
|
|
|
fn from(value: Error) -> Self {
|
|
|
|
match value {
|
|
|
|
Error::InvalidRequest(msg) => Self::new(std::io::ErrorKind::InvalidInput, msg),
|
|
|
|
Error::StorageError(msg) => Self::new(std::io::ErrorKind::Other, msg),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|