There's more than one result in there.
Change-Id: I5d519db51fda050ed293bfb52215a643882e0116
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12879
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
This operation is particularly used when invoking the following
nix commands:
```
nix-store --add-fixed some-path
nix-store --add-fixed --recursive some-path
```
Change-Id: I0f9b129c838c00e10415881f1e6e0d7bc1d7a3a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12800
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
When sending nars over the wire to the nix-daemon, nix protocol versions
>= 1.23 use this framing protocol.
This change implements an AsyncRead for this protocol, to be used in
AddToStoreNar and any other operations when necessary.
Change-Id: I5f7972fe1c9ea145780bf449321bd3efeb833d18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12814
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
When sending nars over the wire to the nix-daemon, nix protocol versions
1.21 to 1.23 use this framing protocol.
This change implements an AsyncRead for this protocol, to be used in
AddToStoreNar and any other operations when necessary.
Change-Id: I571f1adbb2343c14c98503d1a2c12eea4c783ec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12813
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
These are required to support certain nix's local-overlay store
operations, it's safer to return empty results for these
operations than failing with "operation not implemented" errors.
Change-Id: Ic9b69d75dd52af5a826bfb6a8b283b082a0f6bcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This change contains 2 improvements to the tvix-store copy command:
1. Allows reading the reference graph from stdin, using `-` argument
2. Supports json representation produced by `nix path-info --json`
command.
In general it makes is easier and faster to import arbitrary closures
from an existing nix store with e.g the following command:
```
nix path-info ./result --json --closure-size --recursive | \
jq -s '{closure: add}' | \
tvix-store copy -
```
Change-Id: Id6eea2993da233ecfbdc186f1a8c37735b686264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12765
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
By manually writing out the async function, and moving the owned String
we can avoid for S to be Sync in the NixSerialize for StorePath<S> impl.
Co-Authored-By: edef@edef.eu
Change-Id: I8427b38d2bd61de2818088728cbad79cda69d17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12764
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Implement PartialEq/Eq ourselves instead of deriving, by proxying to
name.as_ref() (and digest of course).
Also implement Hash on our own, clippy doesn't like this to be derived,
while Eq/PartialEq is not.
Change-Id: Idbe289a23ba3bc8dabf893d4d8752792ae2778c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12744
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
This change includes only the basic nix handshake protocol handling and
sets up a client session. The only supported operation at this point is
SetOptions.
Additional operations will be implemented in subsequent cls.
Change-Id: I3eccd9e0ceb270c3865929543c702f1491768852
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12743
Autosubmit: Vladimir Kryachko <v.kryachko@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
This groups most `wire` feature gated logic into a single module.
The nix_daemon module will be gated by a feature that adds
nix-compat-derive as a dependency.
All of this is a way to break the crate2nix dependency cycle between
nix-compat and nix-compat-derive(which depends on nix-compat for its
doctests).
Change-Id: I95938a6f280c11967371ff21f8b5a19e6d3d3805
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12761
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This change implements the serialization part that is needed to
implement the nix daemon protocol. Previously was add deserialization
and derivers for that and this then adds the other part of that equation
so that you can write types that can then be read using deserialization.
Change-Id: I2917de634980a93822a4f5a8ad38897b9ce16d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12729
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Existing handshake behavior assumed that the server version is always
at least as new as the client. Meaning that the client's version was
always picked the handshake details as well as for further communication
This change removes that assumption and correctly uses
min(server_version, client_version).
Change-Id: Ia5dad4613dd5f69a0aeb6c9d86982f1f36fe1a4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12722
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is the only (remaining) occurence of it, and not really
more code than just calling store_path::build_ca_path with
`CAHash::Nar(NixHash::Sha256(…))`, especially considering we need the
CAHash in the PathInfo struct later anyways - so let's remove this
function.
Change-Id: Ia82212086062c366e0280ca0823d9e68a3f91d3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12632
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
These are not necessarily strings, and making it paths allows us to stop
converting them to lossy strings.
Change-Id: I11366c721dc5da1778aafe89092a1966b5a43178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12617
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Make this generic on the StorePath<SP> that's being used, similar to the
other functions in there.
Change-Id: I453d1fd3749053d4e5aca156abc18da1f95ca264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12616
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
This is now supported in the standard library via std::sync::LazyLock, but
requires some manual shuffling around of code.
I found at least one dead variable along the way, which I deleted.
Change-Id: I8600c87c49078fb5ff72671994c77b919259e67b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12608
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
We also use S in other places in the same file, but that's for the
string-like references.
SP is now consistently used as the type parameter for StorePath<_> (and
build_output_path) gets support for it).
By being a bit more careful in the order of assignments in nix-compat/
src/derivation, we can nudge the compiler to use the type we want.
Change-Id: Ia7c298e110dff98d3b113d2388674ce9e22b80e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12590
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
This switches the PathInfoService trait from using the proto-derived
PathInfo struct to a more restrictive struct, and updates all
implementations to use it.
It removes a lot of the previous conversion and checks, as invalid
states became nonrepresentable, and validations are expressed on the
type level.
PathInfoService implementations consuming protobuf need to convert and
do the verification internally, and can only return the strongly typed
variant.
The nix_compat::narinfo::NarInfo conversions for the proto PathInfo
are removed, we only keep a version showing a NarInfo representation for
the strong struct.
Converting back to a PathInfo requires the root node now, but is
otherwise trivial, so left to the users.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is useful for building other NAR writers which use custom (async or optimized) I/O to write the blob parts of the NAR.
Change-Id: I447c09914fb0c99044e2fa910d4213660dc51c64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12437
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We always read the length before returning a BytesReader, so len()
cannot be called before the length is read.
Change-Id: Ifa1cbc2c0923dd24d59e0af5e135ab8ed7314aa9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12453
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This will later be used to allow the user to access the original writer.
Change-Id: I1734600c5d580d18b16727f892cd2f6335cc9459
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12442
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In https://cl.tvl.fyi/c/depot/+/12389 I accidentally deleted some
comments. Also fixed some useless no-default-features with futures crate, making it match the previous feature combination.
Change-Id: I72bb2cfb88719ff2f8812d90193de2bd49149cce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12395
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
From now on we will add the dependencies and their version in the root
Cargo.toml and in order to enable the dependency for a workspace member
we set `workspace = true` in the member's Cargo.toml.
Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This updates all the dependencies and their "minimum" versions in
Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s
`cargo upgrade` command that will eventually be merged into `cargo
update`.
Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
The external feature flag was there because I couldn't find a way to
refer to crate and nix-compat with the same name so that the generated
code could be the same.
In essence `use nix_compat::nix_daemon:🇩🇪:NixDeserialize` is an error
when used inside nix_compat crate.
So my best fix was the external feature flag until I found the solution
used here which also removes the flag completely.
Change-Id: Ia3e89c6c350c3fb22ca87f974a39c21542aae152
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12376
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
When only the wire feature was enabled two methods were unused and so
would faild `cargo check --no-default-features --features wire`.
This feature flags those two methods on async feature since that is the
only place they are used.
Change-Id: I6ec18a670e3c6e3ecee8d1417c99f1a5084e0ae7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12346
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This adds a nix-compat-derive derive crate that implements a deriver
for NixDeserialize implementations. This is to reduce the amount of
code needed to implement deserialization for all the types used by
the Nix daemon protocol.
Change-Id: I484724b550e8a1d5e9adad9555d9dc1374ae95c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12022
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Add a trait for deserializing a type from a daemon worker connection.
This adds the NixDeserialize trait which is kind of like the serde
Deserialize trait in that individual types are meant to implement it
and it can potentially be derived in the future.
The NixDeserialize trait takes something that implements NixRead as
input so that you can among other things mock the reader.
Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This adds the MIME types Nix uses when uploading files to a S3 bucket /
HTTP endpoint. It ignores them when retrieving.
Change-Id: I75fa96d5e53349f0470bbe8cbba93f961fbd5f05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12266
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This moves the URL component parsing code we had in nar-bridge to
nix-compat.
We change the function signature to return an Option, not a
Result<_, StatusCode>.
This allows returning more appropriate error codes, as we can
ok_or(…) at the callsite, which we now do: on an upload to an
invalid path, we now return "unauthorized", while on a GET/HEAD, we
return "not found".
This also adds support to parse compression suffixes. While not
supported in nar-bridge, other users of nix-compat might very well want
to parse these paths.
Also fix the error message when parsing NAR urls, it mentioned 32, not
52, which is a copypasta error from the narinfo URL parsing code.
Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Similar to how cl/12253 already did this for `Signature`, we apply the
same logic to `StorePath`.
`StorePathRef<'a>'` is now a `StorePath<&'a str>`, and there's less
redundant code for the two different implementation.
`.as_ref()` returns a `StorePathRef<'_>`, `.to_owned()` gives a
`StorePath<String>` (for now).
I briefly thought about only publicly exporting `StorePath<String>`
as `StorePath`, but the diff is not too large and this will make it
easier to gradually introduce more flexibility in which store paths to
accept.
Also, remove some silliness in `StorePath::from_absolute_path_full`,
which now doesn't allocate anymore.
Change-Id: Ife8843857a1a0a3a99177ca997649fd45b8198e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12258
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Don't call functions bstr or str when they return BString or String,
it's confusing.
Rename them to `string` and `bytes`. We might be able to generalize over
this being BString or Vec<u8> later.
Change-Id: I8198551ed3ba1cfc479bf7e3cbbc13a426faf4c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12257
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This implements a simple DFS locator in listing structures.
It is interoperable with the Rust standard library paths, we also build our
own errors to restrict path values to reasonable secure defaults, e.g.
relative paths with no `..` component.
Tests are added for this new feature for a positive and a negative
check.
In addition, a path validation test was added. The Windows-style prefix
is gated on the Windows platform as UNIX does not parse `C:\\` as a
`Component::Prefix(_)` but as a `Component::Normal(_)`.
Change-Id: Iae2a80bebd8138e41af94aa7d09f2842c3c5a786
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12255
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Requiring `name` to be a `&str` means it'll get annoying to pass around
`Signature`, but being able to pass them around in an owned fashion is
kinda a requirement for a stronger typed `PathInfo` struct, where we
want to have full ownership.
Rework the `Signature` struct to become generic over the type of the
`name` field. This means, it becomes possible to have owned versions
of it.
We don't want to impose `String` or `SmolStr` for example, but want to
leave it up to the nix-compat user to decide.
Provide a type alias for the existing `&str` variant (`SignatureRef`),
and use it where we previously used the non-generic `Signature` one.
Add some tests to ensure it's possible to *use* `Signature` with both
`String` and `SmolStr` (but only pull in `smol_str` as dev dependency
for the tests).
Also, add some more docstrings, these were a bit sparse.
Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
.ls files are useful to seek in a NAR without parsing it entirely.
The responsibility of validating the files is on the caller.
Change-Id: I5d1da28b5479c38f20ca5babe60e362a2217c9ea
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12196
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
It's a `[u8; SIGNATURE_LENGTH]` type alias, and conveys what we're
accepting or returning a bit nicer.
Change-Id: I974cd97d56d383e51417eb0f26e1431a05711922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12252
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
When using the runTests feature of crate2nix the derivation that runs the
tests is put into passthru.test but all default.nix files for Rust crates
in Tvix threw that away.
This commit retains passthru so that you can get access to the test
derivation.
Change-Id: I8b7b7db57a49069348f08c12c00a3b1a41a0c05b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12215
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Use the faster and newer MiMalloc memory allocator for all endpoints in
the workspace.
Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
The decode function didn't check that the input had a valid length and
so would panic when given input with invalid length.
Change-Id: Ie27d006b8fe20f005b4a47a1763821a61e9a95c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12051
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: Brian Olsen <me@griff.name>
This is the `{fixed,fixed:r,text}:{sha*,md5}` prefix used in various
string representations.
Factor that code out, and use it in the two places it can be used in.
Change-Id: Ic9555fa9e1884198d435e55c7f630b8d3ba2a032
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12041
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
This adds a generic `SigningKey` struct that can be used to sign
NARInfos with signers.
It also includes tooling to parse keypairs from bytes generated by Nix,
returning a specialized ed25519_dalek variant.
Change-Id: Ic9780c370939af54e7177c93cde3321adf189fc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12014
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Align these with the way it's called in the ed25519 crates.
Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
We were wrongly using `buf.initialized()` which contains more than the
filled portion to compute the number of bytes read during a poll call.
This made us go into the trailer reading state too early and finally
failing due to invalid trailer data.
Fixes b/405.
Co-authored-by: Florian Klink <flokli@flokli.de>
Change-Id: I66ba6e2116389e6b97305f85c4e0922195000e45
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11978
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
