We also use S in other places in the same file, but that's for the
string-like references.
SP is now consistently used as the type parameter for StorePath<_> (and
build_output_path) gets support for it).
By being a bit more careful in the order of assignments in nix-compat/
src/derivation, we can nudge the compiler to use the type we want.
Change-Id: Ia7c298e110dff98d3b113d2388674ce9e22b80e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12590
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
This switches the PathInfoService trait from using the proto-derived
PathInfo struct to a more restrictive struct, and updates all
implementations to use it.
It removes a lot of the previous conversion and checks, as invalid
states became nonrepresentable, and validations are expressed on the
type level.
PathInfoService implementations consuming protobuf need to convert and
do the verification internally, and can only return the strongly typed
variant.
The nix_compat::narinfo::NarInfo conversions for the proto PathInfo
are removed, we only keep a version showing a NarInfo representation for
the strong struct.
Converting back to a PathInfo requires the root node now, but is
otherwise trivial, so left to the users.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is useful for building other NAR writers which use custom (async or optimized) I/O to write the blob parts of the NAR.
Change-Id: I447c09914fb0c99044e2fa910d4213660dc51c64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12437
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We always read the length before returning a BytesReader, so len()
cannot be called before the length is read.
Change-Id: Ifa1cbc2c0923dd24d59e0af5e135ab8ed7314aa9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12453
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This will later be used to allow the user to access the original writer.
Change-Id: I1734600c5d580d18b16727f892cd2f6335cc9459
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12442
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In https://cl.tvl.fyi/c/depot/+/12389 I accidentally deleted some
comments. Also fixed some useless no-default-features with futures crate, making it match the previous feature combination.
Change-Id: I72bb2cfb88719ff2f8812d90193de2bd49149cce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12395
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
From now on we will add the dependencies and their version in the root
Cargo.toml and in order to enable the dependency for a workspace member
we set `workspace = true` in the member's Cargo.toml.
Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This updates all the dependencies and their "minimum" versions in
Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s
`cargo upgrade` command that will eventually be merged into `cargo
update`.
Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
The external feature flag was there because I couldn't find a way to
refer to crate and nix-compat with the same name so that the generated
code could be the same.
In essence `use nix_compat::nix_daemon:🇩🇪:NixDeserialize` is an error
when used inside nix_compat crate.
So my best fix was the external feature flag until I found the solution
used here which also removes the flag completely.
Change-Id: Ia3e89c6c350c3fb22ca87f974a39c21542aae152
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12376
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
When only the wire feature was enabled two methods were unused and so
would faild `cargo check --no-default-features --features wire`.
This feature flags those two methods on async feature since that is the
only place they are used.
Change-Id: I6ec18a670e3c6e3ecee8d1417c99f1a5084e0ae7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12346
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
This adds a nix-compat-derive derive crate that implements a deriver
for NixDeserialize implementations. This is to reduce the amount of
code needed to implement deserialization for all the types used by
the Nix daemon protocol.
Change-Id: I484724b550e8a1d5e9adad9555d9dc1374ae95c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12022
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Add a trait for deserializing a type from a daemon worker connection.
This adds the NixDeserialize trait which is kind of like the serde
Deserialize trait in that individual types are meant to implement it
and it can potentially be derived in the future.
The NixDeserialize trait takes something that implements NixRead as
input so that you can among other things mock the reader.
Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This adds the MIME types Nix uses when uploading files to a S3 bucket /
HTTP endpoint. It ignores them when retrieving.
Change-Id: I75fa96d5e53349f0470bbe8cbba93f961fbd5f05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12266
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This moves the URL component parsing code we had in nar-bridge to
nix-compat.
We change the function signature to return an Option, not a
Result<_, StatusCode>.
This allows returning more appropriate error codes, as we can
ok_or(…) at the callsite, which we now do: on an upload to an
invalid path, we now return "unauthorized", while on a GET/HEAD, we
return "not found".
This also adds support to parse compression suffixes. While not
supported in nar-bridge, other users of nix-compat might very well want
to parse these paths.
Also fix the error message when parsing NAR urls, it mentioned 32, not
52, which is a copypasta error from the narinfo URL parsing code.
Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Similar to how cl/12253 already did this for `Signature`, we apply the
same logic to `StorePath`.
`StorePathRef<'a>'` is now a `StorePath<&'a str>`, and there's less
redundant code for the two different implementation.
`.as_ref()` returns a `StorePathRef<'_>`, `.to_owned()` gives a
`StorePath<String>` (for now).
I briefly thought about only publicly exporting `StorePath<String>`
as `StorePath`, but the diff is not too large and this will make it
easier to gradually introduce more flexibility in which store paths to
accept.
Also, remove some silliness in `StorePath::from_absolute_path_full`,
which now doesn't allocate anymore.
Change-Id: Ife8843857a1a0a3a99177ca997649fd45b8198e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12258
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Don't call functions bstr or str when they return BString or String,
it's confusing.
Rename them to `string` and `bytes`. We might be able to generalize over
this being BString or Vec<u8> later.
Change-Id: I8198551ed3ba1cfc479bf7e3cbbc13a426faf4c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12257
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This implements a simple DFS locator in listing structures.
It is interoperable with the Rust standard library paths, we also build our
own errors to restrict path values to reasonable secure defaults, e.g.
relative paths with no `..` component.
Tests are added for this new feature for a positive and a negative
check.
In addition, a path validation test was added. The Windows-style prefix
is gated on the Windows platform as UNIX does not parse `C:\\` as a
`Component::Prefix(_)` but as a `Component::Normal(_)`.
Change-Id: Iae2a80bebd8138e41af94aa7d09f2842c3c5a786
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12255
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Requiring `name` to be a `&str` means it'll get annoying to pass around
`Signature`, but being able to pass them around in an owned fashion is
kinda a requirement for a stronger typed `PathInfo` struct, where we
want to have full ownership.
Rework the `Signature` struct to become generic over the type of the
`name` field. This means, it becomes possible to have owned versions
of it.
We don't want to impose `String` or `SmolStr` for example, but want to
leave it up to the nix-compat user to decide.
Provide a type alias for the existing `&str` variant (`SignatureRef`),
and use it where we previously used the non-generic `Signature` one.
Add some tests to ensure it's possible to *use* `Signature` with both
`String` and `SmolStr` (but only pull in `smol_str` as dev dependency
for the tests).
Also, add some more docstrings, these were a bit sparse.
Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
.ls files are useful to seek in a NAR without parsing it entirely.
The responsibility of validating the files is on the caller.
Change-Id: I5d1da28b5479c38f20ca5babe60e362a2217c9ea
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12196
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
It's a `[u8; SIGNATURE_LENGTH]` type alias, and conveys what we're
accepting or returning a bit nicer.
Change-Id: I974cd97d56d383e51417eb0f26e1431a05711922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12252
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
When using the runTests feature of crate2nix the derivation that runs the
tests is put into passthru.test but all default.nix files for Rust crates
in Tvix threw that away.
This commit retains passthru so that you can get access to the test
derivation.
Change-Id: I8b7b7db57a49069348f08c12c00a3b1a41a0c05b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12215
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Use the faster and newer MiMalloc memory allocator for all endpoints in
the workspace.
Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
The decode function didn't check that the input had a valid length and
so would panic when given input with invalid length.
Change-Id: Ie27d006b8fe20f005b4a47a1763821a61e9a95c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12051
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: Brian Olsen <me@griff.name>
This is the `{fixed,fixed:r,text}:{sha*,md5}` prefix used in various
string representations.
Factor that code out, and use it in the two places it can be used in.
Change-Id: Ic9555fa9e1884198d435e55c7f630b8d3ba2a032
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12041
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
This adds a generic `SigningKey` struct that can be used to sign
NARInfos with signers.
It also includes tooling to parse keypairs from bytes generated by Nix,
returning a specialized ed25519_dalek variant.
Change-Id: Ic9780c370939af54e7177c93cde3321adf189fc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12014
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Align these with the way it's called in the ed25519 crates.
Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
We were wrongly using `buf.initialized()` which contains more than the
filled portion to compute the number of bytes read during a poll call.
This made us go into the trailer reading state too early and finally
failing due to invalid trailer data.
Fixes b/405.
Co-authored-by: Florian Klink <flokli@flokli.de>
Change-Id: I66ba6e2116389e6b97305f85c4e0922195000e45
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11978
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
See https://github.com/NixOS/nixpkgs/pull/323753 for details.
Changes:
* git: temporarily comment out dottime patch (it doesn't apply, but it's not critical)
* third-party/cgit: use an older git version where dottime patch still applies
* 3p/crate2nix: remove crate2nix patches included in latest release
* tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs)
* tvix: regenerate Cargo.nix
* tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function
* tvix/nix-compat: remove redundant trait bounds
* tvix/glue: use clone_into() to set drv.{builder,system}
* tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327
* toold/depotfmt: expose treefmt config as passthru
* tools/crate2nix: undo some more hacks in the crate2nix-check drv
Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c
Co-Authored-By: Florian Klink <flokli@flokli.de>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Otherwise these are not accessible after parsing if parsed from another
crate.
Change-Id: Iba7b442722a2eb1d37c2c496a45aa559aa49c1f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11889
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This allows parsing files like `/etc/nix/nix.conf` into the `NixConfig`
struct.
Change-Id: I90b25f43c429dd56127500ff5068e83852adee13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11888
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
We want to use this in another struct that we want to compare and clone.
Change-Id: I5c11d589a4922d85e190e1a9199468411016ff1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11887
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Context: https://github.com/la10736/rstest/issues/256
Cargo will now cause a rebuild whenever a new test case file is added.
Previsouly running `cargo test` after adding a new test case resulted in
that case not being picked up and ignored.
Change-Id: Ibfc420b5bfe3f3ee41d3ebd3fb9d248819fa6ed9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Closes: https://b.tvl.fyi/issues/401
With this change all crate features (and their combinations) will be built and
tested in CI.
From now on, when adding/removing a Cargo feature for a crate,
you will want to add it to the features power set that gets tested in CI.
For each crate there's a default.nix with a `mkFeaturePowerset` invocation,
modify the list to include/remove the feature.
Note that you don't want to add "collection" features,
such as `fs` for tvix-[ca]store or `default`.
Change-Id: I966dde1413d057770787da3296cce9c1924570e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11717
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Makes the following build:
`cargo test --no-default-features --features async`
`cargo test --no-default-features --features wire`
Change-Id: I47ba0c944f08895f67ed3b861706ef2e4ba384b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11739
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We reuse the prev_name allocation for Entry, instead of allocating and
returning a separate Vec.
We encode the `prev_name: None` case as an empty vector, since we don't
allow empty names anyway, and the sorting is equivalent.
Change-Id: I975b37ff873805f5ff099bc82128706891052247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
For small bytestrings (like NAR names), we can read into a preallocated
fixed-size buffer, instead of allocating a Vec every time.
Change-Id: Id8da9e9cea99c814361230c0ec02606b731c79a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11606
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't actually build up names in place here, so we don't need a
capacity field. Saves 8 bytes.
Change-Id: Icb01b45561e28fd525b726612f56d4640bc834c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11604
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
If our underlying reader supports AsyncBufRead, then we can too.
Change-Id: If4b948c983400ca591c1c475bbcf7dc00d562040
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11545
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Even if we have an aligned (or zero) size (and thus have no padding),
we may still have a non-padding trailer.
Since an exact read of only the user data bytes must always read the
trailer bytes as well, this has to happen for aligned sizes too.
For zero-sized trailers, we solve this by reading immediately, since no
read is required of the user at all.
user_len thus becomes a NonZeroU64, and the definition of "body data"
is adjusted to always exclude the final block.
Change-Id: I2c6bb51270fb22ad72a65076a3468673ab56ef68
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11597
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We're using this in the NAR reader now.
Change-Id: I28f17b1ccedd62ffcaf2fa32b517f16bcd036d94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11603
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We specifically structured the code this way to allow using
`this.filled` as-is, so we should use it.
Change-Id: I7e11bddceb4d5f37b1dd4c453a9d53b85fc1f6c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11602
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is a first cut at the async NAR reader, with some rough edges.
Poisoning is left unimplemented for now, pending future work.
Change-Id: Ifaafe0581a5e0e165a13357b909fb441f7bd8bab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11524
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This allows using BytesReader with a custom tag, eg the closing parens
for the NAR reader.
No public constructor is provided for custom-tagged readers, since this
feature isn't public API.
Change-Id: I82e73d064edc4b6783ead1d6fe46a5b35f45c844
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11543
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This documents which Nix version uses which daemon protocol version so that
any implementor of the protocol can make informed choices about how far back to
support legacy.
Change-Id: If2fc11702f7f731816d51928efc9ea9d939e1c00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11579
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
