Commit graph

33 commits

Author SHA1 Message Date
Brian Olsen
ced05a2bb6 feat(tvix/nix-compat-derive): Add deriver for NixDeserialize
This adds a nix-compat-derive derive crate that implements a deriver
for NixDeserialize implementations. This is to reduce the amount of
code needed to implement deserialization for all the types used by
the Nix daemon protocol.

Change-Id: I484724b550e8a1d5e9adad9555d9dc1374ae95c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12022
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-25 15:05:50 +00:00
Brian Olsen
9af6920478 feat(nix-compat): Add NixDeserialize and NixRead traits
Add a trait for deserializing a type from a daemon worker connection.
This adds the NixDeserialize trait which is kind of like the serde
Deserialize trait in that individual types are meant to implement it
and it can potentially be derived in the future.

The NixDeserialize trait takes something that implements NixRead as
input so that you can among other things mock the reader.

Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-25 15:05:50 +00:00
Florian Klink
e03ea11bad feat(nix-compat/nix_http): init parse_nar[info]_str
This moves the URL component parsing code we had in nar-bridge to
nix-compat.

We change the function signature to return an Option, not a
Result<_, StatusCode>.

This allows returning more appropriate error codes, as we can
ok_or(…) at the callsite, which we now do: on an upload to an
invalid path, we now return "unauthorized", while on a GET/HEAD, we
return "not found".

This also adds support to parse compression suffixes. While not
supported in nar-bridge, other users of nix-compat might very well want
to parse these paths.

Also fix the error message when parsing NAR urls, it mentioned 32, not
52, which is a copypasta error from the narinfo URL parsing code.

Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-21 09:40:32 +00:00
Florian Klink
a259613c76 feat(nix-compat/narinfo/signature): generalize name field
Requiring `name` to be a `&str` means it'll get annoying to pass around
`Signature`, but being able to pass them around in an owned fashion is
kinda a requirement for a stronger typed `PathInfo` struct, where we
want to have full ownership.

Rework the `Signature` struct to become generic over the type of the
`name` field. This means, it becomes possible to have owned versions
of it.

We don't want to impose `String` or `SmolStr` for example, but want to
leave it up to the nix-compat user to decide.

Provide a type alias for the existing `&str` variant (`SignatureRef`),
and use it where we previously used the non-generic `Signature` one.

Add some tests to ensure it's possible to *use* `Signature` with both
`String` and `SmolStr` (but only pull in `smol_str` as dev dependency
for the tests).

Also, add some more docstrings, these were a bit sparse.

Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 19:46:34 +00:00
Ilan Joselevich
f648f17ec3 feat(tvix): Jemalloc -> MiMalloc
Use the faster and newer MiMalloc memory allocator for all endpoints in
the workspace.

Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-08-08 08:06:23 +00:00
Florian Klink
afbe995aa9 chore(tvix): bump to data-encoding 2.6.0
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-01 06:57:01 +00:00
Florian Klink
8089682bb5 fix(tvix/nix-compat): async nar reader requires wire feature
It uses it internally.

Change-Id: I8cb8fc9f567260d57f3a203407333d83beddf537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11719
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-26 19:52:49 +00:00
Florian Klink
cd820d07d7 chore(nix-compat): remove unused test-case and test-generator
This is especially nice, as it allows us getting rid of the other
versions of proc-macro2 and quote.

Change-Id: I9fdd012ee6c0ded3e18ec30504b83ac2032d1390
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11474
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-19 19:22:10 +00:00
Florian Klink
2c884b8bd2 feat(tvix/nix-compat): enable more features by default
There's no point in not enabling these, and rust-analyzer is quite
useless too.

nix-compat users outside of here can still explicitly disable default
features if they don't want to pull in these features.

Change-Id: I8f2f24e7734ad152d1733bc3421098fc91af290a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11455
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-18 12:12:36 +00:00
Florian Klink
45cf7ae657 refactor(tvix/nix-compat): move nar writer to tokio
There's little reason to keep the nar writer using Async{Read,Write}
traits from futures, while everything else async in tvix (and
nix-compat) uses tokio.

Change-Id: I8cd1efcd0dd5bb76471de997603c7b701a5095de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11391
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
36b296609b refactor(tvix/nix-compat): reorganize wire and bytes
Move everything bytes-related into its own module, and re-export
both bytes and primitive in a flat space from wire/mod.rs.

Expose this if a `wire` feature flag is set. We only have `async` stuff
in here.

Change-Id: Ia4ce4791f13a5759901cc9d6ce6bd6bbcca587c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11389
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
839c971a0f docs(tvix/eval): remove link to cargo manifest reference
We don't have it in all Cargo.toml files either.

Change-Id: Ifcb333a39603a7e402ab0068e0f0f1b5af769e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11388
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
eeca2d92e2 feat(tvix/nix-compat): add BytesReader
This adds AsyncRead counterpart for read_bytes.

Change-Id: I751da9944984c7a523abee305f8f8a050e705f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11385
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
f3d8c633f2 feat(tvix/nix-compat/wire): introduce BytesWriter
This deals with writing byte packets of larger sizes to an underlying
AsyncWrite.

Its constructor receives the expected size. It also deals with writing
padding if flush/shutdown is called after writing all the payload.

Change-Id: I8acbf992467f3862ffb8c7d669e8c0c8eced14c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11355
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-04-06 14:12:05 +00:00
Picnoir
21481b02b8 feat(tvix/nix-compat): worker protocol operation parser
Change-Id: I7776635b17c44534223603d28cf59c7eebd976e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11229
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-27 12:05:53 +00:00
Florian Klink
907ecff999 feat(nix-compat/wire): add low-level wire format primitives code
This brings some initial Nix wire format parsing code, used in the nix
daemon protocol, remote store/builder protocol, as well as the NAR
format itself (note we already have more specialized code for the last
one).

Thanks to embr, this code already exists, in
https://codeberg.org/gorgon/gorgon/src/branch/main/nix-daemon/src/wire.rs,
and we can vendor it into here, as EUPL is compatible with GPL (in that
direction).

The code uses the tokio::io Reader and Writer traits, not the ones from
the `futures` crate, as they provide some more convenient `read_u64_le`
functions.

More application-specific parsing code, as well as code to read strings,
or bytes are left out for now, as we want to be be more restrictive
w.r.t allowed max sizes, and need to parse bytes, not strings.

The code slightly diverges, as we have clippy looped into CI.
`Ok(…?)` can be turned into just the inner expression, and
some .and_then can be expressed in a simpler fashion.

Change-Id: Ie3adcb485e9d66786673b1962a08d4e5df3781d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11148
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-03-15 10:23:12 +00:00
Florian Klink
8f804f5d9c chore(tvix/nix-compat): bump zstd dependency
Otherwise this causes conflicts when adding a more recent zstd version
to another crate.

Change-Id: I02d1b1a37f18711c0969cb04e761e967fc4655dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11100
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-09 05:47:03 +00:00
Florian Klink
5a82736122 chore(tvix): bump test-case dep to 3.3.1
Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-01-05 16:43:34 +00:00
Florian Klink
acbb613e61 chore(tvix): switch to upstream futures 0.3.30
The bugs have been fixed,
https://github.com/rust-lang/futures-rs/pull/2801 and
https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended
up in that release.

Change-Id: Iefd990d2d1719b884504093343e54e9c5258e2e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10414
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-24 21:45:04 +00:00
Florian Klink
01a3c87a3a chore(tvix): bump criterion to 0.5
Change-Id: I28904ca23437b4bb745c0eb1f4eb9ae33e09eb5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10244
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 03:59:30 +00:00
Florian Klink
bb18556bf3 feat(tvix/nix-compat): add narinfo::Signature::verify
This adds support for verifying signatures found in NARInfo files.

This still needs to be hooked together with the nix+http[s] backend.

Change-Id: Ic1c8ddbdecfb05cefca2492808388b0f7f3f2637
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10081
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-19 22:06:07 +00:00
edef
2997c1c304 feat(nix-compat/narinfo): turn flags into bitfields
Change-Id: I8b95723444013e97bc6ec8d282c7135b1aede114
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9987
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-10 19:05:36 +00:00
edef
9253bf6632 feat(tvix/nix-compat): add narinfo parsing and serialisation
Change-Id: I72c63414794642ca8d85c3f635f49db888420c40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9852
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 16:08:06 +00:00
edef
6a0a75c8e1 refactor(tvix): condense long bytestrings
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 07:37:26 +00:00
Florian Klink
652afd21b3 feat(nix-compat): add bench for aterm parsing
Change-Id: Ie9416f0046b6ee0a02096a7712cdba9cc84db4f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9733
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Gabriel Doriath Döhler <gabriel.doriath.dohler@ens.fr>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-16 12:26:36 +00:00
Florian Klink
2410f2292f feat(nix-compat/{aterm,derivation}): init parser
This provides a nom-based parser for Nix derivations in ATerm format,
which can be reached via `Derivation::from_aterm_bytes`.

Some of the lower-level ATerm primitives are moved into a (new) aterm
module, and some more higher-level ones that construct derivation-
specific types.

Also, move the escape_bytes function into there, this is a generic ATerm
thing.

Change-Id: I2b03b8a1461c7ea2fcb8640c2fc3d1fa3ea719fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9730
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-16 12:23:00 +00:00
Florian Klink
8a0fa69c6c feat(tvix/nix-compat/nixhash/NixHashWithMode): from_algo_mode_hash
Change-Id: Ieae628fab1926a498ae5c3eb27df1b722e6151d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9728
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-15 15:10:07 +00:00
edef
8b35d97b4b feat(tvix/nix-compat): add nar::writer::async
Change-Id: Ib962f889836659fd278beec461ee329b00e64ea5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9615
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 17:13:27 +00:00
Florian Klink
7c9a52e3ee chore(tvix): remove some unused crate dependencies
Change-Id: I31b0e7be3a2ebc268c6f4045ba282e8fb1c72a64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9375
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:24 +00:00
Florian Klink
79531c3dab refactor(tvix/nix-compat): support non-unicode Derivations
Derivations can have non-unicode strings in their env values, so the
ATerm representations are not necessarily String anymore, but Vec<u8>.

Change-Id: Ic23839471eb7f68d9c3c30667c878830946b6607
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8990
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-07-31 21:41:38 +00:00
Florian Klink
b55d1f97ce refactor(tvix/nix-compat): -derivation::Hash, +NixHash
This stops using our own custom Hash structure, which was mostly only
used because we had to parse the JSON representation somehow.

Since cl/8217, there's a `NixHash` struct, which is better suited to
hold this data. Converting the format requires a bit of serde labor
though, but that only really matters when interacting with JSON
representations (which we mostly don't).

Change-Id: Idc5ee511e36e6726c71f66face8300a441b0bf4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8304
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-15 11:50:12 +00:00
Florian Klink
2d24c5f260 refactor(tvix/nix-compat): absorb //tvix/derivation
Put this in its src/derivation.

Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:39 +00:00
Florian Klink
c27bacd905 refactor(tvix): introduce nix-compat crate
Move nixbase32 and store_path into this.

This allows //tvix/cli to not pull in //tvix/store for now.

Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 13:26:18 +00:00