Commit graph

11 commits

Author SHA1 Message Date
Brian Olsen
9af6920478 feat(nix-compat): Add NixDeserialize and NixRead traits
Add a trait for deserializing a type from a daemon worker connection.
This adds the NixDeserialize trait which is kind of like the serde
Deserialize trait in that individual types are meant to implement it
and it can potentially be derived in the future.

The NixDeserialize trait takes something that implements NixRead as
input so that you can among other things mock the reader.

Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-25 15:05:50 +00:00
Florian Klink
25f8ddca6d chore(tvix/tools/crunch-v2): bump deps
h2:
 - https://rustsec.org/advisories/RUSTSEC-2024-0332.html
 - https://rustsec.org/advisories/RUSTSEC-2024-0003.html

mio:
 - https://rustsec.org/advisories/RUSTSEC-2024-0019.html

shlex:
 - https://rustsec.org/advisories/RUSTSEC-2024-0006.html

zerocopy:
 - https://rustsec.org/advisories/RUSTSEC-2024-0006.html

Change-Id: Ia5f6f96378d5f7e77e178dc054f47bda8b766a30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12340
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-24 18:08:08 +00:00
Ilan Joselevich
a8d2d6e97e feat(tvix/tools/crunch-v2): Add crate2nix-check
Change-Id: I922caae2ec3f87a7f3a5dcc8e9d366f0e71ce900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12300
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 19:55:10 +00:00
Florian Klink
e03ea11bad feat(nix-compat/nix_http): init parse_nar[info]_str
This moves the URL component parsing code we had in nar-bridge to
nix-compat.

We change the function signature to return an Option, not a
Result<_, StatusCode>.

This allows returning more appropriate error codes, as we can
ok_or(…) at the callsite, which we now do: on an upload to an
invalid path, we now return "unauthorized", while on a GET/HEAD, we
return "not found".

This also adds support to parse compression suffixes. While not
supported in nar-bridge, other users of nix-compat might very well want
to parse these paths.

Also fix the error message when parsing NAR urls, it mentioned 32, not
52, which is a copypasta error from the narinfo URL parsing code.

Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-21 09:40:32 +00:00
Florian Klink
1d3fce5857 chore(tvix/tools/crunch-v2): bump curve25519-dalek
Fix for RUSTSEC-2024-0344.

Change-Id: I11651ea56e52988ee7cef2e02d0cbf61062f5c7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12213
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-08-16 06:52:15 +00:00
Florian Klink
a0a13126ee chore(tvix/tools/crunch-v2): update Cargo.{lock,nix}
These were out of date.

Change-Id: Ideaf888c2851bb9ec36ae01b11d93165b62cd7a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12209
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-16 06:45:37 +00:00
Ilan Joselevich
716f5624fe chore(tvix/tools/crunch-v2): Regenerate Cargo.nix and filter src
The Cargo.lock also got changed because of crate2nix using --freezed.

Change-Id: I90245bad27f6fee01455557bbccc865301a4f1a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11968
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 16:47:56 +00:00
edef
31d73cd443 refactor(nix-compat/nar/reader): reuse prev_name allocation
We reuse the prev_name allocation for Entry, instead of allocating and
returning a separate Vec.

We encode the `prev_name: None` case as an empty vector, since we don't
allow empty names anyway, and the sorting is equivalent.

Change-Id: I975b37ff873805f5ff099bc82128706891052247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 15:30:46 +00:00
edef
368a11ee0a chore(tvix/tools/crunch-v2): upgrade tokio
This drops the Sized requirement on AsyncReadExt.

Change-Id: I2a89c708922ca528cb493aefd0613f477f648e83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11608
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-05-08 10:19:17 +00:00
Florian Klink
b38be028d9 feat(tvix/tools/crunch-v2): add CLI args
Use clap derive to make the input and output files configurable, as well
as the chunk size parameters.

Change-Id: I02b29126f3bd2c13ba2c6e7e0aa4ff048ff803ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10691
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-01-27 18:40:48 +00:00
edef
4f22203a3a feat(tvix/tools/crunch-v2): init
This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format.
Currently, produced chunks are not preserved, and this purely serves as a way of measuring
compression/deduplication ratios for various chunking and compression parameters.

Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-27 18:23:40 +00:00