Commit graph

3 commits

Author SHA1 Message Date
edef
368a11ee0a chore(tvix/tools/crunch-v2): upgrade tokio
This drops the Sized requirement on AsyncReadExt.

Change-Id: I2a89c708922ca528cb493aefd0613f477f648e83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11608
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-05-08 10:19:17 +00:00
Florian Klink
b38be028d9 feat(tvix/tools/crunch-v2): add CLI args
Use clap derive to make the input and output files configurable, as well
as the chunk size parameters.

Change-Id: I02b29126f3bd2c13ba2c6e7e0aa4ff048ff803ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10691
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-01-27 18:40:48 +00:00
edef
4f22203a3a feat(tvix/tools/crunch-v2): init
This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format.
Currently, produced chunks are not preserved, and this purely serves as a way of measuring
compression/deduplication ratios for various chunking and compression parameters.

Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-27 18:23:40 +00:00