mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(tvix/tools/crunch-v2): init

Browse source 
This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format.
Currently, produced chunks are not preserved, and this purely serves as a way of measuring
compression/deduplication ratios for various chunking and compression parameters.

Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This commit is contained in:
edef 2024-01-17 16:04:03 +00:00
parent e0a1c03b24
commit 4f22203a3a
12 changed files with 15022 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

15
tvix/tools/crunch-v2/default.nix Normal file
View file

@ -0,0 +1,15 @@
{ pkgs, ... }:
let
crates = import ./Cargo.nix {
inherit pkgs;
nixpkgs = pkgs.path;
defaultCrateOverrides = pkgs.defaultCrateOverrides // {
crunch-v2 = prev: {
nativeBuildInputs = (prev.nativeBuildInputs or [ ]) ++ [ pkgs.buildPackages.protobuf ];
};
};
};
in
crates.rootCrate.build