feat(tvix/store): bump fastcdc, use v2020 version
This switches away from the less canonical "ronomon" version to the implementation as described in the [paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et al., in 2020. That version uses 64-bit hash values and tends to be faster than both the ronomon and v2016 versions, and produces the same chunking as the 2016 version. As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's the recommended choice. The crate also gained support for streaming version of chunkers: https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be useful. Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
This commit is contained in:
parent
fbabcf0420
commit
a40d2dcdcd
4 changed files with 6 additions and 6 deletions
|
@ -157,7 +157,7 @@ impl<
|
|||
|
||||
// initialize a new chunker
|
||||
// TODO: play with chunking sizes
|
||||
let chunker = fastcdc::FastCDC::new(
|
||||
let chunker = fastcdc::v2020::FastCDC::new(
|
||||
&blob_contents,
|
||||
64 * 1024 / 4, // min
|
||||
64 * 1024, // avg
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue