feat(tvix/store): use rayon to upload chunks concurrently
Look at the data that's written to us, and upload all chunks but the rest in parallel, using rayon. This required moving `upload_chunk` outside the struct, and accepting a ChunkService to use for upload (which it was previously getting from `self.chunk_service`). This doesn't speed up things too much for now, because things are still mostly linear. Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This commit is contained in:
parent
2ef60282b6
commit
510927e43a
5 changed files with 66 additions and 39 deletions
|
@ -8330,6 +8330,10 @@ rec {
|
|||
name = "prost";
|
||||
packageId = "prost";
|
||||
}
|
||||
{
|
||||
name = "rayon";
|
||||
packageId = "rayon";
|
||||
}
|
||||
{
|
||||
name = "sha2";
|
||||
packageId = "sha2 0.10.6";
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue