Commit graph

4 commits

Author SHA1 Message Date
Florian Klink
ead113cdfc feat(tvix/store/import): use StreamCDC instead of blobwriter
This seems to be way faster.

Change-Id: Ica7cee95d108c51fe67365f07366634ddbbfa060
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8246
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-11 12:54:07 +00:00
Florian Klink
510927e43a feat(tvix/store): use rayon to upload chunks concurrently
Look at the data that's written to us, and upload all chunks but the
rest in parallel, using rayon. This required moving `upload_chunk`
outside the struct, and accepting a ChunkService to use for upload
(which it was previously getting from `self.chunk_service`).

This doesn't speed up things too much for now, because things are still
mostly linear.

Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 11:53:29 +00:00
Florian Klink
2ef60282b6 feat(tvix/store/import): make sure entries are sorted
The Directory service does already reject inserting invalid (wrongly
sorted) Directory messages, but our test case didn't provoke it.

Change-Id: I228e201925e8999186659a2d8da0118db184d9ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8167
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 10:58:55 +00:00
Florian Klink
b29d1ae372 feat(tvix/store): add import::import_path
This imports the contents at a given Path into the tvix store.

It doesn't register the contents at a Path in the store itself, that's up
to the PathInfoService.

Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-03-10 10:58:55 +00:00