Will Dietz
c6209030c4
compression: make parallel sink separate class
2018-02-11 13:50:28 -06:00
Will Dietz
a0bdc96726
compression: print warning if parallel requested but not supported
2018-02-11 13:03:47 -06:00
Will Dietz
5a082ad15a
configure.ac: check if lzma has MT support, fix deb build/etc.
2018-02-09 21:02:25 -06:00
AmineChikhaoui
55ecdfe2a8
make multi threaded compression configurable and use single threaded
...
by default.
2018-02-07 17:54:08 +01:00
AmineChikhaoui
9d1e22f743
set block size to 0 to let the lzma lib choose the right one, add
...
some comments about possible improvements wrt memory usage/threading.
2018-02-07 11:18:55 +01:00
AmineChikhaoui
bc7e3a4dd6
support multi threaded xz encoder, this might be particularly useful in
...
the case of hydra where the overhead of single threaded encoding is more
noticeable e.g most of the time spent in "Sending inputs"/"Receiving outputs"
is due to compression while the actual upload to the binary cache seems
to be negligible.
2018-02-06 22:42:02 +01:00
Will Dietz
9dd2b8ac7b
use libbrotli directly when available
...
* Look for both 'brotli' and 'bro' as external command,
since upstream has renamed it in newer versions.
If neither are found, current runtime behavior
is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
to ensure interrupts are processed in a timely manner.
Testing shows negligible performance impact.
(Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Jörg Thalheim
2fd8f8bb99
Replace Unicode quotes in user-facing strings by ASCII
...
Relevant RFC: NixOS/rfcs#4
$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra
0a7ca24c26
Fix xz decompression
...
Fixes #1285 .
2017-03-22 11:53:33 +01:00
Eelco Dolstra
895a74a814
LocalFSStore::getBuildLog(): Handle corrupted logs
2017-03-21 19:23:35 +01:00
Eelco Dolstra
ed5c0f69f2
Don't hang in decompression if bzip2 data ends prematurely
2017-03-21 19:23:35 +01:00
Eelco Dolstra
25dff2b7db
runProgram(): Distinguish between empty input and no input
...
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
2017-03-15 16:50:19 +01:00
Eelco Dolstra
8b1d65bebe
S3BinaryCacheStore: Support compression of narinfo and log files
...
You can now set the store parameter "text-compression=br" to compress
textual files in the binary cache (i.e. narinfo and logs) using
Brotli. This sets the Content-Encoding header; the extension of
compressed files is unchanged.
You can separately specify the compression of log files using
"log-compression=br". This is useful when you don't want to compress
narinfo files for backward compatibility.
2017-03-15 16:49:28 +01:00
Eelco Dolstra
2691498b5c
Fix assertion failure on SIGINT
...
nix: src/libutil/compression.cc:142: virtual nix::XzSink::~XzSink(): Assertion `finished' failed.
2017-03-15 16:49:06 +01:00
Eelco Dolstra
e8186085e0
Add support for brotli compression
...
Build logs on cache.nixos.org are compressed using Brotli (since this
allows them to be decompressed automatically by Chrome and Firefox),
so it's handy if "nix log" can decompress them.
2017-03-15 16:49:06 +01:00
Eelco Dolstra
215b70f51e
Revert "Get rid of unicode quotes ( #1140 )"
...
This reverts commit f78126bfd6
2016-11-26 00:38:01 +01:00
Guillaume Maudoux
f78126bfd6
Get rid of unicode quotes ( #1140 )
2016-11-25 15:48:27 +01:00
Eelco Dolstra
0d4a10e910
Do compression in a sink
2016-05-04 16:16:53 +02:00
Eelco Dolstra
538a64e8c3
Add a Store::addToStore() variant that accepts a NAR
...
As a side effect, this ensures that signatures are propagated when
copying paths between stores.
Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra
33de2bc080
Fix segfault in xz compression/decompression
2016-05-04 11:50:12 +02:00
Eelco Dolstra
4dde0b0562
BinaryCacheStore: Support bzip2 compression
2016-04-29 17:43:37 +02:00
Eelco Dolstra
5acb691402
BinaryCacheStore: Support "none" compression method
2016-04-29 17:02:57 +02:00
Eelco Dolstra
58c84cda3b
Make compression interruptible
2016-04-22 18:15:02 +02:00
Eelco Dolstra
d1b0909894
BinaryCacheStore::readFile(): Return a shared_ptr to a string
...
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
2016-04-15 15:39:48 +02:00
Eelco Dolstra
eff5021eaa
Add xz compression function
...
This is used by the Hydra queue runner, but since it may also be
useful for the C++ rewrite of nix-push, I'm putting it here.
2016-02-15 21:45:56 +01:00
Eelco Dolstra
bfdacb712c
decompressXZ: Ensure that lzma_end() is called
...
Otherwise we might leak memory.
2016-02-15 21:26:28 +01:00
FrankHB
607d91517e
Fixed unresolved 'BUFSIZ'
...
`BUFSIZ` is defined in header `<cstdio>`.
2016-01-05 13:33:40 +01:00
Eelco Dolstra
1f735a3440
<nix/fetchurl.nix>: Support xz-compressed NARs
2015-10-30 12:34:30 +01:00
