Commit graph

3172 commits

Author SHA1 Message Date
Eelco Dolstra
eba840c8a1
Linux sandbox: Use /build instead of /tmp as $TMPDIR
There is a security issue when a build accidentally stores its $TMPDIR
in some critical place, such as an RPATH. If
TMPDIR=/tmp/nix-build-..., then any user on the system can recreate
that directory and inject libraries into the RPATH of programs
executed by other users. Since /build probably doesn't exist (or isn't
world-writable), this mitigates the issue.
2017-05-04 16:57:03 +02:00
Eelco Dolstra
2da6a42448
nix dump-path: Add
This is primarily useful for extracting NARs from other stores (like
binary caches), which "nix-store --dump" cannot do.
2017-05-04 14:21:22 +02:00
Eelco Dolstra
72fb2a7edc
Fix build on gcc 4.9
http://hydra.nixos.org/build/52408843
2017-05-03 16:08:48 +02:00
Eelco Dolstra
08355643ab
nix-shell: Implement passAsFile 2017-05-03 15:01:15 +02:00
Eelco Dolstra
782c0bff45
nix eval: Add a --raw flag
Similar to "jq -r", this prints the evaluation result (which must be a
string value) unquoted.
2017-05-03 14:08:18 +02:00
Eelco Dolstra
cef8c169b1
Fix "nix ... --all"
When "--all" is used, we should not fill in a default installable.
2017-05-02 15:46:10 +02:00
Eelco Dolstra
c5bea16611
LocalStoreAccessor: Fix handling of diverted stores 2017-05-02 15:46:09 +02:00
Eelco Dolstra
7f6837a0f6
Replace $NIX_REMOTE_SYSTEMS with an option "builder-files"
Also, to unify with hydra-queue-runner, allow it to be a list of
files.
2017-05-02 15:46:09 +02:00
Eelco Dolstra
cd4d2705ec
build-remote: Fix fallback to other machines when connecting fails
Opening an SSHStore or LegacySSHStore does not actually establish a
connection, so the try/catch block here did nothing. Added a
Store::connect() method to test whether a connection can be
established.
2017-05-02 15:46:09 +02:00
Eelco Dolstra
1a68710d4d
Add an option for specifying remote builders
This is useful for one-off situations where you want to specify a
builder on the command line instead of having to mess with
nix.machines. E.g.

  $ nix-build -A hello --argstr system x86_64-darwin \
    --option builders 'root@macstadium1 x86_64-darwin'

will perform the specified build on "macstadium1".

It also removes the need for a separate nix.machines file since you
can specify builders in nix.conf directly. (In fact nix.machines is
yet another hack that predates the general nix.conf configuration
file, IIRC.)

Note: this option is supported by the daemon for trusted users. The
fact that this allows trusted users to specify paths to SSH keys to
which they don't normally have access is maybe a bit too much trust...
2017-05-02 15:42:43 +02:00
Eelco Dolstra
ebc9f36a81
Factor out machines.conf parsing
This allows hydra-queue-runner to use it.
2017-05-02 13:17:37 +02:00
Eelco Dolstra
174b68a2a2
build-hook: If there are no machines defined, quit permanently 2017-05-02 12:16:29 +02:00
Eelco Dolstra
feefcb3a98
build-remote: Ugly hackery to get build logs to work
The build hook mechanism expects build log output to go to file
descriptor 4, so do that.
2017-05-02 12:02:23 +02:00
Eelco Dolstra
3a5f04f48c
build-remote: Don't require signatures
This restores the old behaviour.
2017-05-01 20:03:25 +02:00
Eelco Dolstra
031d70e500
Support arbitrary store URIs in nix.machines
For backwards compatibility, if the URI is just a hostname, ssh://
(i.e. LegacySSHStore) is prepended automatically.

Also, all fields except the URI are now optional. For example, this is
a valid nix.machines file:

  local?root=/tmp/nix

This is useful for testing the remote build machinery since you don't
have to mess around with ssh.
2017-05-01 17:35:30 +02:00
Eelco Dolstra
3e4bdfedee
Minor cleanup 2017-05-01 17:30:17 +02:00
Eelco Dolstra
deac171925
Implement LegacySSHStore::buildDerivation()
This makes LegacySSHStore usable by build-remote and
hydra-queue-runner.
2017-05-01 17:30:16 +02:00
Eelco Dolstra
3f5b98e65a
Chomp log output from the build hook 2017-05-01 17:30:16 +02:00
Eelco Dolstra
d7653dfc6d
Remove $NIX_BUILD_HOOK and $NIX_CURRENT_LOAD
This is to simplify remote build configuration. These environment
variables predate nix.conf.

The build hook now has a sensible default (namely build-remote).

The current load is kept in the Nix state directory now.
2017-05-01 17:30:16 +02:00
Eelco Dolstra
ca9f589a93
build-remote: Don't copy the .drv closure
Since build-remote uses buildDerivation() now, we don't need to copy
the .drv file anymore. This greatly reduces the set of input paths
copied to the remote side (e.g. from 392 to 51 store paths for GNU
hello on x86_64-darwin).
2017-05-01 17:30:16 +02:00
Eelco Dolstra
b986c7f8b1
Pass verbosity level to build hook 2017-05-01 14:43:14 +02:00
Eelco Dolstra
227a48f86f
Reduce severity of EMLINK warnings
Fixes #1357.
2017-05-01 14:36:56 +02:00
Eelco Dolstra
0dddcf867a
Add a dummy Store::buildPaths() method
This default implementation of buildPaths() does nothing if all
requested paths are already valid, and throws an "unsupported
operation" error otherwise. This fixes a regression introduced by
c30330df6f in binary cache and legacy
SSH stores.
2017-05-01 13:43:34 +02:00
Guillaume Maudoux
a143014d73 lexer: remove catch-all rules hiding real errors
With catch-all rules, we hide potential errors.
It turns out that a4744254 made one cath-all useless. Flex detected that
is was impossible to reach.
The other is more subtle, as it can only trigger on unfinished escapes
in unfinished strings, which only occurs at EOF.
2017-05-01 01:18:06 +02:00
Guillaume Maudoux
a474425425 Fix lexer to support $' in multiline strings. 2017-05-01 01:15:40 +02:00
Eelco Dolstra
2f21d522c2
Hopefully fix the Darwin build
http://hydra.nixos.org/build/52080911
2017-04-28 17:13:55 +02:00
Eelco Dolstra
895f00c372
Suppress warning about ssh-auth-sock 2017-04-28 16:55:52 +02:00
Eelco Dolstra
73bba12d8b
Check for libreadline 2017-04-28 16:53:56 +02:00
Eelco Dolstra
a1a5e63e14
Fix brainfart 2017-04-28 16:21:54 +02:00
Eelco Dolstra
41c4558afe
Fix hash computation when importing NARs greater than 4 GiB
This caused "nix-store --import" to compute an incorrect hash on NARs
that don't fit in an unsigned int. The import would succeed, but
"nix-store --verify-path" or subsequent exports would detect an
incorrect hash.

A deeper issue is that the export/import format does not contain a
hash, so we can't detect such issues early.

Also, I learned that -Wall does not warn about this.
2017-04-28 15:24:05 +02:00
Eelco Dolstra
39b08f4c0c Merge pull request #1358 from shlevy/store-nesting
Add Store nesting to fix import-from-derivation within filterSource
2017-04-26 20:28:49 +02:00
Shea Levy
4bc00760f9 Add Store nesting to fix import-from-derivation within filterSource 2017-04-26 14:15:47 -04:00
Eelco Dolstra
45ce2c7413
Doh 2017-04-26 17:58:09 +02:00
Eelco Dolstra
6734c18c99
nix repl: Fix Ctrl-C 2017-04-25 19:19:48 +02:00
Eelco Dolstra
23aa1619da
Minor cleanup 2017-04-25 19:10:47 +02:00
Eelco Dolstra
536f061765
"using namespace std" considered harmful 2017-04-25 18:58:02 +02:00
Eelco Dolstra
5bd8795e1f
nix repl: Use $XDG_DATA_HOME for the readline history 2017-04-25 18:56:29 +02:00
Eelco Dolstra
921a2aeb05
Make "nix repl" build 2017-04-25 18:48:40 +02:00
Eelco Dolstra
c31000bc93
Merge nix-repl repository 2017-04-25 18:14:13 +02:00
Eelco Dolstra
40daf0d800
Cleanup in preparation of merging nix-repl repo into nix repo 2017-04-25 18:13:23 +02:00
Eelco Dolstra
c30330df6f
StorePathCommands: Build installables
So for instance "nix copy --to ... nixpkgs.hello" will build
nixpkgs.hello first. It's debatable whether this is a good idea. It
seems desirable for commands like "nix copy" but maybe not for
commands like "nix path-info".
2017-04-25 16:19:22 +02:00
Eelco Dolstra
d48c973ece
Set default installable
Thus

  $ nix build -f foo.nix

will build foo.nix.

And

  $ nix build

will build default.nix. However, this may not be a good idea because
it's kind of inconsistent, given that "nix build foo" will build the
"foo" attribute from the default installation source (i.e. the
synthesis of $NIX_PATH), rather than ./default.nix. So I may revert
this.
2017-04-25 15:18:05 +02:00
Eelco Dolstra
0b6220fbd6
Interpret any installable containing a slash as a path
So "nix path-info ./result" now works.
2017-04-25 14:09:01 +02:00
Eelco Dolstra
7ee81f3887
Make StorePathsCommand a subclass of InstallablesCommand
This allows commands like 'nix path-info', 'nix copy', 'nix verify'
etc. to work on arbitrary installables. E.g. to copy geeqie to a
binary cache:

  $ nix copy -r --to file:///tmp/binary-cache nixpkgs.geeqie

Or to get the closure size of thunderbird:

  $ nix path-info -S nixpkgs.thunderbird
2017-04-25 13:20:26 +02:00
Eelco Dolstra
c769841bc4
Move code around 2017-04-25 12:07:31 +02:00
Eelco Dolstra
6267d74889
Add "nix eval" command
This replaces "nix-instantiate --eval". The result is evaluated
strictly since this seems more useful.
2017-04-25 11:23:47 +02:00
Eelco Dolstra
bcecc99007
Restructure installables handling in the "nix" command 2017-04-25 11:20:37 +02:00
Eelco Dolstra
1bb87c0487
Remove debug statement 2017-04-24 15:01:28 +02:00
Eelco Dolstra
66577a1c64
Factor out --json 2017-04-24 14:21:36 +02:00
Eelco Dolstra
9b63bb88c8
nix-shell -p: Use runCommandCC
This restores pre-17.03 behaviour by making gcc available.
2017-04-24 12:04:01 +02:00
David McFarland
804ac52489 add helper function to set 'interruptThrown'
this fixes a linker failure on cygwin 64 due to some bad
interaction between tls and shared libraries.

see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64697
2017-04-21 11:28:14 -03:00
Eelco Dolstra
749696e71c
Detect lsof
Also, don't use lsof on Linux since it's not needed.

Fixes #1328.
2017-04-20 19:11:45 +02:00
Eelco Dolstra
efa4bdbfcd
Improve nix show-config --json
In particular, show descriptions. This could be used for manpage
generation etc.
2017-04-20 17:34:47 +02:00
Eelco Dolstra
4410e9d995
Setting: Remove "Tag" template argument 2017-04-20 16:52:53 +02:00
Eelco Dolstra
f05d5f89ff
Read per-user settings from ~/.config/nix/nix.conf 2017-04-20 14:58:16 +02:00
Eelco Dolstra
562585e901
binary-caches-parallel-connections -> http-connections 2017-04-20 14:04:00 +02:00
Eelco Dolstra
76cb3c702c
Reimplement trusted-substituters (aka trusted-binary-caches) 2017-04-20 13:41:29 +02:00
Eelco Dolstra
9cc8047f44
Reimplement connect-timeout
Fixes #1339.
2017-04-19 14:54:52 +02:00
Eelco Dolstra
b0cb117226
getDerivations(): Filter out packages with bad derivation names
In particular, this disallows attribute names containing dots or
starting with dots. Hydra already disallowed these. This affects the
following packages in Nixpkgs master:

  2048-in-terminal
  2bwm
  389-ds-base
  90secondportraits
  lispPackages.3bmd
  lispPackages.hu.dwim.asdf
  lispPackages.hu.dwim.def

Closes #1342.
2017-04-19 14:18:26 +02:00
Eelco Dolstra
67fe3e07b2 Merge pull request #1321 from shlevy/channel-direct-tarball-error
nix-channel: error out if direct tarball unpack fails.
2017-04-18 16:51:46 +02:00
Eelco Dolstra
f8a2e8a552
Shut up some warnings 2017-04-14 14:42:20 +02:00
Eelco Dolstra
dd3714f6ef
Doh 2017-04-14 14:42:08 +02:00
Eelco Dolstra
6520b757c5
Fix 32-bit build
http://hydra.nixos.org/build/51569816
2017-04-14 13:59:39 +02:00
Eelco Dolstra
3872371f25
Minor cleanup 2017-04-14 13:42:22 +02:00
Eelco Dolstra
01dcdfcf33
nix-daemon: Don't set untrusted-* settings
These are no longer used anywhere.
2017-04-14 11:59:51 +02:00
Eelco Dolstra
1673c373c9
nix-daemon: Don't die if the user sends an unknown setting 2017-04-14 11:57:02 +02:00
Eelco Dolstra
872ba75d8b
Add "nix show-config" command
This dumps the entire Nix configuration, including all options that
have default values.
2017-04-13 20:59:38 +02:00
Eelco Dolstra
ba9ad29fdb
Convert Settings to the new config system
This makes all config options self-documenting.

Unknown or unparseable config settings and --option flags now cause a
warning.
2017-04-13 20:53:23 +02:00
Eelco Dolstra
6bd9576aeb
Support arbitrary numeric types for settings 2017-04-13 17:54:05 +02:00
Eelco Dolstra
0bf34de43b
Validate Boolean settings better 2017-04-13 16:31:28 +02:00
Eelco Dolstra
1860070548
Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Eelco Dolstra
2040240e23
Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra
568a099c88
canonPath(): Check against empty paths 2017-04-13 16:03:31 +02:00
Eelco Dolstra
6d97d81656
Add warn function 2017-04-13 16:03:31 +02:00
Eelco Dolstra
31cc9366fc
Initialise logger 2017-04-13 16:03:31 +02:00
Eelco Dolstra
23304f527a Merge pull request #1302 from dtzWill/fix/nix-options
Process nix.conf options in "new" commands, add test
2017-04-12 11:03:19 +02:00
Shea Levy
503cc4431b nix-channel: error out if direct tarball unpack fails.
It's very unlikely a path ending in .tar.gz is a directory

Fixes #1318
2017-04-10 18:16:46 -04:00
Eelco Dolstra
1fe1976e0d Merge pull request #1316 from copumpkin/nix-retries-default
Default to 5 download retries
2017-04-10 15:58:57 +02:00
Dan Peebles
d1fdade755 Add CURLE_WRITE_ERROR as a transient error condition
We've observed it failing downloads in the wild and retrying the same URL
a few moments later seemed to fix it.
2017-04-10 09:28:44 -04:00
Dan Peebles
e43e8be8e7 Default to 5 download retries
This should help certain downloaders that don't request anything special
for the number of retries, like nix-channel.
2017-04-10 09:22:24 -04:00
Eelco Dolstra
105f8ffc98
Minor cleanup
Also, possible fix for #1310 on 32-bit systems.
2017-04-10 11:27:33 +02:00
Eelco Dolstra
95295482ea
Allow "auto" as a store URI
Using the empty string is likely to be ambiguous in some contexts.
2017-04-10 11:27:29 +02:00
Neil Mayhew
f12a048a05 Propagate NIX_BUILD_CORES to nix-shell environments 2017-04-09 08:21:52 -06:00
Will Dietz
30f89e0d65 Process nix.conf options in "new" nix commands, add test.
Without this (minor) change, the options set using "--option"
or read from nix.conf were parsed but not used.
2017-04-08 12:59:42 -05:00
Dan Peebles
98283915f5 Retry downloads on transient SSL errors too 2017-04-06 18:18:43 +00:00
Eelco Dolstra
ba20730b3f
Implement RemoteStore::queryMissing()
This provides a significant speedup, e.g. 64 s -> 12 s for

  nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test

on a cold local and CloudFront cache.

The alternative is to use lots of concurrent daemon connections but
that seems wasteful.
2017-04-06 18:40:19 +02:00
Eelco Dolstra
963f2bf12b
Fix bogus "unexpected Nix daemon error: interrupted by the user" 2017-04-06 17:19:32 +02:00
Eelco Dolstra
6b5e271163
Add a method to allow hydra-queue-runner to flush the path info cache 2017-04-06 15:22:37 +02:00
Eelco Dolstra
256940fc48
nix-daemon: Disable path info cache
This is useless because the client also caches path info, and can
cause problems for long-running clients like hydra-queue-runner
(i.e. it may return cached info about paths that have been
garbage-collected).
2017-04-06 14:30:31 +02:00
Eelco Dolstra
8decb07c31
Allow default sandbox paths to be overriden
E.g. you can now redirect /etc/resolv.conf to a different file.
2017-04-04 17:54:16 +02:00
Eelco Dolstra
488792a87d
Make /var/run/nscd/socket optional
Not every distribution uses nscd.
2017-04-04 17:40:50 +02:00
Eelco Dolstra
29d35805c6
Sandbox: Fix /dev/ptmx on recent kernels
This fixes "No such file or directory" when opening /dev/ptmx
(e.g. http://hydra.nixos.org/build/51094249).

The reason appears to be some changes to /dev/ptmx / /dev/pts handling
between Linux 4.4 and 4.9. See
https://patchwork.kernel.org/patch/7832531/.

The fix is to go back to mounting a proper /dev/pts instance inside
the sandbox. Happily, this now works inside user namespaces, even for
unprivileged users. So

  NIX_REMOTE=local?root=/tmp/nix nix-build \
    '<nixpkgs/nixos/tests/misc.nix>' -A test

works for non-root users.

The downside is that the fix breaks sandbox builds on older kernels
(probably pre-4.6), since mounting a devpts fails inside user
namespaces for some reason I've never been able to figure out. Builds
on those systems will fail with

  error: while setting up the build environment: mounting /dev/pts: Invalid argument

Ah well.
2017-03-31 18:20:19 +02:00
Shea Levy
3ecb09a40a builtins.exec: Make the argument just a list 2017-03-31 11:58:41 -04:00
Shea Levy
d299bd710a Merge branch 'builtins.exec' 2017-03-31 11:22:39 -04:00
Eelco Dolstra
e1509adbbb
Retry curl error 16 2017-03-31 15:50:11 +02:00
Eelco Dolstra
c0745a2531
Merge branch 'remove-perl' of https://github.com/shlevy/nix 2017-03-31 14:13:32 +02:00
Shea Levy
0bb8db257d Add exec primop behind allow-unsafe-native-code-during-evaluation.
Execute a given program with the (optional) given arguments as the
user running the evaluation, parsing stdout as an expression to be
evaluated.

There are many use cases for nix that would benefit from being able to
run arbitrary code during evaluation, including but not limited to:

* Automatic git fetching to get a sha256 from a git revision
* git rev-parse HEAD
* Automatic extraction of information from build specifications from
  other tools, particularly language-specific package managers like
  cabal or npm
* Secrets decryption (e.g. with nixops)
* Private repository fetching

Ideally, we would add this functionality in a more principled way to
nix, but in the mean time 'builtins.exec' can be used to get these
tasks done.

The primop is only available when the
'allow-unsafe-native-code-during-evaluation' nix option is true. That
flag also enables the 'importNative' primop, which is strictly more
powerful but less convenient (since it requires compiling a plugin
against the running version of nix).
2017-03-30 08:04:21 -04:00
Eelco Dolstra
c60715e937
Ignore broken "Deriver: unknown-deriver" fields in .narinfo
These were generated by a legacy tool.
2017-03-28 13:08:13 +02:00
Daiderd Jordan
023217f07c
use std::tuple for ValueMap allocator 2017-03-24 23:05:49 +01:00
Eelco Dolstra
0a7ca24c26
Fix xz decompression
Fixes #1285.
2017-03-22 11:53:33 +01:00
Eelco Dolstra
895a74a814
LocalFSStore::getBuildLog(): Handle corrupted logs 2017-03-21 19:23:35 +01:00
Eelco Dolstra
ed5c0f69f2
Don't hang in decompression if bzip2 data ends prematurely 2017-03-21 19:23:35 +01:00
Eelco Dolstra
e1e49c58e1
Only use cache.nixos.org when the store is /nix/store
This is consistent with the behaviour of the old
download-from-binary-cache substituter.
2017-03-21 17:59:18 +01:00
Eelco Dolstra
ecbc3fedd3
Require signatures by default
This corresponds to the NixOS default.
2017-03-21 15:10:48 +01:00
Eelco Dolstra
4bb38591e5
Restore cache.nixos.org as the default substituter
Fixes #1283.
2017-03-21 15:06:46 +01:00
Eelco Dolstra
f8b84a3b8c
Move istringstream_nocopy to a separate file 2017-03-21 14:43:03 +01:00
Eelco Dolstra
3229f85585
Honor $NIX_SSHOPTS again
NixOps needs this.
2017-03-21 14:35:50 +01:00
Eelco Dolstra
20df50d8e1
Merge branch 'darwin-s3-binary-cache-store' of https://github.com/shlevy/nix 2017-03-20 17:46:35 +01:00
Eelco Dolstra
dc931fe1cd Merge pull request #1275 from steveeJ/patch-1
nix-shell/pure: keep environment variable SHLVL
2017-03-20 17:38:40 +01:00
Eelco Dolstra
558eda0115
nix copy: Make -r option use the "from" store
Previously, we tried to compute the closure in the local store, which
obviously doesn't work.
2017-03-16 14:25:54 +01:00
Eelco Dolstra
287084d688
ssh:// -> ssh-ng://, legacy-ssh:// -> ssh:// 2017-03-16 14:19:32 +01:00
Eelco Dolstra
c5b83d8913
copyPaths(): Use queryValidPaths() to reduce SSH latency 2017-03-16 13:50:01 +01:00
Eelco Dolstra
91d67692cf
copyPaths(): Don't query path info for a path the target already has
For example, this cuts "nix-copy-closure --from" on a NixOS system
closure from 15.9s to 0.5s.
2017-03-16 12:05:51 +01:00
Eelco Dolstra
ea7fa88131
LegacySSHStore: Provide a faster implementation of computeFSClosure()
This avoids the latency of the standard implementation, which can make
a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 11:44:52 +01:00
Eelco Dolstra
7a716ef2a5
Fix nix-copy-closure --to 2017-03-16 10:58:48 +01:00
Eelco Dolstra
0ec7f47b00
Remove "killing process <pid>" messages
They convey no useful information.
2017-03-16 10:52:28 +01:00
Eelco Dolstra
43f158bb08
nix-copy-closure: Fix assertion failure
$ ./inst/bin/nix-copy-closure --to bla $(type -p firefox)
  nix-copy-closure: src/libstore/store-api.cc:80: std::__cxx11::string nix::storePathToHash(const Path&): Assertion `base.size() >= storePathHashLen' failed.
2017-03-16 10:45:45 +01:00
Eelco Dolstra
3f35612c04
More precise compression settings 2017-03-15 17:20:19 +01:00
Eelco Dolstra
25dff2b7db
runProgram(): Distinguish between empty input and no input
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
2017-03-15 16:50:19 +01:00
Eelco Dolstra
042975ea8e
Compress NAR listings using the "text-compression" method
So if "text-compression=br", the .ls file in S3 will get a
Content-Encoding of "br". Brotli appears to compress better than xz
for this kind of file and is natively supported by browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra
45c70382ac
S3BinaryCacheStore: Set Content-Type
This is necessary for serving log files to browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra
8b1d65bebe
S3BinaryCacheStore: Support compression of narinfo and log files
You can now set the store parameter "text-compression=br" to compress
textual files in the binary cache (i.e. narinfo and logs) using
Brotli. This sets the Content-Encoding header; the extension of
compressed files is unchanged.

You can separately specify the compression of log files using
"log-compression=br". This is useful when you don't want to compress
narinfo files for backward compatibility.
2017-03-15 16:49:28 +01:00
Eelco Dolstra
2691498b5c
Fix assertion failure on SIGINT
nix: src/libutil/compression.cc:142: virtual nix::XzSink::~XzSink(): Assertion `finished' failed.
2017-03-15 16:49:06 +01:00
Eelco Dolstra
fbbc4d8dda
Fix deadlock in runProgram() when input is larger than the pipe buffer size 2017-03-15 16:49:06 +01:00
Eelco Dolstra
e8186085e0
Add support for brotli compression
Build logs on cache.nixos.org are compressed using Brotli (since this
allows them to be decompressed automatically by Chrome and Firefox),
so it's handy if "nix log" can decompress them.
2017-03-15 16:49:06 +01:00
Eelco Dolstra
73d7a51ee6
Remove dependency on "curl" binary 2017-03-15 16:48:29 +01:00
Eelco Dolstra
532d73d5d8
BinaryCacheStore: Implement getBuildLog()
We assume that build logs are stored under log/<drv>, e.g.

  /nix/store/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv

maps to

  https://cache.nixos.org/log/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv
2017-03-15 16:48:29 +01:00
Eelco Dolstra
5b86451f02
Add a "nix log" command
This replaces "nix-store --read-log". It checks the local store and
any configured substituters for the requested logs.
2017-03-15 16:48:29 +01:00
Eelco Dolstra
0afeb7f51e
Store: Add a method for getting build logs
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.

Also, remove the log-servers option since we can use substituters for
this.
2017-03-15 16:48:29 +01:00
Shea Levy
96443e94a1 Merge branch 'configurable-aws-region' 2017-03-12 10:01:11 -04:00
Stefan Junker
f628ca2a1f nix-shell/pure: keep environment variable SHLVL 2017-03-12 01:04:21 +01:00
Eelco Dolstra
86227390c5
Merge branch 'allow-import-from-derivation' of https://github.com/shlevy/nix 2017-03-09 13:36:30 +01:00
Adrien Devresse
d853877ce9 Add missing header <sstream> 2017-03-08 22:24:10 +01:00
Shea Levy
93f863be96 Add option to disable import-from-derivation completely, even if the drv is already realized 2017-03-08 08:46:12 -05:00
Shea Levy
3cc18d3753 Properly set the caFile for aws-sdk-cpp s3 2017-03-06 14:30:35 -05:00
Shea Levy
4fc30922cf istringstream_nocopy: Implement in a standards-compliant way.
Fixes the problem mentioned in e6a61b8da7

See #1135
2017-03-06 13:03:02 -05:00
Eelco Dolstra
1cf4801108
Revert fa125b9b28
This causes quadratic performance.
2017-03-06 13:56:20 +01:00
Shea Levy
5789eaa3f4 Add aws-region param to S3 store URLs 2017-03-03 16:12:17 -05:00
Eelco Dolstra
d1158bb816
Cache connection failures 2017-03-03 19:36:01 +01:00
Eelco Dolstra
8490ee37a6
SSHMaster: Make thread-safe 2017-03-03 19:28:27 +01:00
Eelco Dolstra
d3eb1cf3bb
build-remote: Don't use a SSH master
This is unnecessary because we make only one connection.
2017-03-03 19:23:20 +01:00
Eelco Dolstra
d140c75530
Pool: Don't hang if creating a connection fails 2017-03-03 19:21:43 +01:00
Eelco Dolstra
577ebeaefb
Improve SSH handling
* Unify SSH code in SSHStore and LegacySSHStore.

* Fix a race starting the SSH master. We now wait synchronously for
  the SSH master to finish starting. This prevents the SSH clients
  from starting their own connections.

* Don't use a master if max-connections == 1.

* Add a "max-connections" store parameter.

* Add a "compress" store parameter.
2017-03-03 19:05:50 +01:00
Eelco Dolstra
7f62be1bcd
build-remote: Fix passing SSH key 2017-03-03 16:33:18 +01:00
Eelco Dolstra
5a1fb03b8f
build-remote: Misc cleanup 2017-03-03 16:18:49 +01:00
Eelco Dolstra
2f992692e2
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing 2017-03-03 15:40:06 +01:00
Eelco Dolstra
ffcf9d24a6
builtins.fetchgit: Support paths 2017-03-02 11:46:28 +01:00
Eelco Dolstra
fbf17f1ad7
builtins.fetchgit: Fix bad format string 2017-03-02 11:40:11 +01:00
Eelco Dolstra
fa125b9b28
TeeSink: Pre-reserve string space
When receiving a very large file, this can prevent the string from
having tobe copied, which temporarily doubles memory consumption.
2017-03-01 16:16:04 +01:00
Eelco Dolstra
f61f67ddee
RemoteStore::addToStore(): Send NAR rather than string containing NAR
This allows the NAR to be streamed in the future (though we're not
doing that yet).
2017-03-01 16:07:15 +01:00