Commit graph

3525 commits

Author SHA1 Message Date
Eelco Dolstra
e6a61b8da7
Fix S3BinaryCacheStore
It failed with

   AWS error uploading ‘6gaxphsyhg66mz0a00qghf9nqf7majs2.ls.xz’: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header.

possibly because the istringstream_nocopy introduced in
0d2ebb4373 doesn't supply the seek
method that the AWS library expects. So bring back the old version,
but only for S3BinaryCacheStore.
2016-12-08 15:31:27 +01:00
Eelco Dolstra
9a313469a4
Add a hook to run diffoscope when non-determinism is detected 2016-12-07 17:57:35 +01:00
Eelco Dolstra
b07060688a
Keep track of the exact build start/stop times 2016-12-07 16:09:38 +01:00
Eelco Dolstra
dadfddfa7c
Bail out early when non-determinism is detected 2016-12-07 15:31:18 +01:00
Eelco Dolstra
cea4e2c618
Expose enforce-determinism and the result to Hydra 2016-12-07 13:43:40 +01:00
Eelco Dolstra
8bdf83f936
Add an option to make non-determinism non-fatal
That is, when build-repeat > 0, and the output of two rounds differ,
then print a warning rather than fail the build. This is primarily to
let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Eelco Dolstra
ceeedb58d2
Use a steady clock for timeouts
Fixes #1146.
2016-12-06 21:58:04 +01:00
Eelco Dolstra
7a3e7d0e61
nix-store --serve: Suppress log output on stderr when repeating a build 2016-12-06 17:43:39 +01:00
Eelco Dolstra
6069b946ad
nix-store --serve: Support setting build-repeat
This allows Hydra to test whether builds are reproducible.
2016-12-06 17:19:22 +01:00
Eelco Dolstra
603f08506e
Tweak error message 2016-12-06 17:18:40 +01:00
Guillaume Maudoux
e4b82af387 Improve error message on trailing path slashes 2016-11-27 17:48:46 +01:00
Jörg Thalheim
855abd85d8
Simplify remouting with MS_PRIVATE in sandbox build
also fix race condition if mounts are added after mountinfo is read.
2016-11-25 00:15:39 +01:00
Eelco Dolstra
7ee43df862
nix-channel: Fix --update <CHANNELS>
This unbreaks "nixos-rebuild --upgrade".
2016-11-21 15:54:19 +01:00
Eelco Dolstra
349e988903
Typo 2016-11-17 17:55:41 +01:00
Shea Levy
00b8bce4d0 Fix binary-cache-store build 2016-11-17 11:48:10 -05:00
Shea Levy
0d2ebb4373 istringstream_nocopy: Implement in a standards-compliant way
Fixes #1135.
2016-11-17 10:10:32 -05:00
Shea Levy
3f4d3f8a1a nix-daemon: Fix splice faking on non-Linux 2016-11-17 08:10:12 -05:00
Shea Levy
8bf378e999 Update darwin build for optional sandbox paths
Fixes #1132
2016-11-17 08:06:32 -05:00
aszlig
4e1a2cd537
seccomp: Forge return values for *chown32
These syscalls are only available in 32bit architectures, but libseccomp
should handle them correctly even if we're on native architectures that
do not have these syscalls.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:36 +01:00
aszlig
ed64976cec
seccomp: Forge return codes for POSIX ACL syscalls
Commands such as "cp -p" also use fsetxattr() in addition to fchown(),
so we need to make sure these syscalls always return successful as well
in order to avoid nasty "Invalid value" errors.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:14 +01:00
aszlig
b90a435332
libstore/build: Forge chown() to return success
What we basically want is a seccomp mode 2 BPF program like this but for
every architecture:

  BPF_STMT(BPF_LD+BPF_W+BPF_ABS, offsetof(struct seccomp_data, nr)),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_chown, 4, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchown, 3, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchownat, 2, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_lchown, 1, 0),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ERRNO)

However, on 32 bit architectures we do have chown32, lchown32 and
fchown32, so we'd need to add all the architecture blurb which
libseccomp handles for us.

So we only need to make sure that we add the 32bit seccomp arch while
we're on x86_64 and otherwise we just stay at the native architecture
which was set during seccomp_init(), which more or less replicates
setting 32bit personality during runChild().

The FORCE_SUCCESS() macro here could be a bit less ugly but I think
repeating the seccomp_rule_add() all over the place is way uglier.

Another way would have been to create a vector of syscalls to iterate
over, but that would make error messages uglier because we can either
only print the (libseccomp-internal) syscall number or use
seccomp_syscall_resolve_num_arch() to get the name or even make the
vector a pair number/name, essentially duplicating everything again.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:30 +01:00
aszlig
1c52e344c4
Add build dependency for libseccomp
We're going to use libseccomp instead of creating the raw BPF program,
because we have different syscall numbers on different architectures.

Although our initial seccomp rules will be quite small it really doesn't
make sense to generate the raw BPF program because we need to duplicate
it and/or make branches on every single architecture we want to suuport.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:26 +01:00
aszlig
e8838713df
Run builds as root in user namespace again
This reverts commit ff0c0b645c.

We're going to use seccomp to allow "cp -p" and force chown-related
syscalls to always return 0.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:08 +01:00
Eelco Dolstra
4be4f6de56
S3BinaryCacheStore:: Eliminate a string copy while uploading
This cuts hydra-queue-runner's peak memory usage by about a third.
2016-11-16 16:21:30 +01:00
Eelco Dolstra
10ae8fabf1 buildPaths(): Handle ecIncompleteClosure
buildPaths() on a non-derivation would incorrectly not throw an error
if the path didn't have a substitute.
2016-11-14 15:00:17 +01:00
Eelco Dolstra
b77fb8acb5 Don't rely on %m 2016-11-14 13:37:16 +01:00
Ludovic Courtès
ccb1022022 daemon: Do not error out when deduplication fails due to ENOSPC.
This solves a problem whereby if /gnu/store/.links had enough entries,
ext4's directory index would be full, leading to link(2) returning
ENOSPC.

* nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
ENOSPC from link(2), print a message and return instead of throwing a
'SysError'.
2016-11-14 13:35:03 +01:00
Guillaume Maudoux
a5e761dddb Fix comments parsing
Fixed the parsing of multiline strings ending with an even number of
stars, like /** this **/.
Added test cases for comments.
2016-11-13 17:20:34 +01:00
Eelco Dolstra
dd77f7d593 Store::computeFSClosure(): Support a set of paths
This way, callers can exploits the parallelism of computeFSClosure()
when they have multiple paths that they need the (combined) closure of.
2016-11-10 17:45:04 +01:00
Shea Levy
bff3ad767e build-remote: replace strtoull with stoull to take advantage of C++ error handling 2016-11-10 11:09:15 -05:00
Shea Levy
167d12b02c build-remote: Implement in C++ 2016-11-10 11:09:15 -05:00
Eelco Dolstra
e4655b166c copyClosure() / copyStorePath(): Expose dontCheckSigs
Needed by Hydra.
2016-11-09 19:08:20 +01:00
Eelco Dolstra
4b8f1b0ec0 Merge branch 'ssh-store' of https://github.com/shlevy/nix 2016-11-09 18:57:22 +01:00
Eelco Dolstra
21c55ab3b5 Implement backwards-compatible RemoteStore::addToStore()
The SSHStore PR adds this functionality to the daemon, but we have to
handle the case where the Nix daemon is 1.11.

Also, don't require signatures for trusted users. This restores 1.11
behaviour.

Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-09 18:45:06 +01:00
Eelco Dolstra
a83b10f84c Typo 2016-11-08 20:19:02 +01:00
Eelco Dolstra
b3ba762dbf Add missing #include 2016-11-07 14:35:47 +01:00
Vladimír Čunát
818aad3ec4
Detect and disallow base32 hash overflow
Example (before this commit):
$ nix-hash --type sha256 --to-base16 4n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0 \
    | xargs nix-hash --type sha256 --to-base32
0n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0

It's a real-life example:
https://github.com/NixOS/nixpkgs/pull/20208/files#r86695567
2016-11-06 22:17:37 +01:00
Eelco Dolstra
18b7363a69 Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
2016-10-31 17:09:52 +01:00
Shea Levy
c4969aebaf Add nix.conf options for -k and -K
Fixes #1084
2016-10-27 12:49:34 -04:00
Tuomas Tynkkynen
f0fc3dd88b Fix SIGFPE from integer overflow during division
On some architectures (like x86_64 or i686, but not ARM for example)
overflow during integer division causes a crash due to SIGFPE.
Reproduces on a 64-bit system with:

    nix-instantiate --eval -E '(-9223372036854775807 - 1) / -1'

The only way this can happen is when the smallest possible integer is
divided by -1, so just special-case that.
2016-10-26 18:41:59 +03:00
Eelco Dolstra
5c0bd51d49 Fix build log output in nix-store --serve 2016-10-26 17:13:38 +02:00
Eelco Dolstra
d2c58ba605 Restore the CachedFailure status code
The removal of CachedFailure caused the value of TimedOut to change,
which broke timed-out handling in Hydra (so timed-out builds would
show up as "aborted" and would be retried, e.g. at
http://hydra.nixos.org/build/42537427).
2016-10-26 15:02:38 +02:00
Eelco Dolstra
fdbbcc4492 Remove addPathToAccessor 2016-10-21 18:09:30 +02:00
Eelco Dolstra
542ae5c8f8 BinaryCacheStore: Optionally write a NAR listing
The store parameter "write-nar-listing=1" will cause BinaryCacheStore
to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’
added to the binary cache. This file contains an XZ-compressed JSON
file describing the contents of the NAR, excluding the contents of
regular files.

E.g.

  {
    "version": 1,
    "root": {
      "type": "directory",
      "entries": {
        "lib": {
          "type": "directory",
          "entries": {
            "Mcrt1.o": {
              "type": "regular",
              "size": 1288
            },
            "Scrt1.o": {
              "type": "regular",
              "size": 3920
            },
          }
        }
      }
      ...
    }
  }

(The actual file has no indentation.)

This is intended to speed up the NixOS channels programs index
generator [1], since fetching gazillions of large NARs from
cache.nixos.org is currently a bottleneck for updating the regular
(non-small) channel.

[1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-21 16:50:28 +02:00
Eelco Dolstra
307cc8c33d Merge branch 'priorityqueue' of https://github.com/groxxda/nix 2016-10-19 16:37:04 +02:00
Eelco Dolstra
795d9b8668 Merge branch 'regex' of https://github.com/groxxda/nix 2016-10-19 16:32:24 +02:00
Alexander Ried
c935e8eeaf fixup! replace own regex class with std::regex 2016-10-19 16:03:53 +02:00
Eelco Dolstra
19c278de89 Fix Darwin build
Done slightly differently from https://github.com/NixOS/nix/pull/1093.
2016-10-19 15:21:18 +02:00
Eelco Dolstra
29727ff944 Shut up clang warning 2016-10-19 15:02:48 +02:00
Eelco Dolstra
e5346c54df Fix uninitialised variable 2016-10-19 15:02:38 +02:00
Alexander Ried
b05b98df75 replace own regex class with std::regex 2016-10-18 20:22:25 +02:00
Alexander Ried
efb938468c downloader: use priority_queue 2016-10-18 15:45:51 +02:00
Eelco Dolstra
ae8884b949 Fix /dev/ptmx in sandboxes
This was broken since ff0c0b645c. Since
I can't figure out how to mount a devpts instance in the sandbox,
let's just bind-mount the host devpts.
2016-10-14 15:42:13 +02:00
Eelco Dolstra
2f9d60ed7f CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUM
http://hydra.nixos.org/build/42025230
2016-10-14 14:44:15 +02:00
Eelco Dolstra
5e61b422c5 Fix HTTP/2 support
Commit 86e8c67efc broke it, because
CURL_* are not actually #defines.
2016-10-13 17:25:42 +02:00
Eelco Dolstra
fb2dd32100 SSL_CERT_FILE -> NIX_SSL_CERT_FILE
This prevents collisions with the "native" OpenSSL, in particular on
OS X.

Fixes #921.
2016-10-13 17:09:10 +02:00
Eelco Dolstra
816d3e5724 Shut up some warnings 2016-10-12 15:53:38 +02:00
Eelco Dolstra
82e2a070e0 Add some functions needed by hydra 2016-10-12 15:49:37 +02:00
Eelco Dolstra
edf9eb8181 querySubstitutablePaths(): Don't query paths for which we already have a substituter 2016-10-07 19:57:47 +02:00
Eelco Dolstra
0c85ef7090 LocalStore::querySubstitutablePaths(): Implement using queryValidPaths() 2016-10-07 19:46:56 +02:00
Eelco Dolstra
844219f364 Store::queryValidPaths(): Use async queryPathInfo()
This allows the binary cache substituter to pipeline requests.
2016-10-07 19:43:36 +02:00
Eelco Dolstra
c663b84573 Implement generic Store::queryValidPaths() 2016-10-07 19:20:47 +02:00
Eelco Dolstra
35db4f65a0 Add copyClosure utility function for Hydra 2016-10-07 19:15:25 +02:00
Eelco Dolstra
629ab80022 importPaths(): Fix accessor support for Hydra 2016-10-07 18:13:40 +02:00
Eelco Dolstra
b0f7f9c98f toJSON(): Support some more types 2016-10-06 17:00:52 +02:00
Eelco Dolstra
ae522f930d Fix getS3Stats() 2016-10-06 17:00:52 +02:00
Eelco Dolstra
cd128f4bad Merge pull request #998 from veprbl/rx_chmod_fix
override rx directory permissions in deletePath()
2016-10-06 11:08:21 +02:00
Eelco Dolstra
c6a929986a Use std::random_device 2016-10-05 21:29:18 +02:00
Eelco Dolstra
86e8c67efc Fix build with older versions of libcurl 2016-10-04 14:43:23 +02:00
Eelco Dolstra
4bd51d74af Handle the case where signed-binary-caches consists of whitespace 2016-09-22 15:59:25 +02:00
Eelco Dolstra
818ab58cc6 Add sandbox-dev-shm-size option
Fixes #1069.
2016-09-21 16:54:53 +02:00
Eelco Dolstra
4546be1b3e nix-build, nix-shell: Don't print error message if nix-store/nix-instantiate fails 2016-09-21 16:54:53 +02:00
Eelco Dolstra
c55bf085eb printMsg(lvlError, ...) -> printError(...) etc. 2016-09-21 16:54:53 +02:00
Eelco Dolstra
4036185cb4 Some notational convenience for formatting strings
We can now write

  throw Error("file '%s' not found", path);

instead of

  throw Error(format("file '%s' not found") % path);

and similarly

  printError("file '%s' not found", path);

instead of

  printMsg(lvlError, format("file '%s' not found") % path);
2016-09-21 16:54:53 +02:00
Eelco Dolstra
beaefdf706 Tweak 2016-09-20 17:49:31 +02:00
Eelco Dolstra
f12d56b27b Improve robustness 2016-09-20 17:25:44 +02:00
Eelco Dolstra
794e4ab475 Fix "Promise already satisfied" error 2016-09-20 17:25:12 +02:00
Eelco Dolstra
4de0639105 nix-shell: Fix $PATH handling in the impure case
We were passing "p=$PATH" rather than "p=$PATH;", resulting in some
invalid shell code.

Also, construct a separate environment for the child rather than
overwriting the parent's.
2016-09-20 15:41:41 +02:00
Eelco Dolstra
9fc4cb2ae9 nix-shell: Restore CPU affinity
Otherwise the shell and its children will be bound to one CPU core...
2016-09-20 15:41:41 +02:00
Eric Litak
0d38b4c792 armv5tel can be built by armv6l and armv7l (#1063) 2016-09-16 22:03:34 +02:00
Eelco Dolstra
75989bdca7 Make computeFSClosure() single-threaded again
The fact that queryPathInfo() is synchronous meant that we needed a
thread for every concurrent binary cache lookup, even though they end
up being handled by the same download thread. Requiring hundreds of
threads is not a good idea. So now there is an asynchronous version of
queryPathInfo() that takes a callback function to process the
result. Similarly, enqueueDownload() now takes a callback rather than
returning a future.

Thus, a command like

  nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5

that returns 4941 paths now takes 1.87s using only 2 threads (the main
thread and the downloader thread). (This is with a prewarmed
CloudFront.)
2016-09-16 18:54:14 +02:00
Eelco Dolstra
054be50257 printMsg(): Don't check for interrupts
Having the logger function potentially throw exceptions is
Heisenbuggy.
2016-09-16 18:52:42 +02:00
Eelco Dolstra
2e1493037b nix path-info: Add some more examples 2016-09-14 18:20:11 +02:00
Eelco Dolstra
6656ef7b5b Revive binary-caches-parallel-connections
It's a slight misnomer now because it actually limits *all* downloads,
not just binary cache lookups.

Also add a "enable-http2" option to allow disabling use of HTTP/2
(enabled by default).
2016-09-14 16:38:26 +02:00
Eelco Dolstra
90ad02bf62 Enable HTTP/2 support
The binary cache store can now use HTTP/2 to do lookups. This is much
more efficient than HTTP/1.1 due to multiplexing: we can issue many
requests in parallel over a single TCP connection. Thus it's no longer
necessary to use a bunch of concurrent TCP connections (25 by
default).

For example, downloading 802 .narinfo files from
https://cache.nixos.org/, using a single TCP connection, takes 11.8s
with HTTP/1.1, but only 0.61s with HTTP/2.

This did require a fairly substantial rewrite of the Downloader class
to use the curl multi interface, because otherwise curl wouldn't be
able to do multiplexing for us. As a bonus, we get connection reuse
even with HTTP/1.1. All downloads are now handled by a single worker
thread. Clients call Downloader::enqueueDownload() to tell the worker
thread to start the download, getting a std::future to the result.
2016-09-14 16:36:02 +02:00
Eelco Dolstra
a75d11a7e6 Add a toLower utility function 2016-09-14 14:58:49 +02:00
Eelco Dolstra
e07c0dcf5c Move some .drv parsing functions out of util 2016-09-14 10:54:57 +02:00
Shea Levy
b99c6e0e29 nix-daemon: Fix error message 2016-09-12 08:09:41 -04:00
Shea Levy
196815f700 ssh-store: Start master on-demand 2016-09-12 08:07:50 -04:00
Shea Levy
ab31f9986c Inline ssh-store.hh into ssh-store.cc 2016-09-12 08:03:29 -04:00
Eelco Dolstra
e5949b5ce8 Fix build on GCC 4.9
GCC 4.9 doesn't like reassigning a std::stringstream.

http://hydra.nixos.org/build/40371644
2016-09-12 13:22:23 +02:00
Eelco Dolstra
5039d3b9de Write "T x" instead of "auto x = T"
That's just silly. Hopefully this also fixes the Debian build failure:

http://hydra.nixos.org/build/40371644
2016-09-12 12:06:13 +02:00
Eelco Dolstra
ff0c0b645c Don't run builds as root in the user namespace
This largely reverts c68e5913c7. Running
builds as root breaks "cp -p", since when running as root, "cp -p"
assumes that it can succesfully chown() files. But that's not actually
the case since the user namespace doesn't provide a complete uid
mapping. So it barfs with a fatal error message ("cp: failed to
preserve ownership for 'foo': Invalid argument").
2016-09-08 18:16:23 +02:00
Eelco Dolstra
e4bdd49022 Give a better error message if a path in build-sandbox-paths is invalid 2016-09-08 17:30:25 +02:00
Eelco Dolstra
0babc4ff31 Fix assertion failure after early build setup failure 2016-09-08 17:29:50 +02:00
Shea Levy
a5f2750ebb Fix early removal of rc-file for nix-shell
BASH_ENV causes all non-interactive shells called via eg. /etc/bashrc to
remove the rc-file before the main shell gets to run it. Completion
scripts will often do this. Fixes #976.

Adapted from and fixes #1034.
2016-09-08 09:09:52 -04:00
Shea Levy
584f8a62de Implement nar-based addToStore for remote-store 2016-09-02 14:33:58 -04:00
Shea Levy
ecba88de93 Add ssh store implementation 2016-09-02 14:31:38 -04:00
Shea Levy
b4b5e9ce2f Add narFromPath op to nix daemon 2016-09-02 14:26:02 -04:00
Shea Levy
a705e8ce0a Factor a general remote FS accessor out of BinaryCacheStore 2016-09-02 14:24:34 -04:00
Shea Levy
0f39633290 Factor out the unix domain socket-specific code from RemoteStore 2016-09-02 14:15:04 -04:00
Shea Levy
7d4ccd9b17 nix-daemon: add --stdio flag for handling connections over stdin/stdout 2016-09-02 13:30:28 -04:00
Shea Levy
53b27ddce2 Factor a function to get the store type from a URI out of the main RegisterStoreImplementation 2016-09-02 06:39:29 -04:00
Shea Levy
a91954f0c6 Merge openStore and openStoreAt with default arguments 2016-09-02 06:35:48 -04:00
Shea Levy
87b189c2b3 Merge branch 'nix-build-c++' 2016-08-31 12:10:21 -04:00
Shea Levy
821380c77b nix-build: Clean up a bit 2016-08-31 10:08:00 -04:00
Shea Levy
dfe0938614 download.hh: Fix conflicts from nix-channel-c++ merge 2016-08-31 09:57:56 -04:00
Shea Levy
572aba284a Merge branch 'nix-channel-c++' 2016-08-31 09:49:24 -04:00
Eelco Dolstra
aa1ea0d1e4 Merge pull request #1030 from pikajude/rsd-cc
Implement resolve-system-dependencies in C++
2016-08-31 15:19:10 +02:00
Eelco Dolstra
d74c8a3f4e Fix 32-bit build 2016-08-30 17:38:09 +02:00
Eelco Dolstra
6631a6e1a1 Increase the sleep time between download retries 2016-08-30 15:48:24 +02:00
Eelco Dolstra
97b1af1cbe Fix --fallback
This fixes an assertion failure in "assert(goal);" in
Worker::waitForInput() after a substitution goal is cancelled by the
termination of another goal. The problem was the line

  //worker.childTerminated(shared_from_this()); // FIXME

in the SubstitutionGoal destructor. This was disabled because
shared_from_this() obviously doesn't work from a destructor. So we now
use a real pointer for object identity.
2016-08-30 15:45:39 +02:00
Eelco Dolstra
67364a5795 Fix Fedora build 2016-08-30 13:56:22 +02:00
Eelco Dolstra
92d917b71a Fix GC build 2016-08-30 13:12:12 +02:00
Eelco Dolstra
c42d1acfeb forceBool(): Show position info 2016-08-29 19:37:19 +02:00
Eelco Dolstra
26d92017d3 Add builtin function "partition"
The implementation of "partition" in Nixpkgs is O(n^2) (because of the
use of ++), and for some reason was causing stack overflows in
multi-threaded evaluation (not sure why).

This reduces "nix-env -qa --drv-path" runtime by 0.197s and memory
usage by 298 MiB (in non-Boehm mode).
2016-08-29 19:36:54 +02:00
Eelco Dolstra
c0a7b84748 nix path-info: Add --json flag
Also, factor out JSON generation from value-to-json.{cc,hh}, and
support producing indented JSON.
2016-08-29 17:29:24 +02:00
Shea Levy
9fa21765e7 callFunction: Copy functors to the heap
Normally it's impossible to take a reference to the function passed to
callFunction, so some callers (e.g. ExprApp::eval) allocate that value
on the stack. For functors, a reference to the functor itself may be
kept, so we need to have it on the heap.

Fixes #1045
2016-08-29 07:36:28 -04:00
Eelco Dolstra
a9d4f2b303 Fix queryPathFromHashPart()
The inner lambda was returning a SQLite-internal char * rather than a
std::string, leading to Hydra errors liks

  Caught exception in Hydra::Controller::Root->narinfo "path ‘ø˜£â€™ is not in the Nix store at /nix/store/6mvvyb8fgwj23miyal5mdr8ik4ixk15w-hydra-0.1.1234.abcdef/libexec/hydra/lib/Hydra/Controller/Root.pm line 352."
2016-08-24 14:53:33 +02:00
Eelco Dolstra
d74236d1f2 nix build: Use Nix search path
That is, unless --file is specified, the Nix search path is
synthesized into an attribute set. Thus you can say

  $ nix build nixpkgs.hello

assuming $NIX_PATH contains an entry of the form "nixpkgs=...". This
is more verbose than

  $ nix build hello

but is less ambiguous.
2016-08-23 17:11:19 +02:00
Jude Taylor
50c3b5df32 iterate through fat_headers correctly 2016-08-17 10:24:11 -07:00
Eelco Dolstra
22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Jude Taylor
ca0bce2851 remove double casting 2016-08-16 14:09:57 -07:00
Jude Taylor
cfb77d6e5b account for unknown magic numbers 2016-08-16 12:52:49 -07:00
Jude Taylor
90516c5a7b switch to mmap strategy 2016-08-16 12:45:55 -07:00
Eelco Dolstra
ac841a4679 Merge pull request #1031 from abbradar/replacestrings-context
Allow contexted strings in replaceStrings
2016-08-16 12:47:20 +02:00
Nikolay Amiantov
f1b8dd43be Allow contexted strings in replaceStrings 2016-08-16 13:20:56 +03:00
Jude Taylor
7b006122ae ensure presence of directories 2016-08-15 10:46:21 -07:00
Jude Taylor
adf0216d98 code review comments 2016-08-15 10:43:14 -07:00
Shea Levy
d52d391164 builtins.fetch{url,tarball}: Allow name attribute 2016-08-15 07:37:11 -04:00
Jude Taylor
60f4b25d7d make inclusion conditional 2016-08-14 19:10:38 -07:00
Jude Taylor
f37b6fd07e add a missing load command 2016-08-14 18:54:40 -07:00
Jude Taylor
596e4a5693 remove old traces of resolve-system-dependencies 2016-08-13 15:27:49 -07:00
Jude Taylor
2df9a972fc resolve-system-dependencies: implement in C++ 2016-08-13 11:36:22 -07:00
Shea Levy
59124228b3 nix-channel: implement in c++ 2016-08-11 11:34:43 -04:00
Vladimír Čunát
1bffd83e1a nix-build: extend the meaning of $IN_NIX_SHELL
An equivalent was originally filed against the perl version:
https://github.com/NixOS/nix/pull/933
2016-08-11 17:14:09 +02:00
Eelco Dolstra
b0ebad88cd Merge branch 'buildenv-c++' of https://github.com/shlevy/nix 2016-08-11 17:11:17 +02:00
Shea Levy
91978e3b9a buildenv: perl -> c++ 2016-08-11 07:58:33 -04:00
Eelco Dolstra
66adbdfd97 HttpBinaryCacheStore: Retry on transient HTTP errors
This makes us more robust against 500 errors from CloudFront or S3
(assuming the 500 error isn't cached by CloudFront...).
2016-08-10 18:08:23 +02:00
Eelco Dolstra
9204ea7294 Show HTTP error codes 2016-08-10 18:05:35 +02:00
Eelco Dolstra
f294623d1d SQLite:: Add some convenience 2016-08-10 18:05:35 +02:00
Eelco Dolstra
6cb4bdf152 FSAccessor: Throw InvalidPath 2016-08-10 18:05:35 +02:00
Eelco Dolstra
f43823f676 NarAccessor: Fix handling of non-executable files 2016-08-10 18:05:35 +02:00
Eelco Dolstra
d961c29c9c Mark content-addressed paths in the Nix database and in .narinfo
This allows such paths to be imported without signatures.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
36a51ecab3 LocalFSStore::narFromPath(): Handle the diverted store case 2016-08-10 18:05:35 +02:00
Eelco Dolstra
3eb6217508 Add a "root" parameter to local stores
This makes it easier to create a diverted store, i.e.

  NIX_REMOTE="local?root=/tmp/root"

instead of

  NIX_REMOTE="local?real=/tmp/root/nix/store&state=/tmp/root/nix/var/nix" NIX_LOG_DIR=/tmp/root/nix/var/log
2016-08-10 18:05:35 +02:00
Eelco Dolstra
2fad86f361 Remove $NIX_DB_DIR
This variable has no reason to exist, given $NIX_STATE_DIR.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
be64fbb501 RemoteStore / nix-daemon: Drop support for Nix < 1.0 2016-08-10 18:05:35 +02:00
Shea Levy
80ebc553ec nix-build: Port to c++
This was a dumb line-for-line rewrite, because nix build/nix run/etc.
will replace it.
2016-08-09 07:42:20 -04:00
Eelco Dolstra
ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra
06bbfb6004 builtins.{fetchurl,fetchTarball}: Support a sha256 attribute
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a
hash is specified.
2016-07-26 21:16:52 +02:00
Dmitry Kalinkin
f91748ba73 override rx directory permissions in deletePath()
This fixes instantiation of pythonPackages.pytest that produces a
directory with less permissions during one of it's tests that leads to
a nix error like:

error: opening directory ‘/tmp/nix-build-python2.7-pytest-2.9.2.drv-0/pytest-of-user/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
2016-07-25 18:11:46 -04:00
Shea Levy
ee3032e4de Merge branch 'find-runtime-roots-c++' 2016-07-24 07:33:49 -04:00
Shea Levy
18b0808475 Respect --keep-going when a substituter fails.
Fixes #977
2016-07-23 09:16:08 -04:00
Eelco Dolstra
d3cd0f5856 Fix assertion failure 2016-07-21 18:39:32 +02:00
Eelco Dolstra
e19c90fc6b Fix random failures caused by CurlDownloader modifying a shared string 2016-07-21 18:35:31 +02:00
Eelco Dolstra
ea8e8df6c7 NarInfo::NarInfo(): Ensure that we get a NAR size/hash 2016-07-21 18:35:12 +02:00
Eelco Dolstra
e682a8e138 Fix assertion failure in ThreadPool::enqueue() 2016-07-21 18:14:16 +02:00
Eelco Dolstra
d57981bac4 Store::queryMissing(): Use a thread pool
For one particular NixOS configuration, this cut the runtime of
"nix-store -r --dry-run" from 6m51s to 3.4s. It also fixes a bug in
the size calculation that was causing certain paths to be counted
twice, e.g. before:

  these paths will be fetched (1249.98 MiB download, 2995.74 MiB unpacked):

and after:

  these paths will be fetched (1219.56 MiB download, 2862.17 MiB unpacked):
2016-07-21 17:50:39 +02:00
Eelco Dolstra
a728780fbd Store::computeFSClosure(): Use thread pool
This speeds up queries against the binary cache.
2016-07-21 14:13:35 +02:00
Shea Levy
3c46fe62b8 find-runtime-roots: fold into gc.cc 2016-07-21 07:04:41 -04:00
Shea Levy
06068b353d FdSink: Restore move constructor/assignment 2016-07-13 06:27:41 -04:00
Shea Levy
b33e85229d Make Buffered{Source,Sink} move-safe 2016-07-13 06:03:37 -04:00
Shea Levy
cb5e7254b6 Modernize AutoCloseFD 2016-07-11 15:44:44 -04:00
Eelco Dolstra
74dd603495 Re-implement negative binary cache lookup caching 2016-06-20 17:39:05 +02:00
Eelco Dolstra
c68e5913c7 Run builds in a user namespace
This way, all builds appear to have a uid/gid of 0 inside the
chroot. In the future, this may allow using programs like
systemd-nspawn inside builds, but that will require assigning a larger
UID/GID map to the build.

Issue #625.
2016-06-09 18:30:39 +02:00
Eelco Dolstra
202683a4fc Use O_CLOEXEC in most places 2016-06-09 16:37:08 +02:00
Eelco Dolstra
9bdd949cfd Fix "creating statement: table ValidPaths has no column named ultimate" 2016-06-03 16:06:42 +02:00
Eelco Dolstra
5e51ffb1c2 Support sandbox builds by non-root users
This allows an unprivileged user to perform builds on a diverted store
(i.e. where the physical store location differs from the logical
location).

Example:

  $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \
    'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"'

will do a build in the Nix store physically in /tmp/store but
logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:59:28 +02:00
Eelco Dolstra
2f8b0e557b Consistent quotes 2016-06-03 14:03:09 +02:00
Eelco Dolstra
080d607975 Don't call shared_from_this() from a destructor 2016-06-03 14:02:44 +02:00
Eelco Dolstra
30c8c3ba2d Fix some more references to storeDir 2016-06-03 14:02:44 +02:00
Eelco Dolstra
a8dfdc52b8 Merge pull request #925 from abbradar/master
Show both cycle ends
2016-06-03 13:58:51 +02:00
Nikolay Amiantov
c87a56f4d0 Show both cycle ends 2016-06-03 14:25:37 +03:00
Eelco Dolstra
75361b6dce Fix narrowing conversion from int64_t to bool
http://hydra.nixos.org/build/36613774
2016-06-03 13:20:46 +02:00
Eelco Dolstra
f8a8b4d8f8 nix run: Set a reasonable uid/gid 2016-06-02 19:04:09 +02:00
Eelco Dolstra
d64e0c1b64 Make derivationFromPath work on diverted stores 2016-06-02 18:43:36 +02:00
Eelco Dolstra
eda2aaae92 nix run: Handle the case where the /nix/store mount point doesn't exist 2016-06-02 18:24:51 +02:00
Eelco Dolstra
a424ab0444 createTempDir(): Don't do a chown on Linux
It's not needed and can cause problems in a user namespace.
2016-06-02 18:17:30 +02:00
Eelco Dolstra
a24f2c9b84 nix run: Mount the Nix store in a private namespace
This is a convenience command to allow users who are not privileged to
create /nix/store to use Nix with regular binary caches. For example,

  $ NIX_REMOTE="local?state=$HOME/nix/var&real=/$HOME/nix/store" nix run firefox bashInteractive

will download Firefox and bash from cache.nixos.org, then start a
shell in which $HOME/nix/store is mounted on /nix/store.
2016-06-02 16:51:43 +02:00
Eelco Dolstra
6f2d51287c Add basic "nix run" command 2016-06-02 16:29:49 +02:00
Eelco Dolstra
4494000e04 LocalStore: Allow the physical and logical store directories to differ
This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):

  $ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'

However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:

  $ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
2016-06-02 16:02:48 +02:00
Eelco Dolstra
064816ab98 Respect build-use-substitutes 2016-06-02 16:02:48 +02:00
Eelco Dolstra
812c0dfbe2 Allow setting the state directory as a store parameter
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Eelco Dolstra
f2682e6e18 <nix/fetchurl.nix>: Use tarballs.nixos.org
This ensures that the disappearance of upstream bootstrap tarballs
(e.g. https://github.com/NixOS/nixpkgs-channels/pull/1) doesn't break
stdenv rebuilds.
2016-06-01 17:11:51 +02:00
Eelco Dolstra
dead8189ac Typo 2016-06-01 16:56:22 +02:00
Eelco Dolstra
e24e2caaaf Skip substituters with an incompatible store directory 2016-06-01 16:40:49 +02:00
Eelco Dolstra
cf198952d0 HttpBinaryCacheStore: Fix caching of WantMassQuery
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
2016-06-01 16:24:17 +02:00
Eelco Dolstra
7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra
a9fa5e050a Shut up some clang warnings 2016-05-31 13:31:04 +02:00
Eelco Dolstra
10f3a2e5f2 Fix clang build failure
Apparently opinion is divided on whether [[noreturn]] is allowed on a
lambda: http://stackoverflow.com/questions/26888805/how-to-declare-a-lambdas-operator-as-noreturn

http://hydra.nixos.org/build/36462100
2016-05-31 13:23:54 +02:00
Eelco Dolstra
c2d27d30cf nix-copy-closure / build-remote.pl: Disable signature checking
This restores the Nix 1.11 behaviour.
2016-05-31 11:48:05 +02:00
Dmitry Kalinkin
78b00bbd8a use $(LIBLZMA_LIBS) instead of -llzma
This is needed in case of non-standard lzma installation path that will
be specified in pkgconfig manifest as extra -L option for LDFLAGS.
2016-05-31 03:20:11 -04:00
Eelco Dolstra
cd35f0280c Fix build 2016-05-30 16:12:50 +02:00
Eelco Dolstra
57d33013ce Check signatures before downloading the substitute 2016-05-30 15:18:12 +02:00
Eelco Dolstra
bac123ddd9 Test the NAR info cache 2016-05-30 15:18:12 +02:00
Eelco Dolstra
3593c8285d Re-implement binary cache signature checking
This is now done in LocalStore::addToStore(), rather than in the
binary cache substituter (which no longer exists).
2016-05-30 15:18:12 +02:00
Eelco Dolstra
12ddbad458 LocalStore::addToStore: Verify hash of the imported path 2016-05-30 15:18:12 +02:00
Eelco Dolstra
e222484401 Re-implement the WantMassQuery property of binary caches 2016-05-30 15:18:12 +02:00
Eelco Dolstra
b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra
3be2e71ab3 BinaryCacheStore: Remove buildPaths() / ensurePath() 2016-05-30 15:18:12 +02:00
Eelco Dolstra
75d2492f20 Make the aws-cpp-sdk dependency optional 2016-05-04 17:16:48 +02:00
Eelco Dolstra
6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra
0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra
c6a21aed07 More GC fixes 2016-05-04 16:16:53 +02:00
Eelco Dolstra
538a64e8c3 Add a Store::addToStore() variant that accepts a NAR
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra
b6c768fb6a nix-daemon: Fix queryPathInfo() backward compatibility 2016-05-04 12:53:24 +02:00
Eelco Dolstra
33de2bc080 Fix segfault in xz compression/decompression 2016-05-04 11:50:12 +02:00
Eelco Dolstra
f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra
dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra
d8bf0d4859 Support Git repos in the Nix path
E.g.

  $ nix-build -I nixpkgs=git://github.com/NixOS/nixpkgs '<nixpkgs>' -A hello

This is not extremely useful yet because you can't specify a
branch/revision.
2016-04-29 21:07:47 +02:00
Eelco Dolstra
38539b943a Add fetchgit builtin
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
2016-04-29 20:47:36 +02:00
Eelco Dolstra
83258225e6 nix-store -l: Simplify implementation 2016-04-29 17:47:03 +02:00
Eelco Dolstra
4dde0b0562 BinaryCacheStore: Support bzip2 compression 2016-04-29 17:43:37 +02:00
Eelco Dolstra
6963de2091 nix verify --all: Support local binary caches 2016-04-29 17:34:31 +02:00
Eelco Dolstra
0dd988d2e3 Better error message 2016-04-29 17:33:22 +02:00
Eelco Dolstra
5acb691402 BinaryCacheStore: Support "none" compression method 2016-04-29 17:02:57 +02:00
Eelco Dolstra
8e065c6b3e BinaryCacheStore: Make the signing key a parameter 2016-04-29 16:47:20 +02:00
Eelco Dolstra
f6aee2f477 BinaryCacheStore::queryPathInfo: Don't check signatures
Other stores don't do this either. It's up to the caller to check
signatures.
2016-04-29 16:28:57 +02:00
Eelco Dolstra
95d20dfde9 Allow parameters in store URIs
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-29 16:26:16 +02:00
Eelco Dolstra
aa3bc3d5dc Eliminate the substituter mechanism
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Eelco Dolstra
21e9d183cc Really handle carriage return 2016-04-28 14:27:00 +02:00
Eelco Dolstra
ce5776758d Fix error message 2016-04-28 14:12:10 +02:00
Eelco Dolstra
cc804d0dc6 Handle carriage return 2016-04-25 19:18:45 +02:00
Eelco Dolstra
ca9de88a51 nix: Disable verbose builds 2016-04-25 19:18:45 +02:00
Eelco Dolstra
5761827d5b Show the log tail when a build fails
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
6e1b099279 Remove --print-build-trace
This was added to support Hydra, but Hydra no longer uses it.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
9eba2c3945 Fix "path is not in the Nix store" during GC 2016-04-25 19:18:45 +02:00
Eelco Dolstra
697e0b3a6f Remove nix-log2xml 2016-04-25 19:18:45 +02:00
Eelco Dolstra
41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
c879a20850 Factor out parallel processing of work items that have dependencies 2016-04-22 20:50:06 +02:00
Eelco Dolstra
91539d305f nix copy: Parallelise 2016-04-22 18:19:48 +02:00
Eelco Dolstra
b2ce6fde5a ThreadPool: Start doing work as soon as work items are enqueued 2016-04-22 18:19:17 +02:00
Eelco Dolstra
58c84cda3b Make compression interruptible 2016-04-22 18:15:02 +02:00
Eelco Dolstra
456179018a Fold "nix query-path-sigs" into "nix path-info" 2016-04-22 14:39:37 +02:00
Eelco Dolstra
95abf9c402 Add "nix copy" command
This replaces nix-push. For example,

  $ nix copy --to file:///tmp/cache -r $(type -p firefox)

copies the closure of firefox to the specified binary cache. And

  $ nix copy --from file:///tmp/cache --to s3://my-cache /nix/store/abcd...

copies between two binary caches.

It will also replace nix-copy-closure, once we have an SSHStore class,
e.g.

  $ nix copy --from ssh://alice@machine /nix/store/abcd...
2016-04-22 14:33:01 +02:00
Eelco Dolstra
0207272b28 BinaryCacheStore: When adding a path, ensure the references are valid
This prevents copying a partial closure to a binary cache.
2016-04-22 12:15:06 +02:00
Eelco Dolstra
21ef1670b3 Fix test failures 2016-04-21 18:21:25 +02:00
Eelco Dolstra
7d14f5c331 Implement S3BinaryCacheStore::queryAllValidPaths()
This allows commands like "nix verify --all" or "nix path-info --all"
to work on S3 caches.

Unfortunately, this requires some ugly hackery: when querying the
contents of the bucket, we don't want to have to read every .narinfo
file. But the S3 bucket keys only include the hash part of each store
path, not the name part. So as a special exception
queryAllValidPaths() can now return store paths *without* the name
part, and queryPathInfo() accepts such store paths (returning a
ValidPathInfo object containing the full name).
2016-04-21 17:53:47 +02:00
Eelco Dolstra
d155d80155 Move S3BinaryCacheStore from Hydra
This allows running arbitrary Nix commands against an S3 binary cache.

To do: make this a compile time option to prevent a dependency on
aws-sdk-cpp.
2016-04-21 16:08:51 +02:00
Eelco Dolstra
1a71495273 nix path-info: Add
Forgot to commit this earlier...
2016-04-21 15:00:45 +02:00
Eelco Dolstra
69e3ffb076 nix --help: Show usage examples 2016-04-21 14:58:32 +02:00
Eelco Dolstra
1b0088ebb2 nix --help: Show short flags 2016-04-21 14:34:46 +02:00
Eelco Dolstra
ddea253ff8 RemoteStore: Propagate InvalidPath exceptions from the daemon 2016-04-20 15:28:07 +02:00
Eelco Dolstra
c0c4ddcd9c BinaryCacheStore: Insert new paths into the disk cache 2016-04-20 15:27:48 +02:00
Eelco Dolstra
451ebf24ce Cache path info lookups in SQLite
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
2016-04-20 14:12:38 +02:00
Eelco Dolstra
e0204f8d46 Move path info caching from BinaryCacheStore to Store
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
2016-04-19 18:52:53 +02:00
Dan Peebles
608b0265e1 Print out all bad references/requisites at once
Also updates tests to check for new information. Fixes #799
2016-04-16 19:58:26 -04:00
Eelco Dolstra
0423787086 Make the .narinfo cache bigger 2016-04-15 15:39:48 +02:00
Eelco Dolstra
a7d8eaba54 BinaryCacheStore: Do negative caching of .narinfo lookups 2016-04-15 15:39:48 +02:00
Eelco Dolstra
d1b0909894 BinaryCacheStore::readFile(): Return a shared_ptr to a string
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
2016-04-15 15:39:48 +02:00
Eelco Dolstra
99851c6f06 Unify "nix verify-paths" and "nix verify-store"
"verify-store" is now simply an "--all" flag to "nix verify". This
flag can be used for any other store path command as well (e.g. "nix
path-info", "nix copy-sigs", ...).
2016-04-15 15:39:48 +02:00
Eelco Dolstra
31a1a8ed3c Merge pull request #815 from vcunat/p/outputsToInstall
nix-env: respect meta.outputsToInstall
2016-04-14 20:55:41 +02:00
Eelco Dolstra
5169a6da98 Make $NIX_PATH parsing more robust 2016-04-14 17:29:08 +02:00
Eelco Dolstra
c045630522 Support channel:<channel-name> URIs
For convenience, you can now say

  $ nix-env -f channel:nixos-16.03 -iA hello

instead of

  $ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello

Similarly,

  $ nix-shell -I channel:nixpkgs-unstable -p hello
  $ nix-build channel:nixos-15.09 -A hello

Abstracting over the NixOS/Nixpkgs channels location also allows us to
use a more efficient transport (e.g. Git) in the future.
2016-04-14 17:26:57 +02:00
Eelco Dolstra
363f37d084 Make the search path lazier with non-fatal errors
Thus, -I / $NIX_PATH entries are now downloaded only when they are
needed for evaluation. An error to download an entry is a non-fatal
warning (just like non-existant paths).

This does change the semantics of builtins.nixPath, which now returns
the original, rather than resulting path. E.g., before we had

  [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

but now

  [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

Fixes #792.
2016-04-14 15:32:24 +02:00
Eelco Dolstra
3c1c6b8f00 Set RLIMIT_CORE to 0, infinity in builders
This prevents the builder from being affected by whatever the host
system limits happen to be.
2016-04-14 13:39:14 +02:00
Eelco Dolstra
12b257f045 Make primop registration pluggable
This way we don't have to put all primops in one giant file.
2016-04-14 12:50:01 +02:00
Eelco Dolstra
96515b0c0d Fix std::atomic_flag build failure
http://hydra.nixos.org/build/34453794
2016-04-14 12:50:01 +02:00
Eelco Dolstra
867967265b Remove manifest support
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
2016-04-11 16:20:15 +02:00
Eelco Dolstra
8cffec8485 Remove failed build caching
This feature was implemented for Hydra, but Hydra no longer uses it.
2016-04-08 18:19:04 +02:00
Eelco Dolstra
f398949b40 Make LocalStore thread-safe
Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Eelco Dolstra
05fbc606fc nix verify-paths: Add ‘--sigs-needed <N>’ flag
This specifies the number of distinct signatures required to consider
each path "trusted".

Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
2016-04-07 15:16:57 +02:00
Eelco Dolstra
6b2ae52808 Use secret-key-files for verifying 2016-04-07 15:16:57 +02:00
Eelco Dolstra
e39999ed48 Sign locally-built paths
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
2016-04-07 15:16:57 +02:00
Eelco Dolstra
dc82160164 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http://
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
2016-04-06 16:57:20 +02:00
Eelco Dolstra
b654381eb3 Add "nix sign-paths" command
E.g.

  $ nix sign-paths -k ./secret -r $(type -p geeqie)

signs geeqie and all its dependencies using the key in ./secret.
2016-04-05 16:39:29 +02:00
Eelco Dolstra
d0f5719c2a Add "nix copy-sigs" command
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
2016-04-05 16:39:23 +02:00
Eelco Dolstra
80da7a6375 Probably fix SQLITE_BUSY errors 2016-04-05 15:29:56 +02:00
Eelco Dolstra
37a337bcec throwSQLiteError(): Check for SIGINT so we don't loop forever 2016-04-04 15:08:08 +02:00
Eelco Dolstra
6e120b76ee Add missing -pthread
https://hydra.nixos.org/build/33908385
2016-03-31 12:42:48 +02:00
Eelco Dolstra
9cee600c88 LocalStore: Keep track of ultimately trusted paths
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
2016-03-30 17:35:48 +02:00
Eelco Dolstra
3d119f0a3b Improve the SQLite wrapper API
In particular, this eliminates a bunch of boilerplate code.
2016-03-30 15:50:45 +02:00
Eelco Dolstra
d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra
2ae43ced9a Turn retrying SQLite transactions into a higher-order function 2016-03-30 12:04:27 +02:00
Eelco Dolstra
39a6abc0bc nix verify: Support checking against signatures in other stores
Typical usage is to check local paths using the signatures from a
binary cache:

  $ nix verify-paths -r /run/current-system -s https://cache.nixos.org
  path ‘/nix/store/c1k4zqfb74wba5sn4yflb044gvap0x6k-nixos-system-mandark-16.03.git.fc2d7a5M’ is untrusted
  ...
  checked 844 paths, 119 untrusted
2016-03-30 11:39:34 +02:00
Eelco Dolstra
88541569a2 HttpBinaryCacheStore: Treat 403 errors as 404 2016-03-30 11:17:51 +02:00
Eelco Dolstra
de88004a9d CurlDownloader: Fix HTTP error processing 2016-03-30 11:17:35 +02:00
Eelco Dolstra
ab3ce1cc13 Improve SIGINT handling in multi-threaded programs
The flag remembering whether an Interrupted exception was thrown is
now thread-local. Thus, all threads will (eventually) throw
Interrupted. Previously, one thread would throw Interrupted, and then
the other threads wouldn't see that they were supposed to quit.
2016-03-29 16:37:16 +02:00
Eelco Dolstra
4f34c40398 Add "nix verify-store" command
Like "nix-store --verify --check-contents", but with the same
advantages as "nix verify-paths".
2016-03-29 16:37:16 +02:00
Eelco Dolstra
784ee35c80 Add "nix verify-paths" command
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
2016-03-29 16:37:16 +02:00
Eelco Dolstra
0ebe69dc67 Re-enable sync_with_stdio
Otherwise writing to std::cerr is not thread-safe (in particular,
lines will be randomly duplicated).
2016-03-29 16:37:16 +02:00
Eelco Dolstra
374198ad6d Move signature support from NarInfo to ValidPathInfo 2016-03-24 11:41:00 +01:00
Eelco Dolstra
11525377e1 Typos 2016-03-24 11:27:58 +01:00
Eelco Dolstra
8b7839b608 HttpBinaryCacheStore: Make thread-safe 2016-03-24 11:10:05 +01:00
Eelco Dolstra
c7d44bad00 Drop support for daemon socket path >= 108 characters
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as

  error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
2016-03-23 17:16:16 +01:00
Eelco Dolstra
056b3ecfa4 LocalStoreAccessor::stat: Handle ENOTDIR
Closes https://github.com/NixOS/hydra/pull/286.
2016-03-23 11:17:46 +01:00
Eelco Dolstra
bb1034316d Don't overload dumpPath() 2016-03-22 14:21:45 +01:00
Eelco Dolstra
712b616a84 Move signatures from NarInfo to ValidPathInfo
This allows queryPathInfo() to return signatures.
2016-03-21 18:05:47 +01:00
Eelco Dolstra
cebc150b7c nix: Add --store flag
This is a bit user-friendlier than using $NIX_REMOTE.
2016-03-21 18:03:36 +01:00
Eelco Dolstra
1c5f73f529 Add Store::dumpPath() method
This allows applying nix-store --verify-path to binary cache stores:

  NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
2016-03-21 17:55:57 +01:00
Eelco Dolstra
87295b9844 Drop support for upgrading from Nix <= 0.12 2016-03-21 15:09:03 +01:00
Eelco Dolstra
02654f782f Fix Darwin build
http://hydra.nixos.org/build/33279996
2016-03-15 12:11:27 +01:00
Eelco Dolstra
00a75b1cd2 Fix another mismatched tag
http://hydra.nixos.org/build/33279570
2016-03-14 15:32:34 +01:00
Eelco Dolstra
dc4a71aae5 Fix build on clang due to -Wmismatched-tags
http://hydra.nixos.org/build/33073389
2016-03-14 12:37:30 +01:00
Eelco Dolstra
e7c76f7274 BinaryCacheStore::isValidPath(): Use .narinfo cache
If a path is in the .narinfo cache, obviously it's valid.
2016-03-14 12:33:06 +01:00
Eelco Dolstra
674c5ff64f Merge pull request #834 from abbradar/master
Propagate path context via builtins.readFile
2016-03-10 20:17:03 +01:00
Eelco Dolstra
b4e0335d4d Add option binary-cache-secret-key-file for signing binary caches 2016-03-04 17:45:22 +01:00
Eelco Dolstra
af7cdb1096 BinaryCacheStore: Remove publicKeyFile argument
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
2016-03-04 17:45:22 +01:00
Eelco Dolstra
42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra
ce113c32d2 Add warning about "nix" being experimental 2016-03-04 15:54:41 +01:00
Eelco Dolstra
7c9d7a253c Merge branch 'new-cli' 2016-03-04 15:21:23 +01:00
Nikolay Amiantov
f7f0116dd7 Propagate path context via builtins.readFile 2016-03-04 14:04:51 +03:00
Eelco Dolstra
76f1ba4f3b Add file missing from 201b48de60 2016-03-03 18:03:34 +01:00
Nathan Zadoks
fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra
5a8455c85e Provide function required by Hydra 2016-03-02 18:21:48 +01:00
Eelco Dolstra
25eea97873 Force stack trace for boost format errors 2016-03-02 15:46:07 +01:00
Eelco Dolstra
201b48de60 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-03-02 15:46:07 +01:00
Eelco Dolstra
0402b6398d Eliminate local-binary-cache-store.hh 2016-03-02 15:46:07 +01:00
Eelco Dolstra
68a5414982 Make store implementations pluggable
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
2016-03-02 15:46:07 +01:00
Eelco Dolstra
6055d84beb Fix reading symlinks
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
2016-03-02 15:46:07 +01:00
Eelco Dolstra
0a62d9b3d7 Remove bad assertion 2016-03-02 15:46:07 +01:00
Eelco Dolstra
89a2fa68ac FdSource: track number of bytes read 2016-03-02 15:46:07 +01:00
Eelco Dolstra
cca4a8dc1a importPaths(): Optionally add NARs to binary cache accessor
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
2016-03-02 15:46:07 +01:00
Eelco Dolstra
e9c50064b5 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-02-29 18:15:20 +01:00
Eelco Dolstra
6170bb474b Eliminate local-binary-cache-store.hh 2016-02-29 16:14:39 +01:00
Eelco Dolstra
0b907321cc Make store implementations pluggable
This for instance allows hydra-queue-runner to add the S3 backend
at runtime.
2016-02-29 16:11:11 +01:00
Eelco Dolstra
012f8d187c Fix reading symlinks
The st_size field of a symlink doesn't have to be correct, e.g. for
/dev/fd symlinks.
2016-02-28 03:27:34 +01:00
Eelco Dolstra
d0344dd2c1 Remove bad assertion 2016-02-26 21:43:59 +01:00
Eelco Dolstra
581e1bc5b4 FdSource: track number of bytes read 2016-02-26 16:16:08 +01:00
Eelco Dolstra
bcc9943cee importPaths(): Optionally add NARs to binary cache accessor
This enables an optimisation in hydra-queue-runner, preventing a
download of a NAR it just uploaded to the cache when reading files
like hydra-build-products.
2016-02-26 15:20:10 +01:00
Eelco Dolstra
00b2c05749 nix: Add commands to query contents of NARs / binary caches
For example,

  $ NIX_REMOTE=file:///my-cache nix ls-store -lR /nix/store/f4kbgl8shhyy76rkk3nbxr0lz8d2ip7q-binutils-2.23.1
  dr-xr-xr-x                    0 ./bin
  -r-xr-xr-x                30748 ./bin/addr2line
  -r-xr-xr-x                66973 ./bin/ar
  ...

Similarly, "nix ls-nar" lists the contents of a NAR file, "nix
cat-nar" extracts a file from a NAR file, and "nix cat-store" extract
a file from a Nix store.
2016-02-25 17:57:00 +01:00
Eelco Dolstra
24a8f9e27b Merge branch 'master' into new-cli 2016-02-25 17:48:35 +01:00
Eelco Dolstra
1042c10fd0 Add NAR / Store accessor abstraction
This is primary to allow hydra-queue-runner to extract files like
"nix-support/hydra-build-products" from NARs in binary caches.
2016-02-25 17:43:19 +01:00
Eelco Dolstra
c5bc571861 Fix short boolean flags 2016-02-25 13:31:34 +01:00
Eelco Dolstra
7873cfb18d Fix build 2016-02-25 11:55:05 +01:00
Eelco Dolstra
f1bdeac986 Merge branch 'master' into new-cli 2016-02-25 11:25:11 +01:00
Eelco Dolstra
152b1d6bf9 deletePath(): Succeed if path doesn't exist
Also makes it robust against concurrent deletions.
2016-02-24 17:44:55 +01:00
Eelco Dolstra
28e7e29abd Eliminate reserveSpace flag 2016-02-24 17:44:55 +01:00
Eelco Dolstra
5a64e66268 LocalStore::addTextToStore(): Don't read the path we just wrote
This eliminates some unnecessary (presumably cached) I/O.
2016-02-24 17:11:31 +01:00
Eelco Dolstra
9ccbd55c5b BinaryCacheStore: Implement addToStore()
So now you can do

  $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello

and lots of other operations.
2016-02-24 16:52:28 +01:00
Eelco Dolstra
30e9d01516 openStoreAt(): Initialise the binary cache 2016-02-24 16:52:07 +01:00
Eelco Dolstra
ba0a81d14f Show a proper error message for unimplemented binary cache operations 2016-02-24 14:57:30 +01:00
Eelco Dolstra
45c83e5f9b Typo 2016-02-24 14:49:28 +01:00
Eelco Dolstra
263187a2ec Move BinaryCacheStore / LocalBinaryCacheStore from Hydra
So you can now do:

  $ NIX_REMOTE=file:///tmp/binary-cache nix-store -qR /nix/store/...
2016-02-24 14:48:16 +01:00
Eelco Dolstra
b584a0e7de Merge pull request #816 from tsion/incomplete-parse
Throw a specific error for incomplete parse errors.
2016-02-24 13:34:56 +01:00
Eelco Dolstra
bf2adf72c4 std::condition_variable_any -> std::condition_variable
The latter is supposed to be more efficient.
2016-02-24 13:31:46 +01:00
Eelco Dolstra
ccdbf589a4 C++ templates are just a glorified macro facility 2016-02-24 13:07:32 +01:00
Eelco Dolstra
5f862658c3 Remove bad daemon connections from the pool
This is necessary for long-running processes like hydra-queue-runner:
if a nix-daemon worker is killed, we need to stop reusing that
connection.
2016-02-24 11:39:56 +01:00
Scott Olson
6498adb002 Throw a specific error for incomplete parse errors.
`nix-repl` will use this for deciding whether to keep waiting for input or
error out right away.
2016-02-24 04:32:21 -06:00
Eelco Dolstra
d5626bf4c1 Pool<T>: Allow a maximum pool size 2016-02-23 16:40:16 +01:00
Eelco Dolstra
e292144d46 RemoteStore: Make thread-safe
This allows a RemoteStore object to be used safely from multiple
threads concurrently. It will make multiple daemon connections if
necessary.

Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 15:00:59 +01:00
Vladimír Čunát
03cbb9ad59 nix-env: respect meta.outputsToInstall
Discussed on https://github.com/NixOS/nixpkgs/pull/12653#discussion_r51601849
2016-02-23 14:19:14 +01:00
Eelco Dolstra
c0b7a8a0b5 Move ref<t> into a separate header 2016-02-23 13:53:31 +01:00
Eelco Dolstra
8f71bc33d5 Doh 2016-02-22 17:33:27 +01:00
Eelco Dolstra
840056af04 Make OpenSSL usage thread-safe
OpenSSL can randomly segfault unless we register a callback function
to do locking.

https://www.openssl.org/docs/manmaster/crypto/threads.html
2016-02-22 15:51:14 +01:00
Eelco Dolstra
d361901bfe curl: Set CURLOPT_NOSIGNAL
Otherwise using curl is not safe in multi-threaded applications
because it installs a SIGALRM handler.
2016-02-22 15:51:14 +01:00
Eelco Dolstra
7a173a7be1 JSONObject: Support floats and booleans 2016-02-19 14:24:04 +01:00
Eelco Dolstra
b49d323ce2 Fix build without sodium
http://hydra.nixos.org/build/32085949
2016-02-17 12:42:14 +01:00
Eelco Dolstra
c4d22997f3 Add C++ functions for .narinfo processing / signing
This is currently only used by the Hydra queue runner rework, but like
eff5021eaa it presumably will be useful
for the C++ rewrite of nix-push and
download-from-binary-cache. (@shlevy)
2016-02-16 16:38:44 +01:00
Eelco Dolstra
5ac27053e9 Rename ValidPathInfo::hash -> narHash for consistency 2016-02-16 11:49:12 +01:00
Tristan Hume
9b05d5848c Fix to-base16 description 2016-02-16 10:15:58 +01:00
Eelco Dolstra
eff5021eaa Add xz compression function
This is used by the Hydra queue runner, but since it may also be
useful for the C++ rewrite of nix-push, I'm putting it here.
2016-02-15 21:45:56 +01:00
Eelco Dolstra
bfdacb712c decompressXZ: Ensure that lzma_end() is called
Otherwise we might leak memory.
2016-02-15 21:26:28 +01:00
Eelco Dolstra
c8f4d89a34 Expose the export magic value and move LocalStore::queryReferences to Store 2016-02-15 15:01:26 +01:00
Eelco Dolstra
e03d6e0998 Fix broken number parsing in fromJSON
The call to tmp_number.append had its arguments mixed up. Also, JSON
does not allow a trailing "," after array/object members.
2016-02-15 15:01:26 +01:00
Eelco Dolstra
d089372565 Add function to extract hash part of a store path 2016-02-15 15:01:26 +01:00
Eelco Dolstra
b3e8d72770 Merge pull request #762 from ctheune/ctheune-floats
Implement floats
2016-02-12 12:49:59 +01:00
Eelco Dolstra
ae4a3cfa03 Move addPermRoot into Store 2016-02-11 16:14:42 +01:00
Eelco Dolstra
fd205fb6f8 ref<T>: Add cast operator 2016-02-11 15:32:48 +01:00
Eelco Dolstra
206bbb5dc9 Add basic "nix build" command
Currently only builds by attribute from <nixpkgs> or the specified
file, e.g. "nix build hello".
2016-02-09 21:34:24 +01:00
Eelco Dolstra
cd2196b089 Start of new Nix command-line interface 2016-02-09 21:28:29 +01:00
Eelco Dolstra
0db9e6cd1a New command line parsing infrastructure 2016-02-09 21:07:48 +01:00
Eelco Dolstra
c780c1124e More of the same 2016-02-04 15:10:47 +01:00
Eelco Dolstra
fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra
c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra
d367b8e787 Add build mode to compute fixed-output derivation hashes
For example,

  $ nix-build --hash -A nix-repl.src

will build the fixed-output derivation nix-repl.src (a fetchFromGitHub
call), but instead of *verifying* the hash given in the Nix
expression, it prints out the resulting hash, and then moves the
result to its content-addressed location in the Nix store. E.g

  build produced path ‘/nix/store/504a4k6zi69dq0yjc0bm12pa65bccxam-nix-repl-8a2f5f0607540ffe56b56d52db544373e1efb980-src’ with sha256 hash ‘0cjablz01i0g9smnavhf86imwx1f9mnh5flax75i615ml71gsr88’

The goal of this is to make all nix-prefetch-* scripts unnecessary: we
can just let Nix run the real thing (i.e., the corresponding fetch*
derivation).

Another example:

  $ nix-build --hash -E 'with import <nixpkgs> {}; fetchgit { url = "https://github.com/NixOS/nix.git"; sha256 = "ffffffffffffffffffffffffffffffffffffffffffffffffffff"; }'
  ...
  git revision is 9e7c1a4bbd
  ...
  build produced path ‘/nix/store/gmsnh9i7x4mb7pyd2ns7n3c9l90jfsi1-nix’ with sha256 hash ‘1188xb621diw89n25rifqg9lxnzpz7nj5bfh4i1y3dnis0dmc0zp’

(Having to specify a fake sha256 hash is a bit annoying...)
2016-01-31 12:18:09 +01:00
Eelco Dolstra
9e7c1a4bbd Use the daemon when we don't have write access to the Nix database 2016-01-31 10:28:48 +01:00
Eelco Dolstra
4fa08f3edb Untabify 2016-01-28 16:03:32 +01:00
Eelco Dolstra
26566cd28e Untabify 2016-01-28 16:01:01 +01:00
Eelco Dolstra
9f14d7d33a printMissing(): Propagate store argument 2016-01-28 15:16:06 +01:00
Eelco Dolstra
7899fc959d Define HashType's size to allow it to be forward-declared 2016-01-27 17:46:01 +01:00
Eelco Dolstra
739bab0be7 Trivia 2016-01-27 17:18:31 +01:00
Eelco Dolstra
d45ad8fcf5 Make hashLength32() a method of Hash 2016-01-27 17:18:20 +01:00
Eelco Dolstra
5b8c09c124 string2Int: Barf on negative numbers for unsigned types 2016-01-20 16:45:05 +01:00
Eelco Dolstra
5d8b7eb3e1 Revert "Revert "next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"""
This reverts commit b669d3d2e8.
2016-01-20 16:34:42 +01:00
Eelco Dolstra
b669d3d2e8 Revert "next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751""
This reverts commit ed23c8568e. Let's
merge this *after* the 1.11.1 release.
2016-01-20 00:05:28 +01:00
Fabian Schmitthenner
ed23c8568e next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"
This reverts commit 8120b6fb8a and fixes the regression introduced in
8d22b26448.
2016-01-19 20:35:35 +00:00
Eelco Dolstra
8120b6fb8a Revert "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"
This reverts commit 8d22b26448. It
breaks Nixpkgs:

$ nix-env -qa
error: syntax error, unexpected IND_STR, expecting '}', at /home/eelco/Dev/nixpkgs-stable/pkgs/top-level/python-packages.nix:7605:8
2016-01-19 20:33:32 +01:00
Shea Levy
a6d52fcd95 Merge branch 'lexer' of git://github.com/fkz/nix 2016-01-17 10:08:03 -05:00
Fabian Schmitthenner
8d22b26448 don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751 2016-01-12 20:40:41 +00:00
Eelco Dolstra
786046cf13 --option build-repeat: Keep the differing output if -K is given
Similar to 00903fa799. Regardless of -K,
we now also print which output differs.
2016-01-12 18:26:24 +01:00
Eelco Dolstra
8906eda2f9 Canonicalize gids to 0
Previously files in the Nix store were owned by root or by nixbld,
depending on whether they were created by a substituter or by a
builder. This doesn't matter much, but causes spurious diffoscope
differences. So use root everywhere.
2016-01-12 17:27:40 +01:00
Eelco Dolstra
00903fa799 --check: Keep the differing output if -K is given
This makes it easier to investigate the non-determinism, e.g.

  $ nix-build pkgs/stdenv/linux -A stage1.pkgs.zlib --check -K
  error: derivation ‘/nix/store/l54i8wlw22656i4pk05c52ngv9rpl39q-zlib-1.2.8.drv’ may not be deterministic: output ‘/nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8’ differs from ‘/nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8-check’

  $ diffoscope /nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8 /nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8-check
  ...
  ├── lib/libz.a
  │   ├── metadata
  │   │ @@ -1,15 +1,15 @@
  │   │ -rw-r--r-- 30001/30000   3096 Jan 12 15:20 2016 adler32.o
  ...
  │   │ +rw-r--r-- 30001/30000   3096 Jan 12 15:28 2016 adler32.o
  ...
2016-01-12 16:44:26 +01:00
Eelco Dolstra
0cad1f8049 --check: Fix "failed to produce output path"
This occured when sandbox building is disabled, at least one output
exists, and at least one other output does not.
2016-01-12 15:13:47 +01:00
Eelco Dolstra
1c57ab8b31 --check: Fix assertion failure when some outputs are missing
E.g.

  $ nix-build pkgs/stdenv/linux/ -A stage1.pkgs.perl --check
  nix-store: src/libstore/build.cc:1323: void nix::DerivationGoal::tryToBuild(): Assertion `buildMode != bmCheck || validPaths.size() == drv->outputs.size()' failed.

when perl.out exists but perl.man doesn't. The fix is to only check
the outputs that exist. Note that "nix-build -A stage1.pkgs.all
--check" will still give a (proper) error in this case.
2016-01-12 14:54:39 +01:00
Eelco Dolstra
458711e4ee Fix "Bad address" executing build hook
This was observed in the deb_debian7x86_64 build:
http://hydra.nixos.org/build/29973215

Calling c_str() on a temporary should be fine because the temporary
shouldn't be destroyed until after the execl() call, but who knows...
2016-01-07 15:10:14 +01:00
Eelco Dolstra
9aac1861f7 Fix some signedness warnings 2016-01-07 14:37:39 +01:00
Eelco Dolstra
02a66b3fd7 nix-store --repair-path: Rebuild if there is no substituter 2016-01-06 22:07:59 +01:00
Eelco Dolstra
caaaff3954 Fix --repair failure on multiple-output derivations
If repair found a corrupted/missing path that depended on a
multiple-output derivation, and some of the outputs of the latter were
not present, it failed with a message like

  error: path ‘/nix/store/cnfn9d5fjys1y93cz9shld2xwaibd7nn-bash-4.3-p42-doc’ is not valid
2016-01-06 21:49:32 +01:00
Christian Theune
934642155c @eelco's feedback: downgrade to regular float for size, remove unused function. 2016-01-06 08:25:58 +01:00
Fabian Schmitthenner
39d1da7b51 Better error message
Also show types when nix cannot compare values of different types.
This is also more consistent since types are already shown when comparing values of the same not comparable type.
2016-01-05 14:13:54 +01:00
Eelco Dolstra
9a4706eb19 Fix compilation 2016-01-05 14:05:11 +01:00
Shea Levy
14080f3e4b Use __toString when coercing sets to strings.
For example, "${{ foo = "bar"; __toString = x: x.foo; }}" evaluates
to "bar".

With this, we can delay calling functions like mkDerivation,
buildPythonPackage, etc. until we actually need a derivation, enabling
overrides and other modifications to happen by simple attribute set
update.
2016-01-05 13:55:06 +01:00
Eelco Dolstra
6298afc047 Merge pull request #685 from vizanto/master
POSIX compliant directory access (fixes build on Solaris)
2016-01-05 13:49:55 +01:00
FrankHB
607d91517e Fixed unresolved 'BUFSIZ'
`BUFSIZ` is defined in header `<cstdio>`.
2016-01-05 13:33:40 +01:00
Eelco Dolstra
743e310046 Fix non-Darwin build 2016-01-05 13:31:15 +01:00
Vladimír Čunát
29c0199d5c manual: document builtins.functionArgs
The text is just a conversion of comment from src/libexpr/primops.cc
2016-01-05 13:29:17 +01:00
Tuomas Tynkkynen
f770b9e6c5 libstore: mmap() returns MAP_FAILED, not NULL on failure 2016-01-05 13:26:35 +01:00
Christian Theune
a12a43046b Edge condition: parser did not pick up floats starting exactly with 0. 2016-01-05 09:54:49 +01:00
Christian Theune
f872262e08 Fix up float parsing. 2016-01-05 09:46:37 +01:00
Christian Theune
494fc5acbb Try a simplified version of float lexing that didn't work.
The last one I tried was botchered anyway ...
2016-01-05 00:53:22 +01:00
Christian Theune
14ebde5289 First hit at providing support for floats in the language. 2016-01-05 00:40:40 +01:00
Eelco Dolstra
71a93a5f0e Don't allow sandbox profile except in relaxed mode
This makes Darwin consistent with Linux: Nix expressions can't break
out of the sandbox unless relaxed sandbox mode is enabled.

For the normal sandbox mode this will require fixing #759 however.
2016-01-04 20:01:13 +01:00
Eelco Dolstra
77ad443bd1 ~PathLocks(): Handle exceptions
Otherwise, since the call to write a "d" character to the lock file
can fail with ENOSPC, we can get an unhandled exception resulting in a
call to terminate().
2016-01-04 11:34:36 +01:00
Eelco Dolstra
b8258a4475 Fix regression in passAsFile
Caused by 8063fc497a. If tmpDir !=
tmpDirInSandbox (typically when there are multiple concurrent builds
with the same name), the *Path attribute would not point to an
existing file. This caused Nixpkgs' writeTextFile to write an empty
file. In particular this showed up as hanging VM builds (because it
would run an empty run-nixos-vm script and then wait for it to finish
booting).
2015-12-29 15:28:20 +01:00
Eelco Dolstra
52120123a5 Handle /tmp being a symlink
Hopefully fixes Darwin sandbox regression introduced in
8063fc497a.
2015-12-22 17:16:17 +01:00
Eelco Dolstra
68fd01f42a Don't ignore sodium_init() return value 2015-12-22 17:14:04 +01:00
Eelco Dolstra
f696af0fab Fix bad error message in Darwin chroots 2015-12-22 17:05:29 +01:00
Eelco Dolstra
a9faa7bbce showId: Handle empty attribute names
We should probably disallow these, but until then, we shouldn't barf
with an assertion failure.

Fixes #738.
2015-12-17 15:15:28 +01:00
Eelco Dolstra
8f67325a7c Build sandbox support etc. unconditionally on Linux
Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent
ambiguity.
2015-12-10 11:47:17 +01:00
Bjørn Forsman
65bd82d42a Clarify error message for hash mismatches (again)
This is arguably nitpicky, but I think this new formulation is even
clearer. My thinking is that it's easier to comprehend when the
calculated hash value is displayed close to the output path. (I think it
is somewhat similar to eliminating double negatives in logic
statements.)

The formulation is inspired / copied from the OpenEmbedded build tool,
bitbake.
2015-12-08 19:50:25 +01:00
Ludovic Courtès
d1e3bf01bc daemon: Add 'buildMode' parameter to 'buildPaths' RPC 2015-12-02 18:14:49 +01:00
Eelco Dolstra
8063fc497a Use deterministic $TMPDIR in sandbox
Rather than using $<host-TMPDIR>/nix-build-<drvname>-<number>, the
temporary directory is now always /tmp/nix-build-<drvname>-0. This
improves bitwise-exact reproducibility for builds that store $TMPDIR
in their build output. (Of course, those should still be fixed...)
2015-12-02 15:04:00 +01:00
Danny Wilson
69b9d8fdbd Use DT_UNKNOWN when dirent d_type extension is not supported.
edolstra:
“…since callers of readDirectory have to handle the possibility of
 DT_UNKNOWN anyway, and we don't want to do a stat call for every
 directory entry unless it's really needed.”
2015-11-25 21:57:19 +01:00
Shea Levy
f327970129 Merge branch 'auto-call-functor'
autoCallFunction now auto-calls functors
2015-11-25 11:57:31 -05:00
Shea Levy
9533532ce2 autoCallFunction: Auto-call functors 2015-11-25 11:56:14 -05:00
Eelco Dolstra
0ab4d905e7 Merge branch 'p/sandbox-rename-minimal' of https://github.com/vcunat/nix 2015-11-25 14:53:42 +01:00
Eelco Dolstra
a0f0733413 Fix build failure introduced by #704
Also, make the FreeBSD checks conditional on FreeBSD.
2015-11-25 14:41:19 +01:00
Eelco Dolstra
afef347157 Merge pull request #712 from pSub/print-meta-license
Print license information on '--xml --meta'
2015-11-25 14:00:52 +01:00
Eelco Dolstra
cad40adce5 Merge pull request #704 from ysangkok/freebsd-support
FreeBSD support with knowledge about Linux emulation
2015-11-24 19:24:21 +01:00
Jude Taylor
279fa8f618 reintroduce host deps in tandem with sandbox profiles 2015-11-21 15:57:06 -08:00
Shea Levy
e0bd114e09 Revert "remove sandbox-defaults.sb"
As discussed in NixOS/nixpkgs#11001, we still need some of the old
sandbox mechanism.

This reverts commit d760c2638c.
2015-11-21 16:40:24 -05:00
Pascal Wittmann
4921223160 Print license information on '--xml --meta'
The nixpkgs manual prescribes the use of values from stdenv.lib.licenses
for the meta.license attribute. Those values are attribute sets and
currently skipped when running nix-env with '--xml --meta'. This has the
consequence that also nixpkgs-lint will report missing licenses.

With this commit nix-env with '--xml --meta' will print all attributes
of an attribute set that are of type tString. For example the output for
the package nixpkgs.hello is

    <meta name="license" type="strings">
      <string type="url" value="http://spdx.org/licenses/GPL-3.0+" />
      <string type="shortName" value="gpl3Plus" />
      <string type="fullName" value="GNU General Public License v3.0 or later" />
      <string type="spdxId" value="GPL-3.0+" />
    </meta>

This commit fixes nixpkgs-lint, too.
2015-11-21 11:43:44 +01:00
Jude Taylor
b9b7bb1806 re-fix permissions for GHC 2015-11-19 16:06:21 -08:00
Shea Levy
5deb7fbdfb Merge branch 'sandbox-profiles' of git://github.com/pikajude/nix
Temporarily allow derivations to describe their full sandbox profile.
This will be eventually scaled back to a more secure setup, see the
discussion at #695
2015-11-19 17:44:11 -05:00
Peter Simons
6ad10591ce src/libstore/build.cc: clarify error message for hash mismatches
Nix reports a hash mismatch saying:

  output path ‘foo’ should have sha256 hash ‘abc’, instead has ‘xyz’

That message is slightly ambiguous and some people read that statement
to mean the exact opposite of what it is supposed to mean. After this
patch, the message will be:

  Nix expects output path ‘foo’ to have sha256 hash ‘abc’, instead it has ‘xyz’
2015-11-19 12:42:37 +01:00
janus
8a74a125bc FreeBSD can build Linux 32-bit binaries 2015-11-17 14:16:08 +00:00
Shea Levy
1d3529e93a Default arguments belong at declaration, not definition 2015-11-16 05:55:55 -05:00
Shea Levy
9b4cd20752 Fix copy-paste error 2015-11-16 05:54:34 -05:00
Shea Levy
58d2fac91d AutoDelete: Add default constructor with deletion disabled 2015-11-16 05:53:10 -05:00
Shea Levy
4390142315 Use AutoDelete for sandbox profile file 2015-11-15 06:08:50 -05:00
Jude Taylor
bd09a4c967 simplify build.cc using modern C++ features 2015-11-14 14:11:03 -08:00
Jude Taylor
4876bb012e simplify build permissions 2015-11-14 14:11:03 -08:00
Jude Taylor
d760c2638c remove sandbox-defaults.sb 2015-11-14 14:11:03 -08:00
Jude Taylor
8a7f0dfd68 use per-derivation sandbox profiles 2015-11-14 14:10:43 -08:00
Vladimír Čunát
b39622a487 rename chroot to sandbox (fixes #656, close #682)
- rename options but leav old names as lower-priority aliases,
  also "-dirs" -> "-paths" to get closer to the meaning
- update docs to reflect the new names (old aliases are not documented),
  including a new file with release notes
- tests need an update after corresponding changes to nixpkgs
- __noChroot is left as it is (after discussion on the PR)
2015-11-10 22:32:51 +01:00
Eelco Dolstra
8fdd156a65 Add option to verify build determinism
Passing "--option build-repeat <N>" will cause every build to be
repeated N times. If the build output differs between any round, the
build is rejected, and the output paths are not registered as
valid. This is primarily useful to verify build determinism. (We
already had a --check option to repeat a previously succeeded
build. However, with --check, non-deterministic builds are registered
in the DB. Preventing that is useful for Hydra to ensure that
non-deterministic builds don't end up getting published at all.)
2015-11-09 23:16:24 +01:00
Eelco Dolstra
96c2ebf004 Revert "Allow using /bin and /usr/bin as impure prefixes on non-darwin by default"
This reverts commit 79ca503332. Ouch,
never noticed this. We definitely don't want to allow builds to have
arbitrary access to /bin and /usr/bin, because then they can (for
instance) bring in a bunch of setuid programs. Also, we shouldn't be
encouraging the use of impurities in the default configuration.
2015-11-09 21:28:38 +01:00
Eelco Dolstra
4384bbd2e1 optimizePath(): Detect some .links corruption
If automatic store optimisation is enabled, and a hard-linked file in
the store gets corrupted, then the corresponding .links entry will
also be corrupted. In that case, trying to repair with --repair or
--repair-path won't work, because the new "good" file will be replaced
by a hard link to the corrupted file. We can catch most of these cases
by doing a sanity-check on the file sizes.
2015-11-09 21:28:25 +01:00
Eelco Dolstra
7759a56bed Fix namespace issue 2015-11-09 21:28:25 +01:00
Danny Wilson
cdb346c65e Fix build on Solaris
d_type is not part of the POSIX spec unfortunately.
2015-11-07 04:51:33 +01:00
Shea Levy
5bc12454fd Merge branch 'master' of https://github.com/pikajude/nix
> I made this change for two reasons:

> 1. Darwin's locale data doesn't appear to be open source
> 2. Privileged processes will always use /usr/share/locale regardless of environment variables
2015-11-04 17:27:19 -05:00
Eelco Dolstra
6f1743b1a5 Support SHA-512 hashes
Fixes #679.

Note: on x86_64, SHA-512 is considerably faster than SHA-256 (198 MB/s
versus 131 MB/s).
2015-11-04 16:37:49 +01:00
Eelco Dolstra
a6ca68a70c Require OpenSSL 2015-11-04 16:37:49 +01:00
Jude Taylor
c10ef2d134 fix syntax error 2015-11-03 09:03:32 -08:00
Jude Taylor
3b52e54e31 darwin: allow reading system locale and zoneinfo 2015-11-03 08:43:09 -08:00
Jude Taylor
d6bd51ffbe allow reading ICU data 2015-10-31 02:52:42 -07:00
Jude Taylor
95c66cc950 add special devices to sandbox-defaults 2015-10-30 20:32:11 -07:00
Eelco Dolstra
1f735a3440 <nix/fetchurl.nix>: Support xz-compressed NARs 2015-10-30 12:34:30 +01:00
Eelco Dolstra
dae5dc7ade <nix/fetchurl.nix>: Support downloading and unpacking NARs
This removes the need to have multiple downloads in the stdenv
bootstrap process (like a separate busybox binary for Linux, or
curl/mkdir/sh/bzip2 for Darwin). Now all those files can be combined
into a single NAR.
2015-10-30 11:27:47 +01:00
Eelco Dolstra
5c28943e8f int2String() -> std::to_string() 2015-10-29 13:26:55 +01:00
Eelco Dolstra
45b2f32cc1 Merge pull request #668 from svanderburg/master
Fix compilation error due to missing ENOENT on cygwin
2015-10-26 11:12:01 +01:00
Jude Taylor
9155fbd386 use nixDataDir instead of appending /share to PREFIX 2015-10-21 14:44:44 -07:00
Jude Taylor
af1218a86a revert libutil change 2015-10-21 14:40:35 -07:00
Jude Taylor
992cda1b11 clarifying comment 2015-10-21 14:39:16 -07:00
Jude Taylor
76f3ba42fd move preBuildHook defaulting to globals.cc 2015-10-21 14:39:08 -07:00
Jude Taylor
2dc8e19a17 restore old DEFAULT_ALLOWED_IMPURE_PREFIXES 2015-10-21 14:38:49 -07:00
Jude Taylor
ff6953cb03 Add resolve-system-dependencies.pl 2015-10-21 12:38:52 -07:00
Jude Taylor
15c7a79ed3 remove usr paths from allowed inputs 2015-10-21 12:38:52 -07:00
Jude Taylor
5bf4085000 allow access to SystemVersion for python builders 2015-10-21 12:38:52 -07:00
Jude Taylor
6a8cee19b3 fix line reading in preBuildHook 2015-10-21 12:38:52 -07:00
Jude Taylor
1fc0fe1baf remove sandbox defaults into a new file 2015-10-21 12:38:52 -07:00
Jude Taylor
d4cac051f7 restore allowed impure prefixes 2015-10-21 12:38:52 -07:00
Jude Taylor
4b8c71b4ef remove an unneeded default impure-dep 2015-10-21 12:38:52 -07:00
Jude Taylor
e770f941d6 make sandbox builds more permissive 2015-10-21 12:38:52 -07:00
Jude Taylor
1ea2aa9e99 add a few more permissions 2015-10-21 12:38:52 -07:00
Eelco Dolstra
f5b32ff47d Allow builtin fetchurl regardless of the derivation's system attribute 2015-10-21 15:14:42 +02:00
Eelco Dolstra
01615b5f63 Show progress indicator for builtin fetchurl 2015-10-21 15:14:42 +02:00
Eelco Dolstra
5db358d4d7 Disable TLS verification for builtin fetchurl
This makes it consistent with the Nixpkgs fetchurl and makes it work
in chroots. We don't need verification because the hash of the result
is checked anyway.
2015-10-21 15:14:42 +02:00
Eelco Dolstra
357d31b339 Fix segfault in builtin fetchurl
The stack allocated for the builder was way too small (32 KB). This is
sufficient for normal derivations, because they just do some setup and
then exec() the actual builder. But for the fetchurl builtin
derivation it's not enough. Also, allocating the stack on the caller's
stack was fishy business.
2015-10-21 15:14:42 +02:00
Sander van der Burg
058d1950c0 Fix compilation error due to missing ENOENT on cygwin 2015-10-18 19:04:24 +00:00
Tuomas Tynkkynen
2075ec83e3 Allow building ARMv6 stuff on ARMv7
This allows building a Raspberry Pi image on modern, faster boards.
2015-10-08 13:32:28 +02:00
Eelco Dolstra
51cf4455b2 isFunctor: Simplify 2015-10-08 13:22:11 +02:00
Mathnerd314
8a87521636 forceFunction: allow functors as well 2015-10-08 13:19:44 +02:00
Eelco Dolstra
14f7e1fc42 Revert to CURLOPT_PROGRESSFUNCTION
CURLOPT_XFERINFOFUNCTION isn't widely supported yet.

http://hydra.nixos.org/build/26679495
2015-10-08 13:15:21 +02:00
Eelco Dolstra
21ecd106ed Show progress during downloads 2015-10-07 17:31:50 +02:00
Eelco Dolstra
f08449ccbd nix-prefetch-url: Add --name option
This allows overriding the name component of the resulting Nix store
path, which is necessary if the base name of the URI contains
"illegal" characters.
2015-10-07 14:55:33 +02:00
Eelco Dolstra
e4abf8610b nix-prefetch-url -A: Use "name" attribute from Nix expression
This is in particular useful for fetchFromGitHub et al., ensuring that
the store path produced by nix-prefetch-url corresponds to what those
functions expect.
2015-10-07 14:47:39 +02:00
Eelco Dolstra
b54f447df9 nix-prefetch-url: Support unpacking tarballs
This allows nix-prefetch-url to prefetch the output of fetchzip and
its wrappers (like fetchFromGitHub). For example:

  $ nix-prefetch-url --unpack https://github.com/NixOS/patchelf/archive/0.8.tar.gz

or from a Nix expression:

  $ nix-prefetch-url -A nix-repl.src

In the latter case, --unpack can be omitted because nix-repl.src is a
fetchFromGitHub derivation and thus has "outputHashMode" set to
"recursive".
2015-10-07 14:40:10 +02:00
Manuel Jacob
7888b2ba15 Define SOL_LOCAL if not defined already (e.g. on FreeBSD).
Some evidence that defining it to be 0 is right:
* OS X headers define it to be 0.
* Other code uses 0 instead of SOL_LOCAL to check for peer credentials
  (e.g. FreeBSD's implementation of getpeereid).
2015-10-06 22:28:30 +02:00
Manuel Jacob
12c1776df9 Don't try to link libdl on FreeBSD.
There is no libdl on FreeBSD.  Instead the symbols are included in libc.
2015-10-06 22:28:30 +02:00
Manuel Jacob
cbc177ca01 Use pkg-config-provided LDFLAGS for libsqlite3 and libcurl.
Previously, pkg-config was already queried for libsqlite3's and
libcurl's link flags.  However they were not used, but hardcoded
instead.  This commit replaces the hardcoded LDFLAGS by the ones
provided by pkg-config in a similar pattern as already used for
libsodium.
2015-10-06 22:28:30 +02:00
Eelco Dolstra
1abda8e173 nix-store --serve: Implement log size limit 2015-10-06 17:33:30 +02:00
Eelco Dolstra
06f29fafe8 nix-prefetch-url: Support prefetching from a Nix expression
For example,

  $ nix-prefetch-url -A hello.src

will prefetch the file specified by the fetchurl call in the attribute
‘hello.src’ from the Nix expression in the current directory. This
differs from ‘nix-build -A hello.src’ in that it doesn't verify the
hash.

You can also specify a path to the Nix expression:

  $ nix-prefetch-url ~/Dev/nixpkgs -A hello.src

List elements (typically used in ‘patches’ attributes) also work:

  $ nix-prefetch-url -A portmidi.patches.0
2015-10-01 18:07:56 +02:00
Eelco Dolstra
a3c4eb0964 nix-prefetch-url: $PRINT_PATH -> --print-path 2015-10-01 16:53:07 +02:00
Eelco Dolstra
bec3c31608 nix-prefetch-url: Rewrite in C++ 2015-10-01 16:47:43 +02:00
Eelco Dolstra
4ba6bc184c Shut up clang warnings 2015-09-18 01:22:35 +02:00
Vladimír Čunát
42808fa281 nix-env --upgrade: show "downgrading" when doing so
It was strange to show "upgrading" when the version was getting lower.
This is left on "upgrading" when the versions are the same,
as I can't see any better wording.
2015-09-17 12:09:54 +02:00
Vladimír Čunát
64f9b511be nix-env --upgrade: avoid unexpected downgrades
Until now, if one explicitly installed a low-priority version,
nix-env --upgrade would downgrade it by default and even with --leq.
Let's never accept an upgrade with version not matching the upgradeType.
Additionally, let's never decrease the priority of an installed package;
you can use --install to force that.

Also refactor to use variable bestVersion instead of bestName,
as only version was used from it.
2015-09-17 12:06:26 +02:00
Eelco Dolstra
71a5161365 Implement buildDerivation() via the daemon 2015-09-03 12:56:59 +02:00
Eelco Dolstra
73332fd342 Filter build-chroot-dirs entries that conflict with derivation outputs
Fixes https://github.com/NixOS/nixpkgs/issues/9504.

Note that this means we may have a non-functional /bin/sh in the
chroot while rebuilding Bash or one of its dependencies. Ideally those
packages don't rely on /bin/sh though.
2015-09-02 14:54:12 +02:00
Eelco Dolstra
d711fe8f5e Remove unused variable 2015-09-02 14:46:04 +02:00
Eelco Dolstra
e12cf82782 Prevent .chroot from being GC'ed when using LocalStore::buildDerivation()
Fixes #616.
2015-08-24 11:13:31 +02:00
Eelco Dolstra
eadb86f447 nix-collect-garbage: Revive --max-freed
Fixes #609.
2015-08-21 13:57:53 +02:00
Manolis Ragkousis
26221e44eb Remove unneeded HAVE_UNSHARE.
* src/libstore/build.cc (CHROOT_ENABLED): Remove HAVE_UNSHARE.
2015-08-19 14:11:15 +03:00
Kirill Elagin
3b0f60e5c2 baseNameOf: Enhance basename compatibility
* If the path ends with a slash, drop it.
* If the remaining path doesn’t contain slashes, just return it.

Fixes #574.
2015-08-07 03:35:02 +02:00
Eelco Dolstra
34dfbd9394 Don't include <iostream> before config.h
This breaks the build on 32-bit systems.

http://hydra.nixos.org/build/24373658
2015-08-04 11:12:31 +02:00
Eelco Dolstra
30d19a2bdc Handle debug messages from runChild()
Turns out that "nix-build -vvv" with chroots enabled has been broken
for some time, because some debug message got interpreted as an error.
2015-08-03 18:04:32 +02:00
Iwan Aucamp
3db950aab7 Removed unnecessary included 2015-08-03 11:48:34 +02:00
Eelco Dolstra
2bac04c5ff Fix stack consumption 2015-07-31 20:28:25 +02:00
Iwan Aucamp
75837651f1 Output line number on infinite recursion 2015-07-31 20:26:44 +02:00
Eelco Dolstra
76cc8e97a2 Add sort primop 2015-07-28 18:39:39 +02:00
Eelco Dolstra
50807f3dd5 Add primop genList
This can be used to implement functions like ‘imap’ (or for that
matter, ‘map’) without the quadratic complexity incurred by calling
‘++’ repeatedly.
2015-07-28 17:28:35 +02:00
Eelco Dolstra
f3dda728a4 Remove unnecessary parentheses 2015-07-26 12:07:14 +02:00
Eelco Dolstra
d6d5885c15 Add replaceStrings primop
This is a generalisation of replaceChars in Nixpkgs.
2015-07-24 15:32:24 +02:00
Eelco Dolstra
2e8fd4c5cd Add concatStringsSep as a primop
This fixes the quadratic behaviour of concatStrings/concatStringsSep
in Nixpkgs.
2015-07-24 02:38:09 +02:00
Eelco Dolstra
cb4320c1a0 Cleanup 2015-07-23 23:14:07 +02:00
Eelco Dolstra
c8bb2371eb Optimize empty sets
This reduces the number of Bindings allocations by about 10%.
2015-07-23 23:11:08 +02:00
Eelco Dolstra
16c9935fa9 Fix fetchurl of executable file
Pointed out by @cstrahan, thanks!
2015-07-23 22:25:04 +02:00
Eelco Dolstra
19eddecc0f Merge branch 'attr-set-hh' of https://github.com/nbp/nix
Conflicts:
	src/libexpr/eval.cc
2015-07-23 22:16:01 +02:00
Eelco Dolstra
b83801f8b3 Optimize small lists
The value pointers of lists with 1 or 2 elements are now stored in the
list value itself. In particular, this makes the "concatMap (x: if
cond then [(f x)] else [])" idiom cheaper.
2015-07-23 22:05:09 +02:00
Eelco Dolstra
14be783676 Add primops all and any
These are used thousands of times during NixOS evaluation, so it's
useful to speed them up.
2015-07-23 19:23:11 +02:00
Shea Levy
39e27a04b8 Importing derivations: Add name attribute to make a valid drv 2015-07-23 17:04:07 +02:00
Shea Levy
1ed55234d9 Allow derivations-as-srcs in the context of builtins.toFile files 2015-07-23 17:03:47 +02:00
Eelco Dolstra
61af14a921 Add foldl' primop 2015-07-23 17:03:02 +02:00
Eelco Dolstra
887bb5fa5a --version: Print some config info
Such as whether Nix is built with signed binary cache support, and the
location of the configuration file.
2015-07-23 14:38:00 +02:00
Eelco Dolstra
1993b10d11 Fix Darwin build
Turns out getgrouplist() is not POSIX.

http://hydra.nixos.org/build/23881243
2015-07-21 14:45:24 +02:00
Eelco Dolstra
0a2bee307b Make <nix/fetchurl.nix> a builtin builder
This ensures that 1) the derivation doesn't change when Nix changes;
2) the derivation closure doesn't contain Nix and its dependencies; 3)
we don't have to rely on ugly chroot hacks.
2015-07-20 04:38:46 +02:00
Eelco Dolstra
eda2f36c2a Provide more detailed info about build status to hydra-queue-runner
In particular, hydra-queue-runner can now distinguish between remote
build / substitution / already-valid. For instance, if a path already
existed on the remote side, we don't want to store a log file.
2015-07-20 03:20:03 +02:00
Eelco Dolstra
ccf31dbc25 nix-copy-closure: Add -v flag
And make exportPath() less spammy by default.
2015-07-20 01:52:07 +02:00
Eelco Dolstra
db55940d9e Support systemd log severity prefixes
This is mostly useful for hydra-queue-runner.
2015-07-20 01:39:48 +02:00
Eelco Dolstra
b3491c781c More cleanup 2015-07-20 01:16:16 +02:00
Eelco Dolstra
6bd2c7bb38 OCD: foreach -> C++11 ranged for 2015-07-17 20:13:56 +02:00
Eelco Dolstra
1511aa9f48 Allow remote builds without sending the derivation closure
Previously, to build a derivation remotely, we had to copy the entire
closure of the .drv file to the remote machine, even though we only
need the top-level derivation. This is very wasteful: the closure can
contain thousands of store paths, and in some Hydra use cases, include
source paths that are very large (e.g. Git/Mercurial checkouts).

So now there is a new operation, StoreAPI::buildDerivation(), that
performs a build from an in-memory representation of a derivation
(BasicDerivation) rather than from a on-disk .drv file. The only files
that need to be in the Nix store are the sources of the derivation
(drv.inputSrcs), and the needed output paths of the dependencies (as
described by drv.inputDrvs). "nix-store --serve" exposes this
interface.

Note that this is a privileged operation, because you can construct a
derivation that builds any store path whatsoever. Fixing this will
require changing the hashing scheme (i.e., the output paths should be
computed from the other fields in BasicDerivation, allowing them to be
verified without access to other derivations). However, this would be
quite nice because it would allow .drv-free building (e.g. "nix-env
-i" wouldn't have to write any .drv files to disk).

Fixes #173.
2015-07-17 17:57:40 +02:00
Eelco Dolstra
f39979c6d3 Make printValue() interruptible
Fixes #572.
2015-07-17 11:33:39 +02:00
Eelco Dolstra
7c9d0a5969 nix-collect-garbage: Handle ENOENT
Don't barf trying to read a link that just got deleted.

Fixes #575.
2015-07-17 11:24:25 +02:00
Nicolas B. Pierron
db21cfa688 Move attribute set data structures into their own header file.
This modification moves Attr and Bindings structures into their own header
file which is dedicated to the attribute set representation. The goal of to
isolate pieces of code which are related to the attribute set
representation. Thus future modifications of the attribute set
representation will only have to modify these files, and not every other
file across the evaluator.
2015-07-14 19:23:17 +02:00
Guillaume Maudoux
467977f203 Fix the parsing of "$"'s in strings. 2015-07-03 14:09:58 +02:00
Guillaume Maudoux
65e4dcd69b Fix the hack that resets the scanner state. 2015-07-03 13:53:36 +02:00
Eelco Dolstra
dd48c06bb6 Typo 2015-07-02 00:30:16 +02:00
Ludovic Courtès
9aed117395 Preserve supplementary groups of build users
The following patch is an attempt to address this bug (see
<http://bugs.gnu.org/18994>) by preserving the supplementary groups of
build users in the build environment.

In practice, I would expect that supplementary groups would contain only
one or two groups: the build users group, and possibly the “kvm” group.

[Changed &at(0) to data() and removed tabs - Eelco]
2015-07-01 14:57:48 +02:00
Eelco Dolstra
ff4de4cb27 GC: Handle ENOSPC creating/moving to the trash directory
Issue #564.
2015-06-30 21:41:26 +02:00
Eelco Dolstra
2bc9c84327 Use posix_fallocate to create /nix/var/nix/db/reserved 2015-06-22 15:54:55 +02:00
Eelco Dolstra
ba63ec6f39 Make /nix/var/nix/db/reserved bigger
Issue #564.
2015-06-22 15:47:40 +02:00
Eelco Dolstra
65f17cd330 Support URLs in $NIX_PATH
This didn't work (despite claims in the manual), because the colon in
"http://" was parsed as a element separator. So handle "://"
specially.
2015-06-17 16:20:11 +02:00
Eelco Dolstra
b65875f859 Export outputPaths function
This is useful for the new hydra-queue-runner.
2015-06-10 16:17:06 +02:00
Eelco Dolstra
7c4501886d Use std::vector::data() 2015-06-09 10:54:46 +02:00
Eelco Dolstra
b64988bb35 Allow substitutes for builds that have preferLocalBuild set
Not substituting builds with "preferLocalBuild = true" was a bad idea,
because it didn't take the cost of dependencies into account. For
instance, if we can't substitute a fetchgit call, then we have to
download/build git and all its dependencies.

Partially reverts 5558652709 and adds a
new derivation attribute "allowSubstitutes" to specify whether a
derivation may be substituted.
2015-06-04 16:30:22 +02:00
Eelco Dolstra
07d7e7df84 Chown files created for passAsFile
Nixpkgs' writeTextAsFile does this:

  mv "$textPath" "$n"

Since $textPath was owned by root, if $textPath is on the same
filesystem as $n, $n will be owned as root. As a result, the build
result was rejected as having suspicious ownership.

http://hydra.nixos.org/build/22836807
2015-06-04 14:07:43 +02:00
Eelco Dolstra
d8ddf994e7 Don't let unprivileged users repair paths 2015-06-02 02:21:54 +02:00
Eelco Dolstra
7106bb0611 Use StoreAPI::verifyStore() 2015-06-02 02:21:15 +02:00
Ludovic Courtès
b755752f76 Add a ‘verifyStore’ RPC
Hello!

The patch below adds a ‘verifyStore’ RPC with the same signature as the
current LocalStore::verifyStore method.

Thanks,
Ludo’.

>From aef46c03ca77eb6344f4892672eb6d9d06432041 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Mon, 1 Jun 2015 23:17:10 +0200
Subject: [PATCH] Add a 'verifyStore' remote procedure call.
2015-06-02 02:14:24 +02:00
Eelco Dolstra
bc51175dc0 Add tarball tests 2015-06-01 16:18:23 +02:00
Eelco Dolstra
da196ec68f Document tarball downloading 2015-06-01 15:14:44 +02:00
Eelco Dolstra
c1323b53e3 Fix fetchurl/fetchTarball 2015-06-01 15:08:09 +02:00
Eelco Dolstra
1c88e100e7 readFile: Check against nul bytes 2015-06-01 15:07:42 +02:00
Eelco Dolstra
920f5fd4dd Fix import-from-derivation in restricted eval mode
This relaxes restricted mode to allow access to anything in the
store. In the future, it would be better to allow access to only paths
that have been constructed in the current evaluation (so a hard-coded
/nix/store/blabla in a Nix expression would still be
rejected). However, note that reading /nix/store itself is still
rejected, so you can't use this so get access to things you don't know
about.
2015-05-22 12:18:23 +02:00
Eelco Dolstra
7a411e01cf Remove dead code 2015-05-22 11:29:22 +02:00
Eelco Dolstra
4ca5a9dcfd nix-collect-garbage: Don't call nix-env
Also, make sure --delete-older-than doesn't delete the current
generation.
2015-05-21 16:28:30 +02:00
Eelco Dolstra
22b1a8d43f Move profiles.{cc,hh} to libstore 2015-05-21 15:42:59 +02:00
Eelco Dolstra
13493ef97c nix-collect-garbage: Call collectGarbage() internally 2015-05-21 15:21:38 +02:00
Eelco Dolstra
4441e4cc13 nix-collect-garbage: Don't barf on unreadable directories
And don't try to delete generations from unwritable directories.
2015-05-21 15:04:05 +02:00
Eelco Dolstra
8d813fe3e0 nix-collect-garbage: Remove redundant call to getFileType 2015-05-21 14:09:34 +02:00
Eelco Dolstra
a1c1bf3a56 Merge branch 'submit/sparse-generation-symlinks' of https://github.com/ctheune/nix 2015-05-21 12:04:54 +02:00
Christian Theune
12a888894b Mis-read Eelko's request to not make this an option: now, let's not make
it an option. :)
2015-05-20 17:29:52 +02:00
Christian Theune
ea39c98d41 Implement alternative to lazy generations:
* only the last generation can be lazy
* depend on the '--lazy-generation' flag to be set
2015-05-19 20:03:36 +02:00
Eelco Dolstra
1529db702b Don't install nix-worker symlink
It has been obsolete since Nix 1.2.

Closes #417.
2015-05-19 16:43:33 +02:00
Christian Theune
3d83188702 Enable lazy/sparse allocation of generation symlinks: avoid creating
new generations if a generation already exists.

Alternatively or additionally I propose a mode where only the *last* generation will be sparse.
2015-05-18 08:38:49 +02:00
Eelco Dolstra
a010c0ae05 Fix "error: deriver of path ‘’ is not known" 2015-05-13 18:03:22 +02:00
Rok Garbas
dad754843a cygwin: looks like stdout/stdin are reserved words 2015-05-13 09:37:56 +02:00
Rok Garbas
000de699e9 cygwin: explicitly include required c headers 2015-05-13 09:37:12 +02:00
Shea Levy
71083f9e5e Don't try to map /bin/sh to a store path on non-Linux 2015-05-12 16:36:15 -04:00
Luca Bruno
8972f7c14a nix-collect-garbage: Do not pass an empty argument. Closes #530 2015-05-06 15:17:23 +00:00
Eelco Dolstra
6519f06f39 nix-env/nix-instantiate/nix-build: Support URIs
For instance, you can install Firefox from a specific Nixpkgs revision
like this:

  $ nix-env -f 63def04891.tar.gz -iA firefox

Or build a package from the latest nixpkgs-unstable channel:

  $ nix-build https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz -A hello
2015-05-06 14:54:31 +02:00
William A. Kennington III
0705d04dfa nix-collect-garbage: Fix deleting old generations
The call to nix-env expects a string which represents how old the
derivations are or just "old" which means any generations other than
the current one in use. Currently nix-collect-garbage passes an empty
string to nix-env when using the -d option. This patch corrects the call
to nix-env such that it follows the old behavior.
2015-05-06 11:02:36 +02:00
Eelco Dolstra
9451ef3731 Allow URLs in the Nix search path
E.g. to install "hello" from the latest Nixpkgs:

  $ nix-build '<nixpkgs>' -A hello -I nixpkgs=https://nixos.org/channels/nixpkgs-unstable/nixexprs.tar.xz

Or to install a specific version of NixOS:

  $ nixos-rebuild switch -I nixpkgs=63def04891.tar.gz
2015-05-05 17:09:42 +02:00
Eelco Dolstra
35d30d67eb Make downloads interruptable 2015-05-05 14:39:48 +02:00
Luca Bruno
ab2b3d6668 nix-collect-garbage: translate to C++ 2015-04-22 15:08:48 +00:00
Shea Levy
4d652875bd Add the pre-build hook.
This hook can be used to set system-specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files through the interface, but it also has full
access to the chroot root.

The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-support system calls. For example,
on OS X there is a set of system-provided frameworks that can reliably
be accessed by any program linked to them, no matter the version the
program is running on. Unfortunately, those frameworks do not
necessarily live in the same locations on each version of OS X, nor do
their dependencies, and thus nix needs to know the specific version of
OS X currently running in order to make those frameworks available. The
pre-build hook is a perfect mechanism for doing just that.
2015-04-18 16:56:02 -04:00
Shea Levy
fd6774e285 Revert "Add the pre-build hook."
Going to reimplement differently.

This reverts commit 1e4a4a2e9f.
2015-04-18 14:59:58 -04:00
Eelco Dolstra
035aeb9547 Fix using restricted mode with chroots 2015-04-16 18:46:17 +02:00
Shea Levy
1e4a4a2e9f Add the pre-build hook.
This hook can be used to set system specific per-derivation build
settings that don't fit into the derivation model and are too complex or
volatile to be hard-coded into nix. Currently, the pre-build hook can
only add chroot dirs/files.

The specific use case for this is systems where the operating system ABI
is more complex than just the kernel-supported system calls. For
example, on OS X there is a set of system-provided frameworks that can
reliably be accessed by any program linked to them, no matter the
version the program is running on. Unfortunately, those frameworks do
not necessarily live in the same locations on each version of OS X, nor
do their dependencies, and thus nix needs to know the specific version
of OS X currently running in order to make those frameworks available.
The pre-build hook is a perfect mechanism for doing just that.
2015-04-12 12:56:38 -04:00
Eelco Dolstra
4ed2187377 Use cached result if there is a network error 2015-04-09 12:49:13 +02:00
Eelco Dolstra
1fc905ad4c Move curl stuff into a separate file 2015-04-09 12:12:50 +02:00
Eelco Dolstra
c1f04fae35 Implement a TTL on cached fetchurl/fetchTarball results
This is because we don't want to do HTTP requests on every evaluation,
even though we can prevent a full redownload via the cached ETag. The
default is one hour.
2015-04-09 11:55:36 +02:00
Eelco Dolstra
60340ce3e2 Implement caching of fetchurl/fetchTarball results
ETags are used to prevent redownloading unchanged files.
2015-04-09 11:42:04 +02:00
Eelco Dolstra
1711679ea5 Revert /nix/store permission back to 01775
This broke NixOS VM tests.

Mostly reverts 27b7b94923,
5ce50cd99e,
afa433e58c.
2015-04-07 13:21:26 +02:00
Eelco Dolstra
afa433e58c Chroot builds: Provide world-readable /nix/store
This was causing NixOS VM tests to fail mysteriously since
5ce50cd99e. Nscd could (sometimes) no
longer read /etc/hosts:

open("/etc/hosts", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)

Probably there was some wacky interaction between the guest kernel and
the 9pfs implementation in QEMU.
2015-04-02 16:59:40 +02:00
Eelco Dolstra
000b5a000f Add fetchTarball builtin
This function downloads and unpacks the given URL at evaluation
time. This is primarily intended to make it easier to deal with Nix
expressions that have external dependencies. For instance, to fetch
Nixpkgs 14.12:

  with import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/nixos-14.12.tar.gz) {};

Or to fetch a specific revision:

  with import (fetchTarball 2766a4b44e.tar.gz) {};

This patch also adds a ‘fetchurl’ builtin that downloads but doesn't
unpack its argument. Not sure if it's useful though.
2015-03-25 17:29:09 +01:00
Eelco Dolstra
7ea6ecf855 addToStore(): Take explicit name argument 2015-03-25 17:06:12 +01:00
Eelco Dolstra
5114a07d95 Improve setting the default chroot dirs 2015-03-24 11:57:46 +01:00
Eelco Dolstra
fd89f97be9 Add the closure of store paths to the chroot
Thus, for example, to get /bin/sh in a chroot, you only need to
specify /bin/sh=${pkgs.bash}/bin/sh in build-chroot-dirs. The
dependencies of sh will be added automatically.
2015-03-24 11:52:34 +01:00
Eelco Dolstra
5ce50cd99e Tighten permissions on chroot directories 2015-03-24 11:35:53 +01:00
Eelco Dolstra
6f0c6e20e0 Don't rely on __noChroot for corepkgs
This doesn't work anymore if the "strict" chroot mode is
enabled. Instead, add Nix's store path as a dependency. This ensures
that its closure is present in the chroot.
2015-03-24 11:15:45 +01:00
Eelco Dolstra
b005e63ccf Disable scanning for interior pointers
This may remove the "Repeated allocation of very large block"
warnings.
2015-03-19 20:10:08 +01:00
Eelco Dolstra
726f7f7fc9 Fix Boehm API violation
We were calling GC_INIT() after doing an allocation (in the baseEnv
construction), which is not allowed.
2015-03-19 20:02:37 +01:00
Eelco Dolstra
da6b704b19 Check return values from malloc/strdup 2015-03-19 14:11:35 +01:00
Eelco Dolstra
fa47279440 Print some Boehm GC stats 2015-03-18 16:24:54 +01:00
Eelco Dolstra
7a84143910 valueSize(): Take into account list/bindings/env size 2015-03-18 14:41:28 +01:00
Daniel Hahler
e659978ced Fix typos: s/the the/the/ 2015-03-06 16:43:22 +01:00
Eelco Dolstra
17c71334e1 forceValueDeep: Add to error prefix 2015-03-06 15:10:12 +01:00
Eelco Dolstra
5badc8f975 Improve error message 2015-03-06 14:24:08 +01:00
Eelco Dolstra
71b0a3a86b Add option to hide display of missing paths 2015-03-04 15:43:04 +01:00
Eelco Dolstra
75ede65e3d Don't use vfork() before clone()
I'm seeing hangs in Glibc's setxid_mark_thread() again. This is
probably because the use of an intermediate process to make clone()
safe from a multi-threaded program (see
524f89f139) is defeated by the use of
vfork(), since the intermediate process will have a copy of Glibc's
threading data structures due to the vfork(). So use a regular fork()
again.
2015-03-04 15:13:10 +01:00
Shea Levy
c2699be93b Merge branch 'allow-system-library' of git://github.com/copumpkin/nix
Make the default impure prefix include all of /System/Library
2015-03-03 15:01:09 -05:00
Dan Peebles
336c4270c6 Make the default impure prefix (not actual allowed impurities!) include all of /System/Library, since we also want PrivateFrameworks from there and (briefly) TextEncodings, and who knows what else. Yay infectious impurities? 2015-03-02 23:01:24 -05:00
Dan Peebles
66d612f1da Allow local networking in the darwin sandbox to appease tests 2015-03-02 22:55:42 -05:00
Eelco Dolstra
885bebf13b More graceful fallback for chroots on Linux < 2.13 2015-02-23 15:54:31 +01:00
Eelco Dolstra
99897f6979 Use chroots for all derivations
If ‘build-use-chroot’ is set to ‘true’, fixed-output derivations are
now also chrooted. However, unlike normal derivations, they don't get
a private network namespace, so they can still access the
network. Also, the use of the ‘__noChroot’ derivation attribute is
no longer allowed.

Setting ‘build-use-chroot’ to ‘relaxed’ gives the old behaviour.
2015-02-23 15:54:31 +01:00
Eelco Dolstra
15d2d3c34e Add restricted evaluation mode
If ‘--option restrict-eval true’ is given, the evaluator will throw an
exception if an attempt is made to access any file outside of the Nix
search path. This is primarily intended for Hydra, where we don't want
people doing ‘builtins.readFile ~/.ssh/id_dsa’ or stuff like that.
2015-02-23 15:54:31 +01:00
Eelco Dolstra
c33244d7c1 Merge branch 'tilde-paths' of https://github.com/shlevy/nix 2015-02-19 14:55:06 +01:00
Shea Levy
c4653afbcd tilde paths: The rest of the string has to start with a slash anyway 2015-02-19 08:52:13 -05:00
Shea Levy
e3e38a048e tilde paths: construct the entire path at parse time 2015-02-19 08:51:21 -05:00
Shea Levy
4b7c9f834c tilde paths: get HOME at parse time 2015-02-19 08:49:10 -05:00
Eelco Dolstra
9bedd9b09b Remove obsolete reference to ~ operator 2015-02-19 14:41:22 +01:00
Shea Levy
4646e94610 ExprConcatStrings: canonicalize concatenated paths 2015-02-19 08:39:25 -05:00
Shea Levy
e0953d53de Allow the leading component of a path to be a ~ 2015-02-19 08:05:16 -05:00
Eelco Dolstra
147deb236e nix-store --generate-binary-cache-key: Write key to disk
This ensures proper permissions for the secret key.
2015-02-18 11:19:44 +01:00
Eelco Dolstra
bd91064150 Use $<attr>Path instead of $<attr> for passAsFile 2015-02-17 16:42:54 +01:00
Eelco Dolstra
a70d275f3d Allow passing attributes via files instead of environment variables
Closes #473.
2015-02-17 14:42:15 +01:00
Harald van Dijk
5451b8db9d Use pivot_root in addition to chroot when possible
chroot only changes the process root directory, not the mount namespace root
directory, and it is well-known that any process with chroot capability can
break out of a chroot "jail". By using pivot_root as well, and unmounting the
original mount namespace root directory, breaking out becomes impossible.

Non-root processes typically have no ability to use chroot() anyway, but they
can gain that capability through the use of clone() or unshare(). For security
reasons, these syscalls are limited in functionality when used inside a normal
chroot environment. Using pivot_root() this way does allow those syscalls to be
put to their full use.
2015-02-16 12:18:19 +01:00
Eelco Dolstra
1c972cba14 Make libsodium an optional dependency 2015-02-10 11:54:06 +01:00
Eelco Dolstra
a596c525ad Add base64 encoder/decoder 2015-02-10 11:33:33 +01:00
Eelco Dolstra
2be7f79fd4 Remove tab 2015-02-05 17:21:30 +01:00
Shea Levy
d66d9e8425 Require linux 3.13 or later for chroot
Fixes #453
2015-02-04 18:15:56 +01:00
Eelco Dolstra
e0def5bc4b Use libsodium instead of OpenSSL for binary cache signing
Sodium's Ed25519 signatures are much shorter than OpenSSL's RSA
signatures. Public keys are also much shorter, so they're now
specified directly in the nix.conf option ‘binary-cache-public-keys’.

The new command ‘nix-store --generate-binary-cache-key’ generates and
prints a public and secret key.
2015-02-04 17:10:31 +01:00
Eelco Dolstra
0d1dafa0c4 Simplify parseHash32 2015-02-03 18:56:47 +01:00
Eelco Dolstra
db2ec59903 Simplify printHash32 2015-02-03 18:35:24 +01:00
Shea Levy
73bf32ce94 Merge remote-tracking branch 'shlevy/baseNameOf-no-copy'
baseNameOf: Don't copy paths to the store first
2015-01-29 03:29:09 -05:00
Daniel Peebles
f46e329a13 Make inputs writeable in the sandbox (builds still can’t actually write due to user permissions) 2015-01-18 23:25:29 -05:00
Eelco Dolstra
c2a8b5c42d Fix assertion failure in nix-env
$ nix-env -f ~/Dev/nixops/ -iA foo
  nix-env: src/libexpr/eval.hh:57: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity' failed.
  Aborted
2015-01-15 12:15:22 +01:00
Shea Levy
79ca503332 Allow using /bin and /usr/bin as impure prefixes on non-darwin by default
These directories are generally world-readable anyway, and give us the two
most common linux impurities (env and sh)
2015-01-13 15:41:46 +01:00
Eelco Dolstra
fcf57aad27 SysError -> Error 2015-01-13 11:17:56 +01:00
Eelco Dolstra
100961e370 Don't resolve symlinks while checking __impureHostDeps
Since these come from untrusted users, we shouldn't do any I/O on them
before we've checked that they're in an allowed prefix.
2015-01-13 11:16:32 +01:00
Daniel Peebles
f1151a3373 Add basic Apple sandbox support 2015-01-12 12:00:01 +01:00
Eelco Dolstra
2a3b1df423 Fix builtins.readDir on XFS
The DT_UNKNOWN fallback code was getting the type of the wrong path,
causing readDir to report "directory" as the type of every file.

Reported by deepfire on IRC.
2015-01-09 14:56:25 +01:00
Eelco Dolstra
57d64d24aa Doh^2 2015-01-08 16:59:22 +01:00
Eelco Dolstra
57b82256b0 Doh 2015-01-08 16:49:31 +01:00
Eelco Dolstra
27b7b94923 Set /nix/store permission to 1737
I.e., not readable to the nixbld group. This improves purity a bit for
non-chroot builds, because it prevents a builder from enumerating
store paths (i.e. it can only access paths it knows about).
2015-01-08 16:39:07 +01:00
Eelco Dolstra
153a943de7 Show position info for failing <...> lookups 2015-01-07 13:43:55 +01:00
Eelco Dolstra
6fec43ccb3 Remove quotes around filenames in position info 2015-01-07 12:08:10 +01:00
Eelco Dolstra
df05f49dcd Fix building on Darwin
Fixes #433.
2015-01-06 10:49:44 +01:00
Eelco Dolstra
8027083c3a Allow $NIX_PAGER to override $PAGER 2015-01-02 15:26:56 +01:00
aszlig
8b88d25cda libutil: Limit readLink() error to only overflows.
Let's not just improve the error message itself, but also the behaviour
to actually work around the ntfs-3g symlink bug. If the readlink() call
returns a smaller size than the stat() call, this really isn't a problem
even if the symlink target really has changed between the calls.

So if stat() reports the size for the absolute path, it's most likely
that the relative path is smaller and thus it should also work for file
system bugs as mentioned in 93002d69fc58c2b71e2dfad202139230c630c53a.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Tested-by: John Ericson <Ericson2314@Yahoo.com>
2015-01-02 12:53:42 +01:00
aszlig
bbd45ac80f libutil: Improve errmsg on readLink size mismatch.
A message like "error: reading symbolic link `...' : Success" really is
quite confusing, so let's not indicate "success" but rather point out
the real issue.

We could also limit the check of this to just check for non-negative
values, but this would introduce a race condition between stat() and
readlink() if the link target changes between those two calls, thus
leading to a buffer overflow vulnerability.

Reported by @Ericson2314 on IRC. Happened due to a possible ntfs-3g bug
where a relative symlink returned the absolute path (st_)size in stat()
while readlink() returned the relative size.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Tested-by: John Ericson <Ericson2314@Yahoo.com>
2015-01-02 12:53:42 +01:00
Shea Levy
3d97b8d1e7 LocalStore initialization: Don't die if build-users-group doesn't exist
See NixOS/nixpkgs@9245516
2014-12-29 14:40:13 +01:00
Eelco Dolstra
bd0f362d2f Revive running builds in a PID namespace 2014-12-23 17:25:06 +01:00
Eelco Dolstra
4e0607369e Pedantry 2014-12-14 01:51:14 +01:00
Eelco Dolstra
8bdff8c100 Merge branch 'cygwin-master' of https://github.com/ternaris/nix 2014-12-14 01:49:14 +01:00
Eelco Dolstra
fa2063ca35 Better error message 2014-12-13 16:54:40 +01:00
Eelco Dolstra
b77037b8fd Silence some warnings on GCC 4.9 2014-12-12 17:14:28 +01:00
Eelco Dolstra
46f3eb6fdd Shut up a Valgrind warning 2014-12-12 15:10:02 +01:00
Eelco Dolstra
f52b6c944e Fix some memory leaks 2014-12-12 15:01:16 +01:00
Eelco Dolstra
28f22b4653 Ensure we're writing to stderr in the builder
http://hydra.nixos.org/build/17862041
2014-12-12 14:35:44 +01:00
Eelco Dolstra
5a2d451648 Don't abort if we get a signal while waiting for the pager 2014-12-12 14:05:23 +01:00
Eelco Dolstra
54616be64f Get rid of unnecessary "interrupted by the user" message with -vvv 2014-12-12 14:01:14 +01:00
Eelco Dolstra
4acca1a5b9 Remove chatty message
This broke building with "-vv", because the builder is not allowed to
write to stderr at this point.
2014-12-12 13:43:59 +01:00
Eelco Dolstra
ad790022fd Doh 2014-12-12 13:41:00 +01:00
Eelco Dolstra
dcdb254281 Remove tabs 2014-12-12 12:39:50 +01:00
Eelco Dolstra
df319047f4 Remove dead code 2014-12-12 12:30:38 +01:00
Eelco Dolstra
8aedaf111e Remove canary stuff 2014-12-12 10:59:50 +01:00
Eelco Dolstra
c56bfbe863 Provide default pagers
Borrowed from systemd.
2014-12-10 18:16:05 +01:00
Eelco Dolstra
851b47bd7d Don't do vfork in conjunction with setuid 2014-12-10 18:01:01 +01:00
Eelco Dolstra
0e8fc118b3 Use vfork 2014-12-10 17:25:39 +01:00
Eelco Dolstra
b5ed5b6e66 Rename function 2014-12-10 17:25:38 +01:00
Eelco Dolstra
e529823635 Don't wait for PID -1
The pid field can be -1 if forking the substituter process failed.
2014-12-10 13:53:04 +01:00
Eelco Dolstra
ad332e1718 Revert "Use posix_spawn to run the pager"
This reverts commit d34d2b2bbf.
2014-12-10 13:48:50 +01:00
Shea Levy
50c3352811 builtins.readFile: realise context associated with the path 2014-12-10 12:26:53 +01:00
Marko Durkovic
629e8da3aa Explicitly include required C headers 2014-12-09 13:00:59 +01:00
Eelco Dolstra
8d169b2b75 Define ‘environ’
http://hydra.nixos.org/build/17690555
2014-12-05 21:05:24 +01:00
Eelco Dolstra
d34d2b2bbf Use posix_spawn to run the pager
In low memory environments, "nix-env -qa" failed because the fork to
run the pager hit the kernel's overcommit limits. Using posix_spawn
gets around this. (Actually, you have to use posix_spawn with the
undocumented POSIX_SPAWN_USEVFORK flag, otherwise it just uses
fork/exec...)
2014-12-05 20:34:41 +01:00
Shea Levy
608110804c Make all ExternalValueBase functions const 2014-12-02 10:27:10 -05:00
Shea Levy
320659b0cd Allow external code using libnixexpr to add types
Code that links to libnixexpr (e.g. plugins loaded with importNative, or
nix-exec) may want to provide custom value types and operations on
values of those types. For example, nix-exec is currently using sets
where a custom IO value type would be more appropriate. This commit
provides a generic hook for such types in the form of tExternal and the
ExternalBase virtual class, which contains all functions necessary for
libnixexpr's type-polymorphic functions (e.g. `showType`) to be
implemented.
2014-12-02 10:27:04 -05:00
Eelco Dolstra
976df480c9 Add a primop for regular expression pattern matching
The function ‘builtins.match’ takes a POSIX extended regular
expression and an arbitrary string. It returns ‘null’ if the string
does not match the regular expression. Otherwise, it returns a list
containing substring matches corresponding to parenthesis groups in
the regex. The regex must match the entire string (i.e. there is an
implied "^<pat>$" around the regex).  For example:

  match "foo" "foobar" => null
  match "foo" "foo" => []
  match "f(o+)(.*)" "foooobar" => ["oooo" "bar"]
  match "(.*/)?([^/]*)" "/dir/file.nix" => ["/dir/" "file.nix"]
  match "(.*/)?([^/]*)" "file.nix" => [null "file.nix"]

The following example finds all regular files with extension .nix or
.patch underneath the current directory:

  let

    findFiles = pat: dir: concatLists (mapAttrsToList (name: type:
      if type == "directory" then
        findFiles pat (dir + "/" + name)
      else if type == "regular" && match pat name != null then
        [(dir + "/" + name)]
      else []) (readDir dir));

  in findFiles ".*\\.(nix|patch)" (toString ./.)
2014-11-25 11:47:06 +01:00
Eelco Dolstra
4e340a983f forceString(): Accept pos argument 2014-11-25 10:23:36 +01:00
Eelco Dolstra
b7b6e3ddec Build derivations in a more predictable order
Derivations are now built in order of derivation name, so a package
named "aardvark" is built before "baboon".

Fixes #399.
2014-11-24 16:50:46 +01:00
Eelco Dolstra
9e3389c337 Don't create unnecessary substitution goals for derivations 2014-11-24 16:50:46 +01:00
Shea Levy
b0c5c2ac34 import derivation: cleanup
Before this there was a bug where a `find` was being called on a
not-yet-sorted set. The code was just a mess before anyway, so I cleaned
it up while fixing it.
2014-11-20 22:48:12 -05:00
Eelco Dolstra
8299aaf079 Disable vacuuming the DB after garbage collection
Especially in WAL mode on a highly loaded machine, this is not a good
idea because it results in a WAL file of approximately the same size
ad the database, which apparently cannot be deleted while anybody is
accessing it.
2014-11-19 18:14:24 +01:00
Eelco Dolstra
a3e5c99d66 nix-daemon: Call exit(), not _exit()
This was preventing destructors from running. In particular, it was
preventing the deletion of the temproot file for each worker
process. It may also have been responsible for the excessive WAL
growth on Hydra (due to the SQLite database not being closed
properly).

Apparently broken by accident in
8e9140cfde.
2014-11-19 17:09:27 +01:00
Eelco Dolstra
1256ab3b44 Clean up temp roots in a more C++ way 2014-11-19 17:07:29 +01:00
Eelco Dolstra
35aad73bb6 Fix message 2014-11-17 01:00:39 +01:00
Shea Levy
2719627bbe realiseContext: Handle all context types
Avoids an assertion
2014-11-15 21:43:51 -05:00
Shea Levy
997defa166 Add functors (callable attribute sets).
With this, attribute sets with a `__functor` attribute can be applied
just like normal functions. This can be used to attach arbitrary
metadata to a function without callers needing to treat it specially.
2014-11-15 16:12:05 -05:00
Eelco Dolstra
8cfe939b0f Don't use ADDR_LIMIT_3GB
This gives 32-bit builds on x86_64-linux more memory.
2014-11-14 14:16:20 +01:00
Eelco Dolstra
bab8d9b52a Make ~DerivationGoal more reliable 2014-11-12 11:35:53 +01:00
Eelco Dolstra
06a86aee15 nix-store --gc: Don't warn about missing manifests directory 2014-11-04 10:41:29 +01:00
Eelco Dolstra
526811c87a nix-daemon: Get peer credentials on Mac OS X
This makes allowed-users and trusted-users work on Mac OS X.
2014-10-31 10:08:59 +01:00
Eelco Dolstra
e389f4ea55 Improve error message if the daemon worker fails to start 2014-10-31 09:36:09 +01:00
Eelco Dolstra
3a9b4a1467 Fix more warnings 2014-10-31 08:49:15 +01:00
Eelco Dolstra
80893a13a7 Shut up a clang warning 2014-10-31 08:49:15 +01:00
Eelco Dolstra
11aef17a77 Remove comments claiming we use a private PID namespace
This is no longer the case since
524f89f139.
2014-10-29 15:49:34 +01:00
Shea Levy
6062b12160 Fix build on gcc < 4.7 2014-10-20 12:15:50 -04:00
Shea Levy
f040159f77 Revert "Drop support for pre-c++11 compilers."
The breakage this fixed can be worked around without removing support.

This reverts commit 84a13dc576.
2014-10-20 11:33:48 -04:00
Eelco Dolstra
ecc2c8f464 Improve printing of ASTs 2014-10-20 09:13:21 +02:00
Shea Levy
84a13dc576 Drop support for pre-c++11 compilers.
In particular, gcc 4.6's std::exception::~exception has an exception
specification in c++0x mode, which requires us to use that deprecated
feature in nix (and led to breakage after some recent changes that were
valid c++11).

nix already uses several c++11 features and gcc 4.7 has been around for
over 2 years.
2014-10-18 22:44:59 -04:00
Shea Levy
0ee1ca628a baseNameOf: Don't copy paths to the store first 2014-10-18 20:28:28 -04:00
Shea Levy
d16e3c7f09 Export realiseContext in libnixexpr
Useful for importNative plugins
2014-10-17 22:15:09 -04:00
Eelco Dolstra
d04e1ff12a nix-store -q: Check for conflicting flags
Fixes #364.
2014-10-14 12:15:39 +02:00
Eelco Dolstra
c6849e2dee Remove redundant space in usage errors 2014-10-14 11:41:04 +02:00
Eelco Dolstra
2a4dbcff74 Improved error message when encountering unsupported file types
Fixes #269.
2014-10-14 10:51:19 +02:00
Eelco Dolstra
6bb4c0b712 mkList: Scrub better
Clearing v.app.right was not enough, because the length field of a
list only takes 32 bits, so the most significant 32 bits of v.app.left
(a.k.a. v.thunk.env) would remain. This could cause Boehm GC to
interpret it as a valid pointer.

This change reduces maximum RSS for evaluating the ‘tested’ job in
nixos/release-small.nix from 1.33 GiB to 0.80 GiB, and runtime by
about 8%.
2014-10-09 13:08:53 +02:00
Eelco Dolstra
986fbd6fab Typo 2014-10-09 11:34:48 +02:00
Eelco Dolstra
b6809608cc Get rid of some unnecessary ExprConcatStrings nodes in dynamic attrs
This gives a ~18% speedup in NixOS evaluation (after converting
most calls to hasAttr/getAttr to dynamic attrs).
2014-10-05 01:04:58 +02:00
Eelco Dolstra
1418806969 Show total allocations 2014-10-05 00:39:28 +02:00
Eelco Dolstra
c3f0a489f9 Add primop ‘catAttrs’ 2014-10-04 18:15:03 +02:00
Eelco Dolstra
d4fcbe1687 Add primop ‘attrValues’ 2014-10-04 16:41:24 +02:00
Eelco Dolstra
58d8a213b0 Tweak 2014-10-04 11:27:23 +02:00
Eelco Dolstra
3f8576a6ab Remove some duplicate code 2014-10-03 22:37:51 +02:00
Shea Levy
c08c802bf3 Add readDir primop 2014-10-03 22:32:11 +02:00
Eelco Dolstra
3b5fa8d50c Don't recompile the same regex over and over 2014-10-03 21:29:40 +02:00
Eelco Dolstra
104e55bb7f nix-env: Add regular expression support in selectors
So you can now do things like:

  $ nix-env -qa '.*zip.*'
  $ nix-env -qa '.*(firefox|chromium).*'
2014-10-03 21:29:40 +02:00
Eelco Dolstra
3800f441e4 createDirs(): Handle ‘path’ being a symlink
In particular, this fixes "nix-build -o /tmp/result" on Mac OS X
(where /tmp is a symlink).
2014-10-03 16:53:28 +02:00
Eelco Dolstra
a17c23426e printValue(): Don't print <CYCLE> for repeated values 2014-10-01 15:54:40 +02:00
Eelco Dolstra
d61853430a Support control characters in JSON output 2014-09-30 00:41:18 +02:00
Eelco Dolstra
f77be20c16 printMissing(): Print derivations in approximate build order 2014-09-26 14:09:20 +02:00
Eelco Dolstra
9b146a52f1 nix-daemon: Close unnecessary fd 2014-09-25 18:45:43 +02:00
Eelco Dolstra
0ed1b924be Bindings: Remove copy constructor 2014-09-24 15:29:05 +02:00
Shell Turner
cf72a61af2 Add --force-name support for --set in nix-env, to support nix-install-package --set 2014-09-23 15:11:24 +02:00
Eelco Dolstra
ebb1dbb3e1 Add missing static 2014-09-23 15:08:27 +02:00
Eelco Dolstra
53b044c2f6 Don't evaluate inside a "throw"
Workaround for
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41174. This caused
hydra-eval-jobs to ignore SIGINT.
2014-09-22 19:18:05 +02:00
Eelco Dolstra
0cd6596b0e Add ‘deepSeq’ primop
Note that unlike ‘lib.deepSeq’ in Nixpkgs, this handles cycles.
2014-09-22 16:05:00 +02:00
Eelco Dolstra
831fc8ea21 Make forceValueDeep work on values with cycles 2014-09-22 16:05:00 +02:00
Eelco Dolstra
1e0a799bef Rename strictForceValue -> forceValueDeep 2014-09-22 16:05:00 +02:00
Eelco Dolstra
022618c794 Handle cycles when printing a value
So this no longer crashes with a stack overflow:

  nix-instantiate -E --eval 'let as = { x = as; }; in as'

Instead it prints:

  { x = { x = <CYCLE>; }; }
2014-09-22 16:05:00 +02:00
Eelco Dolstra
a54c263402 Add ‘seq’ primop 2014-09-22 16:05:00 +02:00
Eelco Dolstra
eff120d1b9 Add a function ‘valueSize’
It returns the size of value, including all other values and
environments reachable from it. It is intended for debugging memory
consumption issues.
2014-09-22 16:05:00 +02:00
Eelco Dolstra
2d6cd8aafd attrNames: Don't allocate duplicates of the symbols 2014-09-19 18:11:46 +02:00
Eelco Dolstra
ea525a261f Fix off-by-one 2014-09-19 18:08:14 +02:00
Eelco Dolstra
93e4f01ee3 Inline Bindings::find() 2014-09-19 16:56:13 +02:00
Eelco Dolstra
5b58991a71 Store Attrs inside Bindings
This prevents a double allocation per attribute set.
2014-09-19 16:49:41 +02:00
Eelco Dolstra
0342eb1705 Remove bogus comment 2014-09-19 15:07:22 +02:00
Eelco Dolstra
d4a71ec3bf Update spec file
http://hydra.nixos.org/build/14344391
2014-09-18 15:42:01 +02:00
Eelco Dolstra
8be9990cdb Install some pkgconfig files 2014-09-18 12:00:40 +02:00
Eelco Dolstra
d98bfcbf81 On Linux, disable address space randomization 2014-09-17 17:21:13 +02:00
Eelco Dolstra
6e5b02bee4 Add some instrumentation for debugging GC leaks 2014-09-17 15:19:07 +02:00
Eelco Dolstra
d37d012774 Settings: Add bool get() 2014-09-17 15:18:13 +02:00
Eelco Dolstra
9d65287b91 Fix dependency ordering 2014-09-04 20:02:08 +02:00
Eelco Dolstra
9472b4157d Fix boost::too_many_args error
Fixes #333.
2014-09-02 22:53:01 +02:00
Ludovic Courtès
8fb8c26b6d Add an 'optimiseStore' remote procedure call. 2014-09-01 23:53:26 +02:00
Eelco Dolstra
b72e93bca8 Add disallowedReferences / disallowedRequisites
For the "stdenv accidentally referring to bootstrap-tools", it seems
easier to specify the path that we don't want to depend on, e.g.

  disallowedRequisites = [ bootstrapTools ];
2014-08-28 18:57:13 +02:00
Gergely Risko
fd61069a42 Introduce allowedRequisites feature 2014-08-28 18:23:55 +02:00
Joel Taylor
b224ac1520 fix disappearing bash arguments 2014-08-21 23:17:50 +02:00
Eelco Dolstra
809ca33806 Use PR_SET_PDEATHSIG to ensure child cleanup 2014-08-21 15:31:43 +02:00
Eelco Dolstra
524f89f139 Use unshare() instead of clone()
It turns out that using clone() to start a child process is unsafe in
a multithreaded program. It can cause the initialisation of a build
child process to hang in setgroups(), as seen several times in the
build farm:

The reason is that Glibc thinks that the other threads of the parent
exist in the child, so in setxid_mark_thread() it tries to get a futex
that has been acquired by another thread just before the clone(). With
fork(), Glibc runs pthread_atfork() handlers that take care of this
(in particular, __reclaim_stacks()). But clone() doesn't do that.

Fortunately, we can use fork()+unshare() instead of clone() to set up
private namespaces.

See also https://www.mail-archive.com/lxc-devel@lists.linuxcontainers.org/msg03434.html.
2014-08-21 14:08:09 +02:00
Eelco Dolstra
fefd3650d4 Fix a segfault in ‘nix-env -qa’
This was triggered by 47e185847e, which
turned globals.state into a pointer.
2014-08-21 00:05:17 +02:00
Eelco Dolstra
65243ee833 Flush std::cout before closing stdout 2014-08-20 21:44:55 +02:00
Eelco Dolstra
43579c9799 Use pager for more commands 2014-08-20 21:44:55 +02:00
Eelco Dolstra
9481f77693 Provide reasonable default flags for $LESS
Borrowed from systemd.
2014-08-20 21:22:41 +02:00
Eelco Dolstra
11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra
373fad75e1 Add some color 2014-08-20 16:50:17 +02:00
Eelco Dolstra
392430b2c4 nix-store -l: Automatically pipe output into $PAGER 2014-08-20 15:12:58 +02:00
Eelco Dolstra
954188af27 Filter Nix-specific ANSI escape sequences from stderr
The Nixpkgs stdenv prints some custom escape sequences to denote
nesting and stuff like that. Most terminals (e.g. xterm, konsole)
ignore them, but some do not (e.g. xfce4-terminal). So for the benefit
of the latter, filter them out.
2014-08-20 14:30:01 +02:00
Eelco Dolstra
029424d17d Make hook shutdown more reliable 2014-08-19 17:47:30 +02:00
Eelco Dolstra
02843ba4cb Fix --attr parsing 2014-08-18 15:48:23 +02:00
Eelco Dolstra
632f989b51 Doh 2014-08-18 11:35:50 +02:00
Eelco Dolstra
c160ead82f Reduce verbosity 2014-08-17 19:11:50 +02:00
Eelco Dolstra
42e9ad8fd1 Propagate remote timeouts properly 2014-08-17 19:09:03 +02:00
Eelco Dolstra
ad716378dc Fix download-via-ssh
http://hydra.nixos.org/build/13286020
2014-08-17 14:28:17 +02:00