Commit graph

2867 commits

Author SHA1 Message Date
Eelco Dolstra
aa1ea0d1e4 Merge pull request #1030 from pikajude/rsd-cc
Implement resolve-system-dependencies in C++
2016-08-31 15:19:10 +02:00
Eelco Dolstra
d74c8a3f4e Fix 32-bit build 2016-08-30 17:38:09 +02:00
Eelco Dolstra
6631a6e1a1 Increase the sleep time between download retries 2016-08-30 15:48:24 +02:00
Eelco Dolstra
97b1af1cbe Fix --fallback
This fixes an assertion failure in "assert(goal);" in
Worker::waitForInput() after a substitution goal is cancelled by the
termination of another goal. The problem was the line

  //worker.childTerminated(shared_from_this()); // FIXME

in the SubstitutionGoal destructor. This was disabled because
shared_from_this() obviously doesn't work from a destructor. So we now
use a real pointer for object identity.
2016-08-30 15:45:39 +02:00
Eelco Dolstra
67364a5795 Fix Fedora build 2016-08-30 13:56:22 +02:00
Eelco Dolstra
92d917b71a Fix GC build 2016-08-30 13:12:12 +02:00
Eelco Dolstra
c42d1acfeb forceBool(): Show position info 2016-08-29 19:37:19 +02:00
Eelco Dolstra
26d92017d3 Add builtin function "partition"
The implementation of "partition" in Nixpkgs is O(n^2) (because of the
use of ++), and for some reason was causing stack overflows in
multi-threaded evaluation (not sure why).

This reduces "nix-env -qa --drv-path" runtime by 0.197s and memory
usage by 298 MiB (in non-Boehm mode).
2016-08-29 19:36:54 +02:00
Eelco Dolstra
c0a7b84748 nix path-info: Add --json flag
Also, factor out JSON generation from value-to-json.{cc,hh}, and
support producing indented JSON.
2016-08-29 17:29:24 +02:00
Shea Levy
9fa21765e7 callFunction: Copy functors to the heap
Normally it's impossible to take a reference to the function passed to
callFunction, so some callers (e.g. ExprApp::eval) allocate that value
on the stack. For functors, a reference to the functor itself may be
kept, so we need to have it on the heap.

Fixes #1045
2016-08-29 07:36:28 -04:00
Eelco Dolstra
a9d4f2b303 Fix queryPathFromHashPart()
The inner lambda was returning a SQLite-internal char * rather than a
std::string, leading to Hydra errors liks

  Caught exception in Hydra::Controller::Root->narinfo "path ‘ø˜£â€™ is not in the Nix store at /nix/store/6mvvyb8fgwj23miyal5mdr8ik4ixk15w-hydra-0.1.1234.abcdef/libexec/hydra/lib/Hydra/Controller/Root.pm line 352."
2016-08-24 14:53:33 +02:00
Eelco Dolstra
d74236d1f2 nix build: Use Nix search path
That is, unless --file is specified, the Nix search path is
synthesized into an attribute set. Thus you can say

  $ nix build nixpkgs.hello

assuming $NIX_PATH contains an entry of the form "nixpkgs=...". This
is more verbose than

  $ nix build hello

but is less ambiguous.
2016-08-23 17:11:19 +02:00
Jude Taylor
50c3b5df32 iterate through fat_headers correctly 2016-08-17 10:24:11 -07:00
Eelco Dolstra
22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Jude Taylor
ca0bce2851 remove double casting 2016-08-16 14:09:57 -07:00
Jude Taylor
cfb77d6e5b account for unknown magic numbers 2016-08-16 12:52:49 -07:00
Jude Taylor
90516c5a7b switch to mmap strategy 2016-08-16 12:45:55 -07:00
Eelco Dolstra
ac841a4679 Merge pull request #1031 from abbradar/replacestrings-context
Allow contexted strings in replaceStrings
2016-08-16 12:47:20 +02:00
Nikolay Amiantov
f1b8dd43be Allow contexted strings in replaceStrings 2016-08-16 13:20:56 +03:00
Jude Taylor
7b006122ae ensure presence of directories 2016-08-15 10:46:21 -07:00
Jude Taylor
adf0216d98 code review comments 2016-08-15 10:43:14 -07:00
Shea Levy
d52d391164 builtins.fetch{url,tarball}: Allow name attribute 2016-08-15 07:37:11 -04:00
Jude Taylor
60f4b25d7d make inclusion conditional 2016-08-14 19:10:38 -07:00
Jude Taylor
f37b6fd07e add a missing load command 2016-08-14 18:54:40 -07:00
Jude Taylor
596e4a5693 remove old traces of resolve-system-dependencies 2016-08-13 15:27:49 -07:00
Jude Taylor
2df9a972fc resolve-system-dependencies: implement in C++ 2016-08-13 11:36:22 -07:00
Shea Levy
59124228b3 nix-channel: implement in c++ 2016-08-11 11:34:43 -04:00
Vladimír Čunát
1bffd83e1a nix-build: extend the meaning of $IN_NIX_SHELL
An equivalent was originally filed against the perl version:
https://github.com/NixOS/nix/pull/933
2016-08-11 17:14:09 +02:00
Eelco Dolstra
b0ebad88cd Merge branch 'buildenv-c++' of https://github.com/shlevy/nix 2016-08-11 17:11:17 +02:00
Shea Levy
91978e3b9a buildenv: perl -> c++ 2016-08-11 07:58:33 -04:00
Eelco Dolstra
66adbdfd97 HttpBinaryCacheStore: Retry on transient HTTP errors
This makes us more robust against 500 errors from CloudFront or S3
(assuming the 500 error isn't cached by CloudFront...).
2016-08-10 18:08:23 +02:00
Eelco Dolstra
9204ea7294 Show HTTP error codes 2016-08-10 18:05:35 +02:00
Eelco Dolstra
f294623d1d SQLite:: Add some convenience 2016-08-10 18:05:35 +02:00
Eelco Dolstra
6cb4bdf152 FSAccessor: Throw InvalidPath 2016-08-10 18:05:35 +02:00
Eelco Dolstra
f43823f676 NarAccessor: Fix handling of non-executable files 2016-08-10 18:05:35 +02:00
Eelco Dolstra
d961c29c9c Mark content-addressed paths in the Nix database and in .narinfo
This allows such paths to be imported without signatures.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
36a51ecab3 LocalFSStore::narFromPath(): Handle the diverted store case 2016-08-10 18:05:35 +02:00
Eelco Dolstra
3eb6217508 Add a "root" parameter to local stores
This makes it easier to create a diverted store, i.e.

  NIX_REMOTE="local?root=/tmp/root"

instead of

  NIX_REMOTE="local?real=/tmp/root/nix/store&state=/tmp/root/nix/var/nix" NIX_LOG_DIR=/tmp/root/nix/var/log
2016-08-10 18:05:35 +02:00
Eelco Dolstra
2fad86f361 Remove $NIX_DB_DIR
This variable has no reason to exist, given $NIX_STATE_DIR.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
be64fbb501 RemoteStore / nix-daemon: Drop support for Nix < 1.0 2016-08-10 18:05:35 +02:00
Shea Levy
80ebc553ec nix-build: Port to c++
This was a dumb line-for-line rewrite, because nix build/nix run/etc.
will replace it.
2016-08-09 07:42:20 -04:00
Eelco Dolstra
ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra
06bbfb6004 builtins.{fetchurl,fetchTarball}: Support a sha256 attribute
Also, allow builtins.{fetchurl,fetchTarball} in restricted mode if a
hash is specified.
2016-07-26 21:16:52 +02:00
Dmitry Kalinkin
f91748ba73 override rx directory permissions in deletePath()
This fixes instantiation of pythonPackages.pytest that produces a
directory with less permissions during one of it's tests that leads to
a nix error like:

error: opening directory ‘/tmp/nix-build-python2.7-pytest-2.9.2.drv-0/pytest-of-user/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
2016-07-25 18:11:46 -04:00
Shea Levy
ee3032e4de Merge branch 'find-runtime-roots-c++' 2016-07-24 07:33:49 -04:00
Shea Levy
18b0808475 Respect --keep-going when a substituter fails.
Fixes #977
2016-07-23 09:16:08 -04:00
Eelco Dolstra
d3cd0f5856 Fix assertion failure 2016-07-21 18:39:32 +02:00
Eelco Dolstra
e19c90fc6b Fix random failures caused by CurlDownloader modifying a shared string 2016-07-21 18:35:31 +02:00
Eelco Dolstra
ea8e8df6c7 NarInfo::NarInfo(): Ensure that we get a NAR size/hash 2016-07-21 18:35:12 +02:00
Eelco Dolstra
e682a8e138 Fix assertion failure in ThreadPool::enqueue() 2016-07-21 18:14:16 +02:00
Eelco Dolstra
d57981bac4 Store::queryMissing(): Use a thread pool
For one particular NixOS configuration, this cut the runtime of
"nix-store -r --dry-run" from 6m51s to 3.4s. It also fixes a bug in
the size calculation that was causing certain paths to be counted
twice, e.g. before:

  these paths will be fetched (1249.98 MiB download, 2995.74 MiB unpacked):

and after:

  these paths will be fetched (1219.56 MiB download, 2862.17 MiB unpacked):
2016-07-21 17:50:39 +02:00
Eelco Dolstra
a728780fbd Store::computeFSClosure(): Use thread pool
This speeds up queries against the binary cache.
2016-07-21 14:13:35 +02:00
Shea Levy
3c46fe62b8 find-runtime-roots: fold into gc.cc 2016-07-21 07:04:41 -04:00
Shea Levy
06068b353d FdSink: Restore move constructor/assignment 2016-07-13 06:27:41 -04:00
Shea Levy
b33e85229d Make Buffered{Source,Sink} move-safe 2016-07-13 06:03:37 -04:00
Shea Levy
cb5e7254b6 Modernize AutoCloseFD 2016-07-11 15:44:44 -04:00
Eelco Dolstra
74dd603495 Re-implement negative binary cache lookup caching 2016-06-20 17:39:05 +02:00
Eelco Dolstra
c68e5913c7 Run builds in a user namespace
This way, all builds appear to have a uid/gid of 0 inside the
chroot. In the future, this may allow using programs like
systemd-nspawn inside builds, but that will require assigning a larger
UID/GID map to the build.

Issue #625.
2016-06-09 18:30:39 +02:00
Eelco Dolstra
202683a4fc Use O_CLOEXEC in most places 2016-06-09 16:37:08 +02:00
Eelco Dolstra
9bdd949cfd Fix "creating statement: table ValidPaths has no column named ultimate" 2016-06-03 16:06:42 +02:00
Eelco Dolstra
5e51ffb1c2 Support sandbox builds by non-root users
This allows an unprivileged user to perform builds on a diverted store
(i.e. where the physical store location differs from the logical
location).

Example:

  $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \
    'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"'

will do a build in the Nix store physically in /tmp/store but
logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:59:28 +02:00
Eelco Dolstra
2f8b0e557b Consistent quotes 2016-06-03 14:03:09 +02:00
Eelco Dolstra
080d607975 Don't call shared_from_this() from a destructor 2016-06-03 14:02:44 +02:00
Eelco Dolstra
30c8c3ba2d Fix some more references to storeDir 2016-06-03 14:02:44 +02:00
Eelco Dolstra
a8dfdc52b8 Merge pull request #925 from abbradar/master
Show both cycle ends
2016-06-03 13:58:51 +02:00
Nikolay Amiantov
c87a56f4d0 Show both cycle ends 2016-06-03 14:25:37 +03:00
Eelco Dolstra
75361b6dce Fix narrowing conversion from int64_t to bool
http://hydra.nixos.org/build/36613774
2016-06-03 13:20:46 +02:00
Eelco Dolstra
f8a8b4d8f8 nix run: Set a reasonable uid/gid 2016-06-02 19:04:09 +02:00
Eelco Dolstra
d64e0c1b64 Make derivationFromPath work on diverted stores 2016-06-02 18:43:36 +02:00
Eelco Dolstra
eda2aaae92 nix run: Handle the case where the /nix/store mount point doesn't exist 2016-06-02 18:24:51 +02:00
Eelco Dolstra
a424ab0444 createTempDir(): Don't do a chown on Linux
It's not needed and can cause problems in a user namespace.
2016-06-02 18:17:30 +02:00
Eelco Dolstra
a24f2c9b84 nix run: Mount the Nix store in a private namespace
This is a convenience command to allow users who are not privileged to
create /nix/store to use Nix with regular binary caches. For example,

  $ NIX_REMOTE="local?state=$HOME/nix/var&real=/$HOME/nix/store" nix run firefox bashInteractive

will download Firefox and bash from cache.nixos.org, then start a
shell in which $HOME/nix/store is mounted on /nix/store.
2016-06-02 16:51:43 +02:00
Eelco Dolstra
6f2d51287c Add basic "nix run" command 2016-06-02 16:29:49 +02:00
Eelco Dolstra
4494000e04 LocalStore: Allow the physical and logical store directories to differ
This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):

  $ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'

However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:

  $ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
2016-06-02 16:02:48 +02:00
Eelco Dolstra
064816ab98 Respect build-use-substitutes 2016-06-02 16:02:48 +02:00
Eelco Dolstra
812c0dfbe2 Allow setting the state directory as a store parameter
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Eelco Dolstra
f2682e6e18 <nix/fetchurl.nix>: Use tarballs.nixos.org
This ensures that the disappearance of upstream bootstrap tarballs
(e.g. https://github.com/NixOS/nixpkgs-channels/pull/1) doesn't break
stdenv rebuilds.
2016-06-01 17:11:51 +02:00
Eelco Dolstra
dead8189ac Typo 2016-06-01 16:56:22 +02:00
Eelco Dolstra
e24e2caaaf Skip substituters with an incompatible store directory 2016-06-01 16:40:49 +02:00
Eelco Dolstra
cf198952d0 HttpBinaryCacheStore: Fix caching of WantMassQuery
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
2016-06-01 16:24:17 +02:00
Eelco Dolstra
7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra
a9fa5e050a Shut up some clang warnings 2016-05-31 13:31:04 +02:00
Eelco Dolstra
10f3a2e5f2 Fix clang build failure
Apparently opinion is divided on whether [[noreturn]] is allowed on a
lambda: http://stackoverflow.com/questions/26888805/how-to-declare-a-lambdas-operator-as-noreturn

http://hydra.nixos.org/build/36462100
2016-05-31 13:23:54 +02:00
Eelco Dolstra
c2d27d30cf nix-copy-closure / build-remote.pl: Disable signature checking
This restores the Nix 1.11 behaviour.
2016-05-31 11:48:05 +02:00
Dmitry Kalinkin
78b00bbd8a use $(LIBLZMA_LIBS) instead of -llzma
This is needed in case of non-standard lzma installation path that will
be specified in pkgconfig manifest as extra -L option for LDFLAGS.
2016-05-31 03:20:11 -04:00
Eelco Dolstra
cd35f0280c Fix build 2016-05-30 16:12:50 +02:00
Eelco Dolstra
57d33013ce Check signatures before downloading the substitute 2016-05-30 15:18:12 +02:00
Eelco Dolstra
bac123ddd9 Test the NAR info cache 2016-05-30 15:18:12 +02:00
Eelco Dolstra
3593c8285d Re-implement binary cache signature checking
This is now done in LocalStore::addToStore(), rather than in the
binary cache substituter (which no longer exists).
2016-05-30 15:18:12 +02:00
Eelco Dolstra
12ddbad458 LocalStore::addToStore: Verify hash of the imported path 2016-05-30 15:18:12 +02:00
Eelco Dolstra
e222484401 Re-implement the WantMassQuery property of binary caches 2016-05-30 15:18:12 +02:00
Eelco Dolstra
b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra
3be2e71ab3 BinaryCacheStore: Remove buildPaths() / ensurePath() 2016-05-30 15:18:12 +02:00
Eelco Dolstra
75d2492f20 Make the aws-cpp-sdk dependency optional 2016-05-04 17:16:48 +02:00
Eelco Dolstra
6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra
0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra
c6a21aed07 More GC fixes 2016-05-04 16:16:53 +02:00
Eelco Dolstra
538a64e8c3 Add a Store::addToStore() variant that accepts a NAR
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra
b6c768fb6a nix-daemon: Fix queryPathInfo() backward compatibility 2016-05-04 12:53:24 +02:00
Eelco Dolstra
33de2bc080 Fix segfault in xz compression/decompression 2016-05-04 11:50:12 +02:00
Eelco Dolstra
f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra
dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra
d8bf0d4859 Support Git repos in the Nix path
E.g.

  $ nix-build -I nixpkgs=git://github.com/NixOS/nixpkgs '<nixpkgs>' -A hello

This is not extremely useful yet because you can't specify a
branch/revision.
2016-04-29 21:07:47 +02:00
Eelco Dolstra
38539b943a Add fetchgit builtin
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)

Example:

  (import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello

or

  (import (builtins.fetchgit {
    url = git://github.com/NixOS/nixpkgs-channels;
    rev = "nixos-16.03";
  }) {}).hello

Note that the result does not contain a .git directory.
2016-04-29 20:47:36 +02:00
Eelco Dolstra
83258225e6 nix-store -l: Simplify implementation 2016-04-29 17:47:03 +02:00
Eelco Dolstra
4dde0b0562 BinaryCacheStore: Support bzip2 compression 2016-04-29 17:43:37 +02:00
Eelco Dolstra
6963de2091 nix verify --all: Support local binary caches 2016-04-29 17:34:31 +02:00
Eelco Dolstra
0dd988d2e3 Better error message 2016-04-29 17:33:22 +02:00
Eelco Dolstra
5acb691402 BinaryCacheStore: Support "none" compression method 2016-04-29 17:02:57 +02:00
Eelco Dolstra
8e065c6b3e BinaryCacheStore: Make the signing key a parameter 2016-04-29 16:47:20 +02:00
Eelco Dolstra
f6aee2f477 BinaryCacheStore::queryPathInfo: Don't check signatures
Other stores don't do this either. It's up to the caller to check
signatures.
2016-04-29 16:28:57 +02:00
Eelco Dolstra
95d20dfde9 Allow parameters in store URIs
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-29 16:26:16 +02:00
Eelco Dolstra
aa3bc3d5dc Eliminate the substituter mechanism
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Eelco Dolstra
21e9d183cc Really handle carriage return 2016-04-28 14:27:00 +02:00
Eelco Dolstra
ce5776758d Fix error message 2016-04-28 14:12:10 +02:00
Eelco Dolstra
cc804d0dc6 Handle carriage return 2016-04-25 19:18:45 +02:00
Eelco Dolstra
ca9de88a51 nix: Disable verbose builds 2016-04-25 19:18:45 +02:00
Eelco Dolstra
5761827d5b Show the log tail when a build fails
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
6e1b099279 Remove --print-build-trace
This was added to support Hydra, but Hydra no longer uses it.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
9eba2c3945 Fix "path is not in the Nix store" during GC 2016-04-25 19:18:45 +02:00
Eelco Dolstra
697e0b3a6f Remove nix-log2xml 2016-04-25 19:18:45 +02:00
Eelco Dolstra
41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
c879a20850 Factor out parallel processing of work items that have dependencies 2016-04-22 20:50:06 +02:00
Eelco Dolstra
91539d305f nix copy: Parallelise 2016-04-22 18:19:48 +02:00
Eelco Dolstra
b2ce6fde5a ThreadPool: Start doing work as soon as work items are enqueued 2016-04-22 18:19:17 +02:00
Eelco Dolstra
58c84cda3b Make compression interruptible 2016-04-22 18:15:02 +02:00
Eelco Dolstra
456179018a Fold "nix query-path-sigs" into "nix path-info" 2016-04-22 14:39:37 +02:00
Eelco Dolstra
95abf9c402 Add "nix copy" command
This replaces nix-push. For example,

  $ nix copy --to file:///tmp/cache -r $(type -p firefox)

copies the closure of firefox to the specified binary cache. And

  $ nix copy --from file:///tmp/cache --to s3://my-cache /nix/store/abcd...

copies between two binary caches.

It will also replace nix-copy-closure, once we have an SSHStore class,
e.g.

  $ nix copy --from ssh://alice@machine /nix/store/abcd...
2016-04-22 14:33:01 +02:00
Eelco Dolstra
0207272b28 BinaryCacheStore: When adding a path, ensure the references are valid
This prevents copying a partial closure to a binary cache.
2016-04-22 12:15:06 +02:00
Eelco Dolstra
21ef1670b3 Fix test failures 2016-04-21 18:21:25 +02:00
Eelco Dolstra
7d14f5c331 Implement S3BinaryCacheStore::queryAllValidPaths()
This allows commands like "nix verify --all" or "nix path-info --all"
to work on S3 caches.

Unfortunately, this requires some ugly hackery: when querying the
contents of the bucket, we don't want to have to read every .narinfo
file. But the S3 bucket keys only include the hash part of each store
path, not the name part. So as a special exception
queryAllValidPaths() can now return store paths *without* the name
part, and queryPathInfo() accepts such store paths (returning a
ValidPathInfo object containing the full name).
2016-04-21 17:53:47 +02:00
Eelco Dolstra
d155d80155 Move S3BinaryCacheStore from Hydra
This allows running arbitrary Nix commands against an S3 binary cache.

To do: make this a compile time option to prevent a dependency on
aws-sdk-cpp.
2016-04-21 16:08:51 +02:00
Eelco Dolstra
1a71495273 nix path-info: Add
Forgot to commit this earlier...
2016-04-21 15:00:45 +02:00
Eelco Dolstra
69e3ffb076 nix --help: Show usage examples 2016-04-21 14:58:32 +02:00
Eelco Dolstra
1b0088ebb2 nix --help: Show short flags 2016-04-21 14:34:46 +02:00
Eelco Dolstra
ddea253ff8 RemoteStore: Propagate InvalidPath exceptions from the daemon 2016-04-20 15:28:07 +02:00
Eelco Dolstra
c0c4ddcd9c BinaryCacheStore: Insert new paths into the disk cache 2016-04-20 15:27:48 +02:00
Eelco Dolstra
451ebf24ce Cache path info lookups in SQLite
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
2016-04-20 14:12:38 +02:00
Eelco Dolstra
e0204f8d46 Move path info caching from BinaryCacheStore to Store
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
2016-04-19 18:52:53 +02:00
Dan Peebles
608b0265e1 Print out all bad references/requisites at once
Also updates tests to check for new information. Fixes #799
2016-04-16 19:58:26 -04:00
Eelco Dolstra
0423787086 Make the .narinfo cache bigger 2016-04-15 15:39:48 +02:00
Eelco Dolstra
a7d8eaba54 BinaryCacheStore: Do negative caching of .narinfo lookups 2016-04-15 15:39:48 +02:00
Eelco Dolstra
d1b0909894 BinaryCacheStore::readFile(): Return a shared_ptr to a string
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
2016-04-15 15:39:48 +02:00
Eelco Dolstra
99851c6f06 Unify "nix verify-paths" and "nix verify-store"
"verify-store" is now simply an "--all" flag to "nix verify". This
flag can be used for any other store path command as well (e.g. "nix
path-info", "nix copy-sigs", ...).
2016-04-15 15:39:48 +02:00
Eelco Dolstra
31a1a8ed3c Merge pull request #815 from vcunat/p/outputsToInstall
nix-env: respect meta.outputsToInstall
2016-04-14 20:55:41 +02:00
Eelco Dolstra
5169a6da98 Make $NIX_PATH parsing more robust 2016-04-14 17:29:08 +02:00
Eelco Dolstra
c045630522 Support channel:<channel-name> URIs
For convenience, you can now say

  $ nix-env -f channel:nixos-16.03 -iA hello

instead of

  $ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello

Similarly,

  $ nix-shell -I channel:nixpkgs-unstable -p hello
  $ nix-build channel:nixos-15.09 -A hello

Abstracting over the NixOS/Nixpkgs channels location also allows us to
use a more efficient transport (e.g. Git) in the future.
2016-04-14 17:26:57 +02:00
Eelco Dolstra
363f37d084 Make the search path lazier with non-fatal errors
Thus, -I / $NIX_PATH entries are now downloaded only when they are
needed for evaluation. An error to download an entry is a non-fatal
warning (just like non-existant paths).

This does change the semantics of builtins.nixPath, which now returns
the original, rather than resulting path. E.g., before we had

  [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

but now

  [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

Fixes #792.
2016-04-14 15:32:24 +02:00
Eelco Dolstra
3c1c6b8f00 Set RLIMIT_CORE to 0, infinity in builders
This prevents the builder from being affected by whatever the host
system limits happen to be.
2016-04-14 13:39:14 +02:00
Eelco Dolstra
12b257f045 Make primop registration pluggable
This way we don't have to put all primops in one giant file.
2016-04-14 12:50:01 +02:00
Eelco Dolstra
96515b0c0d Fix std::atomic_flag build failure
http://hydra.nixos.org/build/34453794
2016-04-14 12:50:01 +02:00
Eelco Dolstra
867967265b Remove manifest support
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
2016-04-11 16:20:15 +02:00
Eelco Dolstra
8cffec8485 Remove failed build caching
This feature was implemented for Hydra, but Hydra no longer uses it.
2016-04-08 18:19:04 +02:00
Eelco Dolstra
f398949b40 Make LocalStore thread-safe
Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Eelco Dolstra
05fbc606fc nix verify-paths: Add ‘--sigs-needed <N>’ flag
This specifies the number of distinct signatures required to consider
each path "trusted".

Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
2016-04-07 15:16:57 +02:00
Eelco Dolstra
6b2ae52808 Use secret-key-files for verifying 2016-04-07 15:16:57 +02:00
Eelco Dolstra
e39999ed48 Sign locally-built paths
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
2016-04-07 15:16:57 +02:00
Eelco Dolstra
dc82160164 Fix "tar: This does not look like a tar archive" with fetchTarball / -I http://
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
2016-04-06 16:57:20 +02:00
Eelco Dolstra
b654381eb3 Add "nix sign-paths" command
E.g.

  $ nix sign-paths -k ./secret -r $(type -p geeqie)

signs geeqie and all its dependencies using the key in ./secret.
2016-04-05 16:39:29 +02:00
Eelco Dolstra
d0f5719c2a Add "nix copy-sigs" command
This imports signatures from one store into another. E.g.

  $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
  imported 595 signatures
2016-04-05 16:39:23 +02:00
Eelco Dolstra
80da7a6375 Probably fix SQLITE_BUSY errors 2016-04-05 15:29:56 +02:00
Eelco Dolstra
37a337bcec throwSQLiteError(): Check for SIGINT so we don't loop forever 2016-04-04 15:08:08 +02:00
Eelco Dolstra
6e120b76ee Add missing -pthread
https://hydra.nixos.org/build/33908385
2016-03-31 12:42:48 +02:00
Eelco Dolstra
9cee600c88 LocalStore: Keep track of ultimately trusted paths
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
2016-03-30 17:35:48 +02:00
Eelco Dolstra
3d119f0a3b Improve the SQLite wrapper API
In particular, this eliminates a bunch of boilerplate code.
2016-03-30 15:50:45 +02:00
Eelco Dolstra
d9c5e3bbf0 Factour out SQLite handling 2016-03-30 13:27:25 +02:00
Eelco Dolstra
2ae43ced9a Turn retrying SQLite transactions into a higher-order function 2016-03-30 12:04:27 +02:00
Eelco Dolstra
39a6abc0bc nix verify: Support checking against signatures in other stores
Typical usage is to check local paths using the signatures from a
binary cache:

  $ nix verify-paths -r /run/current-system -s https://cache.nixos.org
  path ‘/nix/store/c1k4zqfb74wba5sn4yflb044gvap0x6k-nixos-system-mandark-16.03.git.fc2d7a5M’ is untrusted
  ...
  checked 844 paths, 119 untrusted
2016-03-30 11:39:34 +02:00
Eelco Dolstra
88541569a2 HttpBinaryCacheStore: Treat 403 errors as 404 2016-03-30 11:17:51 +02:00
Eelco Dolstra
de88004a9d CurlDownloader: Fix HTTP error processing 2016-03-30 11:17:35 +02:00
Eelco Dolstra
ab3ce1cc13 Improve SIGINT handling in multi-threaded programs
The flag remembering whether an Interrupted exception was thrown is
now thread-local. Thus, all threads will (eventually) throw
Interrupted. Previously, one thread would throw Interrupted, and then
the other threads wouldn't see that they were supposed to quit.
2016-03-29 16:37:16 +02:00
Eelco Dolstra
4f34c40398 Add "nix verify-store" command
Like "nix-store --verify --check-contents", but with the same
advantages as "nix verify-paths".
2016-03-29 16:37:16 +02:00
Eelco Dolstra
784ee35c80 Add "nix verify-paths" command
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.

Example use:

$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
2016-03-29 16:37:16 +02:00
Eelco Dolstra
0ebe69dc67 Re-enable sync_with_stdio
Otherwise writing to std::cerr is not thread-safe (in particular,
lines will be randomly duplicated).
2016-03-29 16:37:16 +02:00
Eelco Dolstra
374198ad6d Move signature support from NarInfo to ValidPathInfo 2016-03-24 11:41:00 +01:00
Eelco Dolstra
11525377e1 Typos 2016-03-24 11:27:58 +01:00
Eelco Dolstra
8b7839b608 HttpBinaryCacheStore: Make thread-safe 2016-03-24 11:10:05 +01:00
Eelco Dolstra
c7d44bad00 Drop support for daemon socket path >= 108 characters
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as

  error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory

Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
2016-03-23 17:16:16 +01:00
Eelco Dolstra
056b3ecfa4 LocalStoreAccessor::stat: Handle ENOTDIR
Closes https://github.com/NixOS/hydra/pull/286.
2016-03-23 11:17:46 +01:00
Eelco Dolstra
bb1034316d Don't overload dumpPath() 2016-03-22 14:21:45 +01:00
Eelco Dolstra
712b616a84 Move signatures from NarInfo to ValidPathInfo
This allows queryPathInfo() to return signatures.
2016-03-21 18:05:47 +01:00
Eelco Dolstra
cebc150b7c nix: Add --store flag
This is a bit user-friendlier than using $NIX_REMOTE.
2016-03-21 18:03:36 +01:00
Eelco Dolstra
1c5f73f529 Add Store::dumpPath() method
This allows applying nix-store --verify-path to binary cache stores:

  NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
2016-03-21 17:55:57 +01:00
Eelco Dolstra
87295b9844 Drop support for upgrading from Nix <= 0.12 2016-03-21 15:09:03 +01:00
Eelco Dolstra
02654f782f Fix Darwin build
http://hydra.nixos.org/build/33279996
2016-03-15 12:11:27 +01:00
Eelco Dolstra
00a75b1cd2 Fix another mismatched tag
http://hydra.nixos.org/build/33279570
2016-03-14 15:32:34 +01:00
Eelco Dolstra
dc4a71aae5 Fix build on clang due to -Wmismatched-tags
http://hydra.nixos.org/build/33073389
2016-03-14 12:37:30 +01:00
Eelco Dolstra
e7c76f7274 BinaryCacheStore::isValidPath(): Use .narinfo cache
If a path is in the .narinfo cache, obviously it's valid.
2016-03-14 12:33:06 +01:00
Eelco Dolstra
674c5ff64f Merge pull request #834 from abbradar/master
Propagate path context via builtins.readFile
2016-03-10 20:17:03 +01:00
Eelco Dolstra
b4e0335d4d Add option binary-cache-secret-key-file for signing binary caches 2016-03-04 17:45:22 +01:00
Eelco Dolstra
af7cdb1096 BinaryCacheStore: Remove publicKeyFile argument
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
2016-03-04 17:45:22 +01:00
Eelco Dolstra
42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Eelco Dolstra
ce113c32d2 Add warning about "nix" being experimental 2016-03-04 15:54:41 +01:00
Eelco Dolstra
7c9d7a253c Merge branch 'new-cli' 2016-03-04 15:21:23 +01:00
Nikolay Amiantov
f7f0116dd7 Propagate path context via builtins.readFile 2016-03-04 14:04:51 +03:00
Eelco Dolstra
76f1ba4f3b Add file missing from 201b48de60 2016-03-03 18:03:34 +01:00
Nathan Zadoks
fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra
5a8455c85e Provide function required by Hydra 2016-03-02 18:21:48 +01:00
Eelco Dolstra
25eea97873 Force stack trace for boost format errors 2016-03-02 15:46:07 +01:00
Eelco Dolstra
201b48de60 Add an HTTP binary cache store
Allowing stuff like

  NIX_REMOTE=https://cache.nixos.org nix-store -qR /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1

or

  NIX_REMOTE=https://cache.nixos.org nix-store --export /nix/store/x1p1gl3a4kkz5ci0nfbayjqlqmczp1kq-geeqie-1.1 | nix-store --import
2016-03-02 15:46:07 +01:00