Eelco Dolstra
30c8c3ba2d
Fix some more references to storeDir
2016-06-03 14:02:44 +02:00
Eelco Dolstra
a8dfdc52b8
Merge pull request #925 from abbradar/master
...
Show both cycle ends
2016-06-03 13:58:51 +02:00
Nikolay Amiantov
c87a56f4d0
Show both cycle ends
2016-06-03 14:25:37 +03:00
Eelco Dolstra
75361b6dce
Fix narrowing conversion from int64_t to bool
...
http://hydra.nixos.org/build/36613774
2016-06-03 13:20:46 +02:00
Eelco Dolstra
d64e0c1b64
Make derivationFromPath work on diverted stores
2016-06-02 18:43:36 +02:00
Eelco Dolstra
a24f2c9b84
nix run: Mount the Nix store in a private namespace
...
This is a convenience command to allow users who are not privileged to
create /nix/store to use Nix with regular binary caches. For example,
$ NIX_REMOTE="local?state=$HOME/nix/var&real=/$HOME/nix/store" nix run firefox bashInteractive
will download Firefox and bash from cache.nixos.org, then start a
shell in which $HOME/nix/store is mounted on /nix/store.
2016-06-02 16:51:43 +02:00
Eelco Dolstra
4494000e04
LocalStore: Allow the physical and logical store directories to differ
...
This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):
$ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'
However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:
$ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
2016-06-02 16:02:48 +02:00
Eelco Dolstra
064816ab98
Respect build-use-substitutes
2016-06-02 16:02:48 +02:00
Eelco Dolstra
812c0dfbe2
Allow setting the state directory as a store parameter
...
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Eelco Dolstra
f2682e6e18
<nix/fetchurl.nix>: Use tarballs.nixos.org
...
This ensures that the disappearance of upstream bootstrap tarballs
(e.g. https://github.com/NixOS/nixpkgs-channels/pull/1 ) doesn't break
stdenv rebuilds.
2016-06-01 17:11:51 +02:00
Eelco Dolstra
dead8189ac
Typo
2016-06-01 16:56:22 +02:00
Eelco Dolstra
e24e2caaaf
Skip substituters with an incompatible store directory
2016-06-01 16:40:49 +02:00
Eelco Dolstra
cf198952d0
HttpBinaryCacheStore: Fix caching of WantMassQuery
...
Also, test HttpBinaryCacheStore in addition to LocalBinaryCacheStore.
2016-06-01 16:24:17 +02:00
Eelco Dolstra
7850d3d279
Make the store directory a member variable of Store
2016-06-01 16:24:17 +02:00
Eelco Dolstra
a9fa5e050a
Shut up some clang warnings
2016-05-31 13:31:04 +02:00
Eelco Dolstra
10f3a2e5f2
Fix clang build failure
...
Apparently opinion is divided on whether [[noreturn]] is allowed on a
lambda: http://stackoverflow.com/questions/26888805/how-to-declare-a-lambdas-operator-as-noreturn
http://hydra.nixos.org/build/36462100
2016-05-31 13:23:54 +02:00
Eelco Dolstra
c2d27d30cf
nix-copy-closure / build-remote.pl: Disable signature checking
...
This restores the Nix 1.11 behaviour.
2016-05-31 11:48:05 +02:00
Eelco Dolstra
cd35f0280c
Fix build
2016-05-30 16:12:50 +02:00
Eelco Dolstra
57d33013ce
Check signatures before downloading the substitute
2016-05-30 15:18:12 +02:00
Eelco Dolstra
bac123ddd9
Test the NAR info cache
2016-05-30 15:18:12 +02:00
Eelco Dolstra
3593c8285d
Re-implement binary cache signature checking
...
This is now done in LocalStore::addToStore(), rather than in the
binary cache substituter (which no longer exists).
2016-05-30 15:18:12 +02:00
Eelco Dolstra
12ddbad458
LocalStore::addToStore: Verify hash of the imported path
2016-05-30 15:18:12 +02:00
Eelco Dolstra
e222484401
Re-implement the WantMassQuery property of binary caches
2016-05-30 15:18:12 +02:00
Eelco Dolstra
b66ab6cdbc
Fix repair during substitution
2016-05-30 15:18:12 +02:00
Eelco Dolstra
3be2e71ab3
BinaryCacheStore: Remove buildPaths() / ensurePath()
2016-05-30 15:18:12 +02:00
Eelco Dolstra
75d2492f20
Make the aws-cpp-sdk dependency optional
2016-05-04 17:16:48 +02:00
Eelco Dolstra
6c75cf69c3
Cleanup: Remove singleton()
2016-05-04 16:16:53 +02:00
Eelco Dolstra
0d4a10e910
Do compression in a sink
2016-05-04 16:16:53 +02:00
Eelco Dolstra
c6a21aed07
More GC fixes
2016-05-04 16:16:53 +02:00
Eelco Dolstra
538a64e8c3
Add a Store::addToStore() variant that accepts a NAR
...
As a side effect, this ensures that signatures are propagated when
copying paths between stores.
Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra
f435f82475
Remove OpenSSL-based signing
2016-05-04 11:01:48 +02:00
Eelco Dolstra
dfebfc835f
Add a copyStorePath() utility function
2016-05-04 11:01:48 +02:00
Eelco Dolstra
38539b943a
Add fetchgit builtin
...
The function builtins.fetchgit fetches Git repositories at evaluation
time, similar to builtins.fetchTarball. (Perhaps the name should be
changed, being confusing with respect to Nixpkgs's fetchgit function,
with works at build time.)
Example:
(import (builtins.fetchgit git://github.com/NixOS/nixpkgs) {}).hello
or
(import (builtins.fetchgit {
url = git://github.com/NixOS/nixpkgs-channels;
rev = "nixos-16.03";
}) {}).hello
Note that the result does not contain a .git directory.
2016-04-29 20:47:36 +02:00
Eelco Dolstra
4dde0b0562
BinaryCacheStore: Support bzip2 compression
2016-04-29 17:43:37 +02:00
Eelco Dolstra
6963de2091
nix verify --all: Support local binary caches
2016-04-29 17:34:31 +02:00
Eelco Dolstra
0dd988d2e3
Better error message
2016-04-29 17:33:22 +02:00
Eelco Dolstra
5acb691402
BinaryCacheStore: Support "none" compression method
2016-04-29 17:02:57 +02:00
Eelco Dolstra
8e065c6b3e
BinaryCacheStore: Make the signing key a parameter
2016-04-29 16:47:20 +02:00
Eelco Dolstra
f6aee2f477
BinaryCacheStore::queryPathInfo: Don't check signatures
...
Other stores don't do this either. It's up to the caller to check
signatures.
2016-04-29 16:28:57 +02:00
Eelco Dolstra
95d20dfde9
Allow parameters in store URIs
...
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-29 16:26:16 +02:00
Eelco Dolstra
aa3bc3d5dc
Eliminate the substituter mechanism
...
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Eelco Dolstra
21e9d183cc
Really handle carriage return
2016-04-28 14:27:00 +02:00
Eelco Dolstra
ce5776758d
Fix error message
2016-04-28 14:12:10 +02:00
Eelco Dolstra
cc804d0dc6
Handle carriage return
2016-04-25 19:18:45 +02:00
Eelco Dolstra
5761827d5b
Show the log tail when a build fails
...
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
6e1b099279
Remove --print-build-trace
...
This was added to support Hydra, but Hydra no longer uses it.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
9eba2c3945
Fix "path is not in the Nix store" during GC
2016-04-25 19:18:45 +02:00
Eelco Dolstra
41633f9f73
Improved logging abstraction
...
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra
0207272b28
BinaryCacheStore: When adding a path, ensure the references are valid
...
This prevents copying a partial closure to a binary cache.
2016-04-22 12:15:06 +02:00
Eelco Dolstra
21ef1670b3
Fix test failures
2016-04-21 18:21:25 +02:00
Eelco Dolstra
7d14f5c331
Implement S3BinaryCacheStore::queryAllValidPaths()
...
This allows commands like "nix verify --all" or "nix path-info --all"
to work on S3 caches.
Unfortunately, this requires some ugly hackery: when querying the
contents of the bucket, we don't want to have to read every .narinfo
file. But the S3 bucket keys only include the hash part of each store
path, not the name part. So as a special exception
queryAllValidPaths() can now return store paths *without* the name
part, and queryPathInfo() accepts such store paths (returning a
ValidPathInfo object containing the full name).
2016-04-21 17:53:47 +02:00
Eelco Dolstra
d155d80155
Move S3BinaryCacheStore from Hydra
...
This allows running arbitrary Nix commands against an S3 binary cache.
To do: make this a compile time option to prevent a dependency on
aws-sdk-cpp.
2016-04-21 16:08:51 +02:00
Eelco Dolstra
ddea253ff8
RemoteStore: Propagate InvalidPath exceptions from the daemon
2016-04-20 15:28:07 +02:00
Eelco Dolstra
c0c4ddcd9c
BinaryCacheStore: Insert new paths into the disk cache
2016-04-20 15:27:48 +02:00
Eelco Dolstra
451ebf24ce
Cache path info lookups in SQLite
...
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
2016-04-20 14:12:38 +02:00
Eelco Dolstra
e0204f8d46
Move path info caching from BinaryCacheStore to Store
...
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.
This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
2016-04-19 18:52:53 +02:00
Dan Peebles
608b0265e1
Print out all bad references/requisites at once
...
Also updates tests to check for new information. Fixes #799
2016-04-16 19:58:26 -04:00
Eelco Dolstra
0423787086
Make the .narinfo cache bigger
2016-04-15 15:39:48 +02:00
Eelco Dolstra
a7d8eaba54
BinaryCacheStore: Do negative caching of .narinfo lookups
2016-04-15 15:39:48 +02:00
Eelco Dolstra
d1b0909894
BinaryCacheStore::readFile(): Return a shared_ptr to a string
...
This allows readFile() to indicate that a file doesn't exist, and
might eliminate some large string copying.
2016-04-15 15:39:48 +02:00
Eelco Dolstra
c045630522
Support channel:<channel-name> URIs
...
For convenience, you can now say
$ nix-env -f channel:nixos-16.03 -iA hello
instead of
$ nix-env -f https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz -iA hello
Similarly,
$ nix-shell -I channel:nixpkgs-unstable -p hello
$ nix-build channel:nixos-15.09 -A hello
Abstracting over the NixOS/Nixpkgs channels location also allows us to
use a more efficient transport (e.g. Git) in the future.
2016-04-14 17:26:57 +02:00
Eelco Dolstra
3c1c6b8f00
Set RLIMIT_CORE to 0, infinity in builders
...
This prevents the builder from being affected by whatever the host
system limits happen to be.
2016-04-14 13:39:14 +02:00
Eelco Dolstra
96515b0c0d
Fix std::atomic_flag build failure
...
http://hydra.nixos.org/build/34453794
2016-04-14 12:50:01 +02:00
Eelco Dolstra
867967265b
Remove manifest support
...
Manifests have been superseded by binary caches for years. This also
gets rid of nix-pull, nix-generate-patches and bsdiff/bspatch.
2016-04-11 16:20:15 +02:00
Eelco Dolstra
8cffec8485
Remove failed build caching
...
This feature was implemented for Hydra, but Hydra no longer uses it.
2016-04-08 18:19:04 +02:00
Eelco Dolstra
f398949b40
Make LocalStore thread-safe
...
Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Eelco Dolstra
05fbc606fc
nix verify-paths: Add ‘--sigs-needed <N>’ flag
...
This specifies the number of distinct signatures required to consider
each path "trusted".
Also renamed ‘--no-sigs’ to ‘--no-trust’ for the flag that disables
verifying whether a path is trusted (since a path can also be trusted
if it has no signatures, but was built locally).
2016-04-07 15:16:57 +02:00
Eelco Dolstra
6b2ae52808
Use secret-key-files for verifying
2016-04-07 15:16:57 +02:00
Eelco Dolstra
e39999ed48
Sign locally-built paths
...
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
2016-04-07 15:16:57 +02:00
Eelco Dolstra
dc82160164
Fix "tar: This does not look like a tar archive" with fetchTarball / -I http://
...
The 304 Not Modified was not handled correctly, so the empty result
from the conditional request would overwrite the previous tarball.
2016-04-06 16:57:20 +02:00
Eelco Dolstra
b654381eb3
Add "nix sign-paths" command
...
E.g.
$ nix sign-paths -k ./secret -r $(type -p geeqie)
signs geeqie and all its dependencies using the key in ./secret.
2016-04-05 16:39:29 +02:00
Eelco Dolstra
d0f5719c2a
Add "nix copy-sigs" command
...
This imports signatures from one store into another. E.g.
$ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/
imported 595 signatures
2016-04-05 16:39:23 +02:00
Eelco Dolstra
80da7a6375
Probably fix SQLITE_BUSY errors
2016-04-05 15:29:56 +02:00
Eelco Dolstra
37a337bcec
throwSQLiteError(): Check for SIGINT so we don't loop forever
2016-04-04 15:08:08 +02:00
Eelco Dolstra
9cee600c88
LocalStore: Keep track of ultimately trusted paths
...
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
2016-03-30 17:35:48 +02:00
Eelco Dolstra
3d119f0a3b
Improve the SQLite wrapper API
...
In particular, this eliminates a bunch of boilerplate code.
2016-03-30 15:50:45 +02:00
Eelco Dolstra
d9c5e3bbf0
Factour out SQLite handling
2016-03-30 13:27:25 +02:00
Eelco Dolstra
2ae43ced9a
Turn retrying SQLite transactions into a higher-order function
2016-03-30 12:04:27 +02:00
Eelco Dolstra
88541569a2
HttpBinaryCacheStore: Treat 403 errors as 404
2016-03-30 11:17:51 +02:00
Eelco Dolstra
de88004a9d
CurlDownloader: Fix HTTP error processing
2016-03-30 11:17:35 +02:00
Eelco Dolstra
784ee35c80
Add "nix verify-paths" command
...
Unlike "nix-store --verify-path", this command verifies signatures in
addition to store path contents, is multi-threaded (especially useful
when verifying binary caches), and has a progress indicator.
Example use:
$ nix verify-paths --store https://cache.nixos.org -r $(type -p thunderbird)
...
[17/132 checked] checking ‘/nix/store/rawakphadqrqxr6zri2rmnxh03gqkrl3-autogen-5.18.6’
2016-03-29 16:37:16 +02:00
Eelco Dolstra
374198ad6d
Move signature support from NarInfo to ValidPathInfo
2016-03-24 11:41:00 +01:00
Eelco Dolstra
11525377e1
Typos
2016-03-24 11:27:58 +01:00
Eelco Dolstra
8b7839b608
HttpBinaryCacheStore: Make thread-safe
2016-03-24 11:10:05 +01:00
Eelco Dolstra
c7d44bad00
Drop support for daemon socket path >= 108 characters
...
Doing a chdir() is a bad idea in multi-threaded programs, leading to
failures such as
error: cannot connect to daemon at ‘/nix/var/nix/daemon-socket/socket’: No such file or directory
Since Linux doesn't have a connectat() syscall like FreeBSD, there is
no way we can support this in a race-free way.
2016-03-23 17:16:16 +01:00
Eelco Dolstra
056b3ecfa4
LocalStoreAccessor::stat: Handle ENOTDIR
...
Closes https://github.com/NixOS/hydra/pull/286 .
2016-03-23 11:17:46 +01:00
Eelco Dolstra
bb1034316d
Don't overload dumpPath()
2016-03-22 14:21:45 +01:00
Eelco Dolstra
712b616a84
Move signatures from NarInfo to ValidPathInfo
...
This allows queryPathInfo() to return signatures.
2016-03-21 18:05:47 +01:00
Eelco Dolstra
1c5f73f529
Add Store::dumpPath() method
...
This allows applying nix-store --verify-path to binary cache stores:
NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
2016-03-21 17:55:57 +01:00
Eelco Dolstra
87295b9844
Drop support for upgrading from Nix <= 0.12
2016-03-21 15:09:03 +01:00
Eelco Dolstra
02654f782f
Fix Darwin build
...
http://hydra.nixos.org/build/33279996
2016-03-15 12:11:27 +01:00
Eelco Dolstra
00a75b1cd2
Fix another mismatched tag
...
http://hydra.nixos.org/build/33279570
2016-03-14 15:32:34 +01:00
Eelco Dolstra
e7c76f7274
BinaryCacheStore::isValidPath(): Use .narinfo cache
...
If a path is in the .narinfo cache, obviously it's valid.
2016-03-14 12:33:06 +01:00
Eelco Dolstra
b4e0335d4d
Add option binary-cache-secret-key-file for signing binary caches
2016-03-04 17:45:22 +01:00
Eelco Dolstra
af7cdb1096
BinaryCacheStore: Remove publicKeyFile argument
...
The public key can be derived from the secret key, so there's no need
for the user to supply it separately.
2016-03-04 17:45:22 +01:00
Eelco Dolstra
42bc395b63
Eliminate some large string copying
2016-03-04 16:49:56 +01:00
Eelco Dolstra
7c9d7a253c
Merge branch 'new-cli'
2016-03-04 15:21:23 +01:00
Eelco Dolstra
76f1ba4f3b
Add file missing from 201b48de60
2016-03-03 18:03:34 +01:00
Nathan Zadoks
fe2be8f016
build.cc: fs.h doesn't appear to be necessary anymore
2016-03-03 14:11:00 +01:00
Eelco Dolstra
5a8455c85e
Provide function required by Hydra
2016-03-02 18:21:48 +01:00