Commit graph

3525 commits

Author SHA1 Message Date
Eelco Dolstra
e09161d05c
Remove signed-binary-caches as the default for require-sigs
This was for backward compatibility. However, with security-related
configuration settings, it's best not to have any confusion.

Issue #495.
2018-01-26 17:12:15 +01:00
Eelco Dolstra
479692a068
Merge pull request #1811 from dtzWill/update/json-3.0.1
nlohmann-json: 2.1.1 -> 3.0.1
2018-01-24 17:11:25 +01:00
Dan Peebles
d43a8b25f0 Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments 2018-01-23 14:45:50 -05:00
Will Dietz
f7c26365eb nlohmann-json: 2.1.1 -> 3.0.1 2018-01-22 12:19:50 -06:00
Eelco Dolstra
89a2a11d9f
Don't use [[noreturn]] 2018-01-19 15:00:38 +01:00
Eelco Dolstra
3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra
2896bb6826
Don't retry CURLE_URL_MALFORMAT 2018-01-19 14:05:08 +01:00
Eelco Dolstra
0c95776c3e
Don't define builtins.{currentSystem,currentTime} in pure mode
This makes it easier to provide a default, e.g.

  system = builtins.currentSystem or "x86_64-linux";
2018-01-18 16:38:48 +01:00
Eelco Dolstra
27b510af5c
nix eval: Stop progress bar before printing the result 2018-01-17 12:04:44 +01:00
Eelco Dolstra
16e0287556
nix eval: Take only one argument
Thus --json no longer produces a list.
2018-01-17 12:03:06 +01:00
Eelco Dolstra
d8b4cfad82
Typo 2018-01-17 11:53:08 +01:00
Eelco Dolstra
75b9670df6
Make show-trace a config setting 2018-01-16 19:24:23 +01:00
Eelco Dolstra
d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra
23fa7e3606
parseExprFromFile -> evalFile
parseExprFromFile() should be avoided since it doesn't cache anything.
2018-01-16 17:11:58 +01:00
Eelco Dolstra
ba75c69e00
Barf when using a diverted store on macOS
Fixes #1792.
2018-01-15 12:14:43 +01:00
Will Dietz
6454ca393a nix log: use pager 2018-01-12 15:05:26 -06:00
Eelco Dolstra
74f75c8558
import, builtins.readFile: Handle diverted stores
Fixes #1791
2018-01-12 17:31:08 +01:00
Renzo Carbonara
b0328c244d nix.conf: builders-use-substitutes
Fixes #937
2018-01-09 22:40:07 +01:00
Will Dietz
428680b307 fetchGit: fix creation of uninitialized cache dir, let git create it
fetchGit test (as modified in previous commit) now passes.
2018-01-09 09:05:18 -06:00
Eelco Dolstra
7b9583680e
Improve error message with --repair for untrusted users 2018-01-08 19:13:48 +01:00
Eelco Dolstra
44272d8719
Rename "use-substitutes" to "substitute"
Commit c2154d4c84 renamed
"build-use-substitutes" to "use-substitutes", but that broke
"nix-copy-closure --use-substitutes".
2018-01-04 16:58:39 +01:00
Shea Levy
689b2783fc
Add hasContext primop 2018-01-02 12:25:14 -05:00
Will Dietz
9dd2b8ac7b use libbrotli directly when available
* Look for both 'brotli' and 'bro' as external command,
  since upstream has renamed it in newer versions.
  If neither are found, current runtime behavior
  is preserved: try to find 'bro' on PATH.
* Limit amount handed to BrotliEncoderCompressStream
  to ensure interrupts are processed in a timely manner.
  Testing shows negligible performance impact.
  (Other compression sinks don't seem to require this)
2017-12-30 20:26:33 -06:00
Shea Levy
6a0dd63508
Merge branch 'fix/issue-1757' of git://github.com/dtzWill/nix 2017-12-27 18:51:05 -05:00
Will Dietz
bd17ccf1d8 nix repl: use linenoiseKeyType to differentiate ^C and ^D
Fixes #1757.
2017-12-26 19:25:50 -06:00
Will Dietz
2e6f06c37e fetchGit: Fix handling of local repo when not using 'master' branch
Add tests checking this behavior.
2017-12-22 15:29:52 -06:00
Eelco Dolstra
aa43cbb764
Check aws-sdk-cpp version 2017-12-22 12:05:13 +01:00
Will Dietz
71987b18d4 linenoise.cpp: allow completions from empty input
Fixes #1742.
2017-12-19 12:52:10 -06:00
Eelco Dolstra
8efe937a35 Fix build on gcc 7
Fixes #1738.
2017-12-16 00:49:31 +01:00
Ben Gamari
626a94d70e edit: Catch stoi exceptions from line number parsing 2017-12-14 19:11:56 -05:00
Ben Gamari
f9bcbddef2 json-to-value: Throw sensible error message on invalid numbers 2017-12-14 19:09:45 -05:00
Eelco Dolstra
5d5b931fb1 Fix build against current aws-sdk-cpp 2017-12-14 16:45:40 +01:00
Maximilian Bosch
0413aeb35d
search.cc: sort attribute names with std::map 2017-12-12 15:13:36 +01:00
Eelco Dolstra
1ca8e95178
Merge pull request #1722 from bhipple/fix-for-older-libcurl
Fix for builds with system libcurl < 7.30
2017-12-12 11:48:08 +01:00
Will Dietz
00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Will Dietz
e4d43f1f94 logging.cc: add missing 'override' 2017-12-11 12:08:40 -06:00
Will Dietz
c800f47395 Fix forward-decl of Bindings as "class", match definition.
(appease clang -Wmismatched-tags warning)
2017-12-11 11:51:55 -06:00
Shea Levy
5ba5ca7888
Merge branch 'feature/linenoise-ng' of git://github.com/dtzWill/nix 2017-12-11 07:52:20 -05:00
Benjamin Hipple
d76c4fe770 Fix for builds with system libcurl < 7.30
CentOS 7.4 and RHEL 7.4 ship with libcurl-devel-7.29.0-42.el7.x86_64; this flag
was added in 7.30.0
https://curl.haxx.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html
2017-12-09 20:02:21 -05:00
Eelco Dolstra
2df9cbeb47 Provide random access to cached NARs
E.g.

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m4.139s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m0.024s

(Before, the second call took ~0.220s.)

This will use a NAR listing in
/tmp/nars/b0w2hafndl09h64fhb86kw6bmhbmnpm1.ls containing all metadata,
including the offsets of regular files inside the NAR. Thus, we don't
need to read the entire NAR. (We do read the entire listing, but
that's generally pretty small. We could use a SQLite DB by borrowing
some more code from nixos-channel-scripts/file-cache.hh.)

This is primarily useful when Hydra is serving files from an S3 binary
cache, in particular when you have giant NARs. E.g. we had some 12 GiB
NARs, so accessing individuals files was pretty slow.
2017-12-07 01:07:07 +01:00
Eelco Dolstra
338f29dbd4 nix ls-{nar,store}: Return offset of files in the NAR if known
E.g.

  $ nix ls-store --json --recursive --store https://cache.nixos.org /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79 \
    | jq .entries.bin.entries.blender.narOffset
  400
2017-12-07 01:07:07 +01:00
Daiderd Jordan
2cb8aaa610
ssh-store: fix length when writing nar
This fixes nix copy and other things that use copyStorePath.
2017-12-06 11:41:08 +01:00
Shea Levy
11a7f8ce14
Merge branch 'fetchGit-fast-revision-update' 2017-12-05 11:16:39 -05:00
Eelco Dolstra
7f2c324ed1
Simplify build by including nlohmann/json.hpp 2017-12-04 17:11:36 +01:00
Eelco Dolstra
f1efb97075
Fix #1635. 2017-12-04 13:19:16 +01:00
Linus Heckemann
f986a44980 Fix "Unexpected EOF reading a line" error
propagated-user-env-packages files in nixpkgs aren't all terminated by
newlines, as buildenv expected. Now it does not require a terminating
newline; note that this introduces a behaviour change: propagated user
env packages may now be spread across multiple lines. However, nix
1.11.x still expects them to be on a single line so this shouldn't be
used in nixpkgs for now.
2017-11-29 11:56:15 +00:00
Eelco Dolstra
b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Will Dietz
964349e44d no "linenoiseFree" in linenoise-ng 2017-11-27 18:30:05 -06:00
Will Dietz
21f515724c replace vendored linenoise with linenoise-ng 2017-11-27 18:27:28 -06:00
aszlig
6567ab95a0
build-remote: Fix missing log output
The storeUri variable in the build-remote hook is declared very much to
the start of the main function and a bunch of lines later, the same
variable gets checked via hasPrefix() but it gets assigned *after* that
check when the most suitable machine for the build was choosen.

So I guess this was just a typo in d16fd24973
and what we really want is to either checkd the prefix *after* assigning
storeUri or use bestMachine->storeUri directly.

I choose the latter, because the former could introduce even more
regressions if the try block where the variable gets assigned terminates
early.

Nevertheless, the reason why the log output didn't work is because
hasPrefix() checked for "ssh://" in front of storeUri, but if the
storeUri isn't set correctly (or at all), we don't get the log file
descriptor set up properly, leading to no log output.

I've adjusted the remote-builds test to include a regression test for
this, so that we can make sure we get a build output when using remote
builds.

In addition to that I've tested this with two of my build farms and the
build logs are emitted correctly again.

Signed-off-by: aszlig <aszlig@nix.build>
2017-11-25 01:34:18 +01:00
Eelco Dolstra
549c3706a5
nix path-info: Show URL of NARs in binary caches 2017-11-24 18:08:50 +01:00
Eelco Dolstra
90948a4e3a
nix-shell/nix-build: Support .drv files again
Fixes #1663.

Also handle '!<output-name>' (#1694).
2017-11-24 18:08:35 +01:00
Shea Levy
eedbc4e06c
fetchGit: Ignore tarballTtl if rev is set and not in the repo.
Fixes #1697.
2017-11-24 06:09:24 -05:00
Eelco Dolstra
d7da6c9ea9
fetchGit/fetchMercurial: Fix directory inclusion check
E.g. the existence of .gitignore would cause .git to be included.
2017-11-21 19:34:46 +01:00
Eelco Dolstra
6cdaa858d0
Propagate flags like --sandbox to the daemon properly 2017-11-21 18:50:56 +01:00
Eelco Dolstra
f5f29dc2b7
nix eval --raw: Use coerceToString()
Thus,

  $ nix eval --raw '("foo")'
  foo

  $ nix eval --raw nixpkgs.hello
  /nix/store/1y6ckg6khrdsvll54s5spcmf3w6ka9k4-hello-2.10

  $ nix eval --raw '(/etc/resolv.conf)'
  /nix/store/vml92ama92i8mz013nny461mlvg8mvap-resolv.conf
2017-11-20 18:27:29 +01:00
Eelco Dolstra
1ff01187e2
nix run: Fix "flag '--command' requires 2 argument(s)" 2017-11-20 18:07:58 +01:00
Eelco Dolstra
4eb9e20028
nix run: Fix accidental removal of /nix/store existence check
Parenthetical to #1686, we don't need to create a new root if we can
just bind-mount on top of the existing /nix/store.
2017-11-20 17:59:32 +01:00
Eelco Dolstra
a3aa850f0f
nix run: Ignore non-directories while setting up the chroot
Fixes #1686.
2017-11-20 17:58:47 +01:00
Eelco Dolstra
91a1987607
signed-binary-caches -> require-sigs
Unlike signed-binary-caches (which could only be '*' or ''),
require-sigs is a proper Boolean option. The default is true.
2017-11-20 17:44:07 +01:00
Eelco Dolstra
d0b88db441
Cleanup 2017-11-20 17:32:58 +01:00
Eelco Dolstra
7a2b64e55c
binary-cache-public-keys -> trusted-public-keys
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
2017-11-20 17:32:34 +01:00
Eelco Dolstra
72804dc0bd
Merge pull request #1645 from twhitehead/stat-race
Fix (highly unlikely) race condition in readLink
2017-11-20 17:32:27 +01:00
Eelco Dolstra
7474ac871b
nix copy: Abbreviate "daemon" 2017-11-20 15:17:11 +01:00
Eelco Dolstra
8956ae1987
Add a "profile" option to S3BinaryCacheStore
This allows specifying the AWS configuration profile to use. E.g.

  nix copy --from s3://my-cache?profile=aws-dev-account /nix/store/cf3isrlqavvd5w7rpky1fa8j9lcnlggm-...
2017-11-15 14:18:20 +01:00
Will Dietz
9d7ce0bf45 anchor nix::Exit exception 2017-11-14 12:39:30 -06:00
Eelco Dolstra
ec5b04862b
nix sign-paths: Support binary caches 2017-11-14 18:44:05 +01:00
Eelco Dolstra
2c39e4eca0
Revert "Don't parse "x:x" as a URI"
This reverts commit f90f660b24.

This broke Hydra's release.nix, which contained

  preCheck = ''export LOGNAME=${LOGNAME:-foo}'';
2017-11-14 15:10:52 +01:00
Eelco Dolstra
5773d667ee
nix ls-{nar,store}: Don't abort on missing files 2017-11-14 14:49:06 +01:00
Eelco Dolstra
4db0a9555e
nix ls-{nar,store} --json: Respect -R 2017-11-14 14:31:38 +01:00
Eelco Dolstra
c8155e9f5f
Don't indent JSON output 2017-11-14 14:28:03 +01:00
Eelco Dolstra
bac8055652
nix ls-{store,nar}: Add --json flag 2017-11-14 14:23:53 +01:00
Eelco Dolstra
5ef5d70b5e
Make config options available to legacy commands 2017-11-14 14:04:09 +01:00
Eelco Dolstra
9d87d03331
Remove extraneous comment 2017-11-08 16:17:32 +01:00
Eelco Dolstra
513b143cd8
Merge pull request #1650 from copumpkin/darwin-sandbox-unix-socket
Always allow builds to use unix domain sockets in Darwin sandbox
2017-11-08 16:16:42 +01:00
Eelco Dolstra
dc30856141
Merge pull request #1632 from AmineChikhaoui/sigint-copy
run query paths in parallel during nix copy and handle SIGINT
2017-11-06 13:36:28 +01:00
Eelco Dolstra
7a4d9574d9
fetchgit.cc -> fetchGit.cc 2017-11-03 13:55:31 +01:00
Eelco Dolstra
6cf7c6a6b0
Remove git:// support in NIX_PATH
This didn't support specifying a revision/branch, and was restricted
to git:// URIs (since https:// or ssh:// would be ambiguous).
2017-11-03 13:55:31 +01:00
Eelco Dolstra
ee6ac38848
fetchGit/fetchMercurial: Filter out directories with untracked files 2017-11-03 13:55:31 +01:00
Eelco Dolstra
4dee01da7c
fetchGit: Add a test 2017-11-03 13:55:30 +01:00
Eelco Dolstra
0e77aa3982
fetchGit: Don't do a remote fetch if we already have the rev 2017-11-03 13:55:30 +01:00
Eelco Dolstra
9ccea31dc2
Merge pull request #1651 from LnL7/darwin-sandbox-getpwuid
Allow getpwuid in the darwin sandbox
2017-11-03 10:55:31 +01:00
Daiderd Jordan
453f675810
Allow getpwuid in the darwin sandbox. 2017-11-03 10:50:49 +01:00
Eelco Dolstra
e104842f8b
Merge pull request #1660 from 4z3/patch-1
fetchMercurial: fix error message
2017-11-03 10:49:57 +01:00
Eelco Dolstra
4070264613
Merge pull request #1655 from copumpkin/patch-1
Don't freak out if we get a 403 from S3
2017-11-03 10:49:38 +01:00
Daniel Peebles
3105679226 Don't freak out if we get a 403 from S3
As far as we're concerned, not being able to access a file just means
the file is missing. Plus, AWS explicitly goes out of its way to
return a 403 if the file is missing and the requester doesn't have
permission to list the bucket.

Also getting rid of an old hack that Eelco said was only relevant
to an older AWS SDK.
2017-11-03 01:31:42 +01:00
tv
5ab37f0e44
fetchMercurial: fix error message 2017-11-02 23:37:42 +01:00
Eelco Dolstra
212e72c609 Fix build
https://hydra.nixos.org/build/63172338
2017-11-01 21:32:30 +01:00
Eelco Dolstra
e026bc3b05
fetchMercurial: Don't fetch hashes we already have 2017-11-01 18:43:11 +01:00
Eelco Dolstra
1969f357b7
Add fetchMercurial primop
E.g.

  $ nix eval '(fetchMercurial https://www.mercurial-scm.org/repo/hello)'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "82e55d328c8ca4ee16520036c0aaace03a5beb65"; revCount = 1; shortRev = "82e55d328c8c"; }

  $ nix eval '(fetchMercurial { url = https://www.mercurial-scm.org/repo/hello; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; })'
  { branch = "default"; outPath = "/nix/store/alvb9y1kfz42bjishqmyy3pphnrh1pfa-source"; rev = "0a04b987be5ae354b710cefeba0e2d9de7ad41a9"; revCount = 0; shortRev = "0a04b987be5a"; }

  $ nix eval '(fetchMercurial /tmp/unclean-hg-tree)'
  { branch = "default"; outPath = "/nix/store/cm750cdw1x8wfpm3jq7mz09r30l9r024-source"; rev = "0000000000000000000000000000000000000000"; revCount = 0; shortRev = "000000000000"; }
2017-11-01 17:45:32 +01:00
Eelco Dolstra
cd532a9251
Fix filterSource 2017-10-31 16:18:32 +01:00
Dan Peebles
bc6b3f7e8f Always allow builds to use unix domain sockets in Darwin sandbox 2017-10-31 15:33:57 +01:00
Eelco Dolstra
72cd52c3cd
builtins.fetchgit: Support importing a working tree
For example, you can write

  src = fetchgit ./.;

and if ./. refers to an unclean working tree, that tree will be copied
to the Nix store. This removes the need for "cleanSource".
2017-10-30 19:59:25 +01:00
Eelco Dolstra
197922ea4e
Merge pull request #1646 from copumpkin/optional-sandbox-local-network
Allow optional localhost network access to sandboxed derivations
2017-10-30 18:54:40 +01:00
Dan Peebles
4a4a009f78 Allow optional localhost network access to sandboxed derivations
This will allow bind and connect to 127.0.0.1, which can reduce purity/
security (if you're running a vulnerable service on localhost) but is
also needed for a ton of test suites, so I'm leaving it turned off by
default but allowing certain derivations to turn it on as needed.

It also allows DNS resolution of arbitrary hostnames but I haven't found
a way to avoid that. In principle I'd just want to allow resolving
localhost but that doesn't seem to be possible.

I don't think this belongs under `build-use-sandbox = relaxed` because we
want it on Hydra and I don't think it's the end of the world.
2017-10-30 17:59:12 +01:00
Eelco Dolstra
f90f660b24
Don't parse "x:x" as a URI
URIs now have to contain "://" or start with "channel:".
2017-10-30 17:58:01 +01:00
Tyson Whitehead
07d2c6d213
Fix (highly unlikely) race condition in readLink
Used to determine symlink size with stat and value with readlink.
This could technically result in garbage if symlink changed between
calls.  Also gets around the broken stat implementation in our
network filesystem (returns size + 1 giving a byte of garbage).
2017-10-30 11:49:55 -04:00
Eelco Dolstra
12991152be
nix-build: Fix --hash 2017-10-30 13:31:05 +01:00
Eelco Dolstra
63c80ae26f
Make "fetchGit /path" work 2017-10-30 13:18:28 +01:00
Eelco Dolstra
a5c392a80e
fetchGit: Fix broken assertion
Different URIs can map to the same cache entry if they have the same
revision.
2017-10-30 12:55:46 +01:00
Eelco Dolstra
812e027e1d
Add option allowed-uris
This allows network access in restricted eval mode.
2017-10-30 12:41:49 +01:00
Eelco Dolstra
e38382895d
builtins.fetchGit: Return an attrset with revision info
This adds rev, shortRev and revCount attributes, equal to what Hydra
provides. E.g.

  $ nix eval '(fetchGit https://github.com/NixOS/patchelf.git)'
  { outPath = "/nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source"; rev = "29c085fd9d3fc972f75b3961905d6b4ecce7eb2b"; revCount = 303; shortRev = "29c085f"; }
2017-10-30 11:49:03 +01:00
Eelco Dolstra
f9686885be
enable-http2 -> http2 2017-10-30 11:00:59 +01:00
Eelco Dolstra
049322702b
fetchgit -> fetchGit
Almost all other primops are camelCase so no reason not to use that
here.
2017-10-30 10:25:08 +01:00
Eelco Dolstra
23ce4b3393
fetchTarball: Use "source" as the default name
This ensures that it produces the same output as fetchgit:

  $ nix eval --raw '(builtins.fetchgit https://github.com/NixOS/patchelf.git)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source

  $ nix eval --raw '(fetchTarball https://github.com/NixOS/patchelf/archive/master.tar.gz)'
  /nix/store/ghigrkw02l440g8vfxa9wj4c3zpfmw99-source
2017-10-30 10:22:58 +01:00
Eelco Dolstra
66ddbef754
fetchurl/fetchTarball: Respect name changes
The computation of urlHash didn't take the name into account, so
subsequent fetchurl calls with the same URL but a different name would
resolve to the same cached store path.
2017-10-30 10:22:58 +01:00
Eelco Dolstra
f25791c196
builtins.fetchgit: Don't mess up the progress indicator 2017-10-30 10:22:58 +01:00
Eelco Dolstra
65b5f177b5
builtins.fetchgit: Support a "name" attribute
The "name" attribute defaults to "source", which we should use for all
similar functions (e.g. fetchTarball and in Hydra) to ensure that we
get a consistent store path regardless of how the tree is fetched.

"source" is not necessarily a correct label, but using an empty name
is problematic: you get an ugly store path ending in a dash, and it's
impossible to have a fixed-output derivation that produces that path
because ".drv" is not a valid store name.

Fixes #904.
2017-10-30 10:22:07 +01:00
Eelco Dolstra
c1ae18941a
nix repl :t: Indicate strings that have a context 2017-10-29 17:11:22 +01:00
AmineChikhaoui
0f9a7225ab respect SIGINT in nix copy during the paths queries #1629 2017-10-25 16:51:45 +01:00
AmineChikhaoui
54a2cd9ce4 Merge branch 'master' of github.com:NixOS/nix into sigint-copy 2017-10-25 16:50:52 +01:00
Eelco Dolstra
9971d875a4
Fix building on clang
https://hydra.nixos.org/build/62945761
2017-10-25 17:20:47 +02:00
AmineChikhaoui
9f01a3f0a8 attempt to fix #1630: make the queries of store paths run in parallel using a thread pool 2017-10-25 16:13:49 +01:00
Eelco Dolstra
82327e3cc4
exportReferencesGraph: Allow exporting a list of store paths 2017-10-25 15:18:49 +02:00
Eelco Dolstra
8191992c83
Bump language version 2017-10-25 15:18:37 +02:00
Eelco Dolstra
3395e3bbc4
Fix exportReferencesGraph in the structured attrs case 2017-10-25 14:08:29 +02:00
Eelco Dolstra
2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra
86aab6e9a0
Add --store argument to legacy commands 2017-10-24 15:41:11 +02:00
Eelco Dolstra
d16fd24973
Allow shorter syntax for chroot stores
You can now say '--store /tmp/nix' instead of '--store local?root=/tmp/nix'.
2017-10-24 15:32:38 +02:00
Eelco Dolstra
3460e4cf00
More progress indicator improvements
In particular, don't show superfluous "fetching path" and "building
path(s)" messages, and show the current round (with --repeat).
2017-10-24 15:32:38 +02:00
Eelco Dolstra
96051dd057
More progress indicator improvements
Fixes #1599.
2017-10-24 14:47:23 +02:00
Eelco Dolstra
be220702a7
Progress indicator: Show on what machine we're building
E.g.

  $ nix build nixpkgs.hello --builders 'root@wendy'
  [1/0/1 built] building hello-2.10 on ssh://root@wendy: checking for minix/config.h... no
2017-10-24 14:24:57 +02:00
Eelco Dolstra
f8624762ac
build-remote: Work properly on a chrooted store 2017-10-24 14:14:28 +02:00
Eelco Dolstra
fe9d2f974d
Handle log messages from build-remote
This makes the progress indicator show statuses like "connecting to
'root@machine'".
2017-10-24 14:05:22 +02:00
Eelco Dolstra
0d59f1ca49
nix: Respect -I, --arg, --argstr
Also, random cleanup to argument handling.
2017-10-24 12:58:34 +02:00
Eelco Dolstra
25f32625e2
Remove the remote-builds option
This is superfluous since you can now just set "builders" to empty,
e.g. "--builders ''".
2017-10-24 11:00:16 +02:00
Eelco Dolstra
af241ae7d3
Remove the builder-files option
You can now include files via the "builders" option, using the syntax
"@<filename>". Having only one option makes it easier to override
builders completely.

For backward compatibility, the default is "@/etc/nix/machines", or
"@<filename>" for each file name in NIX_REMOTE_SYSTEMS.
2017-10-24 10:54:43 +02:00
Eelco Dolstra
37fbfffd8e
Pass all settings to build-remote
This ensures that command line flags such as --builders get passed
correctly.
2017-10-23 20:50:28 +02:00
Eelco Dolstra
f32cdc4fab
Turn $NIX_REMOTE into a configuration option 2017-10-23 20:50:28 +02:00
Eelco Dolstra
863a45f1f3
build-remote: Put current load under the store state directory
Fixes the error

  error: opening lock file '/nix/var/nix/current-load/main-lock': Permission denied

when using a chroot store.
2017-10-23 20:50:24 +02:00
Eelco Dolstra
f31c66d392 Merge pull request #1616 from copumpkin/separate-darwin-sandbox
Shift Darwin sandbox to separate installed files
2017-10-20 12:00:27 +02:00
Shea Levy
c7af84ce84
nix-shell: Fix file name resolution in shebangs 2017-10-19 15:02:40 -04:00
Eelco Dolstra
ef1d0142a0
Revert "Let's not populate the NAR cache from hydra-queue-runner for now"
This reverts commit 908590dc6c. Since
hydra-server can have a different store URI from hydra-queue-runner
now, we don't really need this.
2017-10-19 12:11:07 +02:00
Eelco Dolstra
908590dc6c
Let's not populate the NAR cache from hydra-queue-runner for now 2017-10-18 15:45:39 +02:00
Eelco Dolstra
75cd75b1ae
Errors writing to the NAR cache should not be fatal 2017-10-18 15:41:14 +02:00
Eelco Dolstra
d8306148e0
Suppress "copying 0 paths" message 2017-10-18 15:02:58 +02:00
Dan Peebles
6e5165b773 Shift Darwin sandbox to separate installed files
This makes it slightly more manageable to see at a glance what in a
build's sandbox profile is unique to the build and what is standard. Also
a first step to factoring more of our Darwin logic into scheme functions
that will allow us a bit more flexibility. And of course less of that
nasty codegen in C++! 😀
2017-10-17 19:15:51 -04:00
Eelco Dolstra
b24b8ef77c BinaryCacheStore::addToStore(): Add NARs to the local cache 2017-10-17 21:39:48 +02:00
Eelco Dolstra
ca580bec35 BinaryCacheStore: Support local caching of NARs
This speeds up commands like "nix cat-store". For example:

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf
  real    0m4.336s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf
  real    0m0.045s

The primary motivation is to allow hydra-server to serve files from S3
binary caches. Previously Hydra had a hack to do "nix-store -r
<path>", but that fetches the entire closure so is prohibitively
expensive.

There is no garbage collection of the NAR cache yet. Also, the entire
NAR is read when accessing a single member file. We could generate the
NAR listing to provide random access.

Note: the NAR cache is indexed by the store path hash, not the content
hash, so NAR caches should not be shared between binary caches, unless
you're sure that all your builds are binary-reproducible.
2017-10-17 21:15:33 +02:00
Eelco Dolstra
11ba4302e3 Remove BinaryCacheStoreAccessor
Probably as a result of a bad merge in
4b8f1b0ec0, we had both a
BinaryCacheStoreAccessor and a
RemoteFSAccessor. BinaryCacheStore::getFSAccessor() returned the
latter, but BinaryCacheStore::addToStore() checked for the
former. This probably caused hydra-queue-runner to download paths that
it just uploaded.
2017-10-17 20:51:42 +02:00
Shea Levy
cbc216911d
fetchgit: Remove incomplete/unneeded ref check.
This check fails for tags and branches, and is made redundant by the
checks git itself will do when fetching the repo.
2017-10-17 12:30:54 -04:00
Shea Levy
4e58294ae6
fetchgit: Remove incomplete/unneeded isURI check.
This check spuriously fails for e.g. git@github.com:NixOS/nixpkgs.git,
and even for ssh://git@github.com/NixOS/nixpkgs.git, and is made
redundant by the checks git itself will do when fetching the repo. We
instead pass a -- before passing the URI to git to avoid injection.
2017-10-16 14:35:28 -04:00
Eelco Dolstra
1dd29d7aeb
Add option to disable the seccomp filter
I needed this to test ACL/xattr removal in
canonicalisePathMetaData(). Might also be useful if you need to build
old Nixpkgs that doesn't have the required patches to remove
setuid/setgid creation.
2017-10-12 18:21:55 +02:00
Eelco Dolstra
97307811ee Merge pull request #1571 from teto/improve_priority_msg
Improve error message for conflicting priorities
2017-10-12 13:10:44 +02:00
Eelco Dolstra
177aee07c6 Merge pull request #1580 from copumpkin/retry-after-timeout
Retry in all error cases but a few
2017-10-12 13:05:45 +02:00
Eelco Dolstra
fda7b95cb0
Fix a hang in ThreadPool
The worker threads could exit prematurely if they finished processing
all items while the main thread was still adding items. In particular,
this caused hanging nix-store --serve processes in the build farm.

Also, process items from the main thread.
2017-10-09 15:41:09 +02:00
Dan Peebles
476493dbf5 Reverse retry logic to retry in all but a few cases
It was getting too much like whac-a-mole listing all the retriable error
conditions, so we now retry by default and list the cases where retrying
is almost certainly hopeless.
2017-10-02 23:22:02 -04:00
Eelco Dolstra
f3e0d46821
nix run: Restore signals
Otherwise Ctrl-C doesn't work.
2017-09-28 17:58:59 +02:00
Eelco Dolstra
5421ad243e
nix copy: make recursive by default 2017-09-27 18:28:54 +02:00
Eelco Dolstra
b8875213dc
Tweak message 2017-09-27 13:21:25 +02:00