Commit graph

1633 commits

Author SHA1 Message Date
AmineChikhaoui
47ad88099b
move the parallel-compression setting to binary-cache-store, the setting
can be done now from the url e.g s3://nix-cache?parallel-compression=1
instead of nix.conf.
2018-02-07 21:06:11 +01:00
AmineChikhaoui
55ecdfe2a8
make multi threaded compression configurable and use single threaded
by default.
2018-02-07 17:54:08 +01:00
Eelco Dolstra
84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra
abe6be578b
Merge pull request #1816 from shlevy/add-path
Add path primop.
2018-02-07 13:32:35 +01:00
Shea Levy
69d82e5c58
Add path primop.
builtins.path allows specifying the name of a path (which makes paths
with store-illegal names now addable), allows adding paths with flat
instead of recursive hashes, allows specifying a filter (so is a
generalization of filterSource), and allows specifying an expected
hash (enabling safe path adding in pure mode).
2018-02-06 16:48:08 -05:00
Eelco Dolstra
bb1d046f5c
Allow substituters to be marked as trusted
This is needed by nixos-install, which uses the Nix store on the
installation CD as a substituter. We don't want to disable signature
checking entirely because substitutes from cache.nixos.org should
still be checked. So now we can pas "local?trusted=1" to mark only the
Nix store in /nix as not requiring signatures.

Fixes #1819.
2018-02-05 18:08:30 +01:00
Eelco Dolstra
2175eee9fe
Fix segfault using non-binary cache stores as substituters 2018-02-05 17:46:43 +01:00
Eelco Dolstra
84722d67d2
Remove nix-build --hash
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.

It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
2018-02-03 10:08:05 +01:00
Shea Levy
de96daf54f
Merge branch 'master' of git://github.com/catern/nix 2018-02-01 13:21:45 -05:00
Eelco Dolstra
855699855f
Remove obsolete references to manifests
Closes #323.
2018-02-01 10:39:16 +01:00
Spencer Baugh
e5432574e2 document ability to set NIX_REMOTE=unix://path/to/socket 2018-01-31 22:47:16 +00:00
Eelco Dolstra
c129fc6ee8
Merge pull request #1801 from catern/master
remote_store: register for NIX_REMOTE=unix://path
2018-01-31 23:12:39 +01:00
Eelco Dolstra
88f4f0231b Fix building without aws-sdk-cpp 2018-01-31 22:34:51 +01:00
Eelco Dolstra
c287d73121
Rename 1.12 -> 2.0
Following discussion with Shea and Graham. It's a big enough change
from the last release. Also, from a semver perspective, 2.0 makes more
sense because we did remove some interfaces (like nix-pull/nix-push).
2018-01-31 18:58:45 +01:00
Eelco Dolstra
30370f168f
Cleanup 2018-01-31 15:14:03 +01:00
Eelco Dolstra
6270b2e50f
Merge branch 'http-binary-cache-put-upsert' of https://github.com/adelbertc/nix 2018-01-31 15:10:12 +01:00
Eelco Dolstra
478e3e4649
Indent properly 2018-01-31 13:46:38 +01:00
Spencer Baugh
746f8aed86 remote_store: register for NIX_REMOTE=unix://path
This allows overriding the socket path so the daemon may be listening at
an arbitrary Unix domain socket location.

Fixes #1800
2018-01-26 22:05:16 +00:00
Shea Levy
1d5d277ac7
HttpBinaryCacheStore: Support upsertFile with PUT.
Some servers, such as Artifactory, allow uploading with PUT and BASIC
auth. This allows nix copy to work to upload binaries to those
servers.

Worked on together with @adelbertc
2018-01-26 11:12:30 -08:00
Eelco Dolstra
e09161d05c
Remove signed-binary-caches as the default for require-sigs
This was for backward compatibility. However, with security-related
configuration settings, it's best not to have any confusion.

Issue #495.
2018-01-26 17:12:15 +01:00
Dan Peebles
d43a8b25f0 Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments 2018-01-23 14:45:50 -05:00
Eelco Dolstra
3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra
2896bb6826
Don't retry CURLE_URL_MALFORMAT 2018-01-19 14:05:08 +01:00
Eelco Dolstra
75b9670df6
Make show-trace a config setting 2018-01-16 19:24:23 +01:00
Eelco Dolstra
d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra
ba75c69e00
Barf when using a diverted store on macOS
Fixes #1792.
2018-01-15 12:14:43 +01:00
Eelco Dolstra
74f75c8558
import, builtins.readFile: Handle diverted stores
Fixes #1791
2018-01-12 17:31:08 +01:00
Renzo Carbonara
b0328c244d nix.conf: builders-use-substitutes
Fixes #937
2018-01-09 22:40:07 +01:00
Eelco Dolstra
44272d8719
Rename "use-substitutes" to "substitute"
Commit c2154d4c84 renamed
"build-use-substitutes" to "use-substitutes", but that broke
"nix-copy-closure --use-substitutes".
2018-01-04 16:58:39 +01:00
Eelco Dolstra
aa43cbb764
Check aws-sdk-cpp version 2017-12-22 12:05:13 +01:00
Eelco Dolstra
8efe937a35 Fix build on gcc 7
Fixes #1738.
2017-12-16 00:49:31 +01:00
Eelco Dolstra
5d5b931fb1 Fix build against current aws-sdk-cpp 2017-12-14 16:45:40 +01:00
Eelco Dolstra
1ca8e95178
Merge pull request #1722 from bhipple/fix-for-older-libcurl
Fix for builds with system libcurl < 7.30
2017-12-12 11:48:08 +01:00
Will Dietz
00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Benjamin Hipple
d76c4fe770 Fix for builds with system libcurl < 7.30
CentOS 7.4 and RHEL 7.4 ship with libcurl-devel-7.29.0-42.el7.x86_64; this flag
was added in 7.30.0
https://curl.haxx.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html
2017-12-09 20:02:21 -05:00
Eelco Dolstra
2df9cbeb47 Provide random access to cached NARs
E.g.

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m4.139s

  $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nars \
    /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79/share/icons/hicolor/scalable/apps/blender.svg > /dev/null
  real    0m0.024s

(Before, the second call took ~0.220s.)

This will use a NAR listing in
/tmp/nars/b0w2hafndl09h64fhb86kw6bmhbmnpm1.ls containing all metadata,
including the offsets of regular files inside the NAR. Thus, we don't
need to read the entire NAR. (We do read the entire listing, but
that's generally pretty small. We could use a SQLite DB by borrowing
some more code from nixos-channel-scripts/file-cache.hh.)

This is primarily useful when Hydra is serving files from an S3 binary
cache, in particular when you have giant NARs. E.g. we had some 12 GiB
NARs, so accessing individuals files was pretty slow.
2017-12-07 01:07:07 +01:00
Eelco Dolstra
338f29dbd4 nix ls-{nar,store}: Return offset of files in the NAR if known
E.g.

  $ nix ls-store --json --recursive --store https://cache.nixos.org /nix/store/b0w2hafndl09h64fhb86kw6bmhbmnpm1-blender-2.79 \
    | jq .entries.bin.entries.blender.narOffset
  400
2017-12-07 01:07:07 +01:00
Daiderd Jordan
2cb8aaa610
ssh-store: fix length when writing nar
This fixes nix copy and other things that use copyStorePath.
2017-12-06 11:41:08 +01:00
Eelco Dolstra
b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Eelco Dolstra
549c3706a5
nix path-info: Show URL of NARs in binary caches 2017-11-24 18:08:50 +01:00
Eelco Dolstra
6cdaa858d0
Propagate flags like --sandbox to the daemon properly 2017-11-21 18:50:56 +01:00
Eelco Dolstra
91a1987607
signed-binary-caches -> require-sigs
Unlike signed-binary-caches (which could only be '*' or ''),
require-sigs is a proper Boolean option. The default is true.
2017-11-20 17:44:07 +01:00
Eelco Dolstra
7a2b64e55c
binary-cache-public-keys -> trusted-public-keys
The name had become a misnomer since it's not only for substitution
from binary caches, but when adding/copying any
(non-content-addressed) path to a store.
2017-11-20 17:32:34 +01:00
Eelco Dolstra
7474ac871b
nix copy: Abbreviate "daemon" 2017-11-20 15:17:11 +01:00
Eelco Dolstra
8956ae1987
Add a "profile" option to S3BinaryCacheStore
This allows specifying the AWS configuration profile to use. E.g.

  nix copy --from s3://my-cache?profile=aws-dev-account /nix/store/cf3isrlqavvd5w7rpky1fa8j9lcnlggm-...
2017-11-15 14:18:20 +01:00
Eelco Dolstra
ec5b04862b
nix sign-paths: Support binary caches 2017-11-14 18:44:05 +01:00
Eelco Dolstra
5773d667ee
nix ls-{nar,store}: Don't abort on missing files 2017-11-14 14:49:06 +01:00
Eelco Dolstra
4db0a9555e
nix ls-{nar,store} --json: Respect -R 2017-11-14 14:31:38 +01:00
Eelco Dolstra
bac8055652
nix ls-{store,nar}: Add --json flag 2017-11-14 14:23:53 +01:00
Eelco Dolstra
9d87d03331
Remove extraneous comment 2017-11-08 16:17:32 +01:00