Commit graph

3634 commits

Author SHA1 Message Date
Eelco Dolstra
1d9742b95f
<nix/buildenv.nix>: Ignore dangling symlinks
Fixes

  error: getting status of '/nix/store/j8p0vv89k1pf0cn7kmfsdcs7bshwga1i-firefox-52.7.2esr/share/icons/hicolor/48x48/apps/firefox.png': No such file or directory

https://github.com/NixOS/nix/issues/1934

Also improve error message on directory/non-directory collisions.
2018-04-05 20:04:42 +02:00
Eelco Dolstra
5c904e10eb
<nix/buildenv.nix>: Skip non-directory top-level paths
Fixes https://github.com/NixOS/nix/issues/1934 (at least the "error:
opening directory '/nix/store/...-linux-config-4.4.14': Not a
directory" issue).
2018-04-05 19:34:56 +02:00
Tuomas Tynkkynen
af86132e1a libexpr: Make unsafeGetAttrPos not crash on noPos
Currently e.g. `builtins.unsafeGetAttrPos "abort" builtins` will
eventually segfault because pos->file is an unset Symbol.

Found by afl-fuzz.
2018-04-03 15:54:42 +03:00
zimbatm
865ca2402f nix copy: add an example with S3
I couldn't find a good example how to use it with non-us-east-1 buckets.
2018-03-30 02:31:14 +01:00
Eelco Dolstra
446bb88f13
Remove reference to non-existent manpages 2018-03-29 12:29:10 +02:00
Eelco Dolstra
2b61c74922
Process --option use-case-hack properly
Fixes https://github.com/NixOS/nix/issues/2009.
2018-03-29 12:28:29 +02:00
Eelco Dolstra
9d1220a01d
ValidPathInfo::isContentAddressed(): Ensure there are no references 2018-03-29 12:27:42 +02:00
Will Dietz
74da813912 download: improve error for hash mismatch ("store mismatch")
Fixes #1905
2018-03-22 15:22:06 -05:00
Will Dietz
59b32403f2 download.cc: fix path for cached files 2018-03-22 15:05:46 -05:00
Eelco Dolstra
28eb9621cc
Fix build 2018-03-22 13:46:19 +01:00
Eelco Dolstra
dce8af59a6
Tweak 2018-03-22 13:32:18 +01:00
Eelco Dolstra
f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Shea Levy
c24e0439b2
Merge branch 'fix/execl-null-sentinel' of git://github.com/dtzWill/nix 2018-03-21 23:25:53 -04:00
Eelco Dolstra
ec91840e70 Also make the backwards compatible case of RemoteStore::addToStore run in constant memory
Just because we can.
2018-03-21 23:46:03 +01:00
Eelco Dolstra
97002b684c Make 'nix copy --to daemon' run in constant memory 2018-03-21 23:42:52 +01:00
Will Dietz
f66ae22288 shared.cc: style: prefer "nullptr" over '(char *)NULL' 2018-03-21 17:17:37 -05:00
Eelco Dolstra
11898d6a37 Make 'nix copy --to ssh://...' run in constant memory 2018-03-21 23:13:06 +01:00
Eelco Dolstra
47f7e5585b Make 'nix copy --from ssh://...' run in constant memory
For instance, this reduced the memory consumption of

  $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a

from 632 MiB to 16 MiB.
2018-03-21 23:10:40 +01:00
Eelco Dolstra
8615cfb130
LegacySSHStore: Allow overriding the path to nix-store 2018-03-21 15:10:46 +01:00
Will Dietz
02d36fdab7 execl: cast NULL sentinel to (char *), per man page and compiler warning
From exec(3):

> The list of arguments must be terminated by a null pointer, and, since these
> are variadic functions, this pointer must be cast (char *) NULL
2018-03-20 13:31:58 -05:00
Eelco Dolstra
03d8136b02
Remove unused channel-cache directory 2018-03-20 18:22:49 +01:00
Eelco Dolstra
e0c1597910
Style fix 2018-03-20 17:33:45 +01:00
Eelco Dolstra
668ac3ea2c
Make <nix/buildenv.nix> a builtin builder
This avoids sandbox annoyances.
2018-03-20 17:28:09 +01:00
Eelco Dolstra
9d40787938
Move builtinFetchurl to its own file 2018-03-20 17:01:36 +01:00
Eelco Dolstra
5df427f35b
Slight simplification 2018-03-20 16:32:59 +01:00
Eelco Dolstra
21ebf6a160
SSHMaster: Optionally pass -v to ssh 2018-03-20 15:30:55 +01:00
Eelco Dolstra
1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Will Dietz
f5a2136a40 serialise.cc: remove pessimising move
from clang6:

src/libutil/serialise.cc:189:23: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
2018-03-19 19:18:47 -05:00
Will Dietz
9b33201e72 util: rename stdout/stdin members to avoid conflicts w/standard macro
(cherry picked from commit c389a7fb617ed7bcd617efa68c6a48c00405310d)
2018-03-19 11:18:56 -05:00
Eelco Dolstra
138af2e554
Shut up signedness warning 2018-03-19 11:57:15 +01:00
Linus Heckemann
639c166647 build-remote: take extra-platforms into account 2018-03-16 22:50:57 +00:00
Linus Heckemann
637701b604 rename build-extra-platforms -> extra-platforms
also document it
2018-03-16 22:50:27 +00:00
Linus Heckemann
12913ccf45 nix-shell: allow symlinks to .drvs
This makes persistent shell environments easier to use.
2018-03-16 22:15:24 +00:00
Eelco Dolstra
48662d151b
Reduce substitution memory consumption
copyStorePath() now pipes the output of srcStore->narFromPath()
directly into dstStore->addToStore(). The sink used by the former is
converted into a source usable by the latter using
boost::coroutine2. This is based on [1].

This reduces the maximum resident size of

  $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs

from 418592 KiB to 53416 KiB. (The previous commit also reduced the
runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will
be to download files into a Sink.

[1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18

Issue #1969.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
3e6b194d78
decompress(): Use a Source and Sink
This allows decompression to happen in O(1) memory.
2018-03-16 20:35:59 +01:00
Eelco Dolstra
64441f0551
Merge pull request #1939 from dezgeg/lexer-fix
libexpr: Recognize newline in more places in lexer
2018-03-16 12:14:58 +01:00
Eelco Dolstra
eb75bc5afb
Don't retry on CURLE_SSL_CACERT_BADFILE
The certificates won't get any better if we retry.
2018-03-16 12:03:18 +01:00
Eelco Dolstra
4868721506
Filter ANSI colors when not writing to a terminal
Fixes https://github.com/NixOS/nixpkgs/issues/37114.
2018-03-15 16:08:07 +01:00
Eelco Dolstra
7b8914825a
Merge pull request #1965 from masaeedu/master
Wrap thread local in function for Cygwin
2018-03-15 14:56:21 +01:00
Shea Levy
cfdbfa6b2c
Catch more possible instances of passing NULL to memcpy.
Actually fixes #1976.
2018-03-14 23:53:43 -04:00
Shea Levy
d25d9f7cec
Merge branch 'fix/errno-sighandler' of git://github.com/dtzWill/nix 2018-03-14 23:45:35 -04:00
Shea Levy
e2088febf3
concatLists: Don't pass NULL pointers to memcpy.
This is UB, even if the size is 0. See #1976.

Fixes #1976.
2018-03-14 23:44:02 -04:00
Will Dietz
d98755b0c2 nix-daemon: preserve errno in signal handler (thanks tsan) 2018-03-14 21:39:46 -05:00
Eelco Dolstra
55aa622fb1
Fix compatibility with latest boost::format 2018-03-14 19:25:09 +01:00
Eelco Dolstra
ca14b14200
Use boost::format from the boost package
Note that this only requires headers from boost so it doesn't add a
runtime dependency.

Also, use Nixpkgs 18.03.
2018-03-14 19:24:04 +01:00
Shea Levy
56f2ed0081
Merge pull request #1906 from dtzWill/fix/nix-search
nix search: tests and fix #1893 and part of #1892
2018-03-13 18:43:47 -06:00
Eelco Dolstra
7afdc8d4a1
build-remote: Don't substitute the build result 2018-03-13 17:56:51 +01:00
Eelco Dolstra
6e60141a80
Typo 2018-03-13 17:48:09 +01:00
Guillaume Maudoux
80735c4cc9
fetchGit: Fix debug message 2018-03-13 10:28:23 +01:00
Asad Saeeduddin
be54f4a0b6 Wrap thread local in function for Cygwin
Fixes #1826. See #1352 for a previous instance of a similar change.
2018-03-12 00:56:41 -04:00
Eelco Dolstra
24b739817f
Fix double free in Store::queryPathInfo()
It was holding on to a Value* (i.e. a std::shared_ptr<ValidPathInfo>*)
outside of the pathInfoCache lock, so the std::shared_ptr could be
destroyed between the release of the lock and the decrement of the
std::shared_ptr refcount. This can happen if more than
'path-info-cache-size' paths are added in the meantime, *or* if
clearPathInfoCache() is called. The hydra-queue-runner queue monitor
thread periodically calls the later, so is likely to trigger a crash.

Fixes https://github.com/NixOS/hydra/issues/542.
2018-03-09 14:19:51 +01:00
Tim Engler
383afab9ad Modified MakeBinOp to no longer produce its name using concatenation and "##".
Doing so prevents emacs tags from working, as well as makes the code extremely
confusing for a newbie.

In the prior state, if someone wants to find the definition of "ExprApp" for
example, a grep through the code reveals nothing. Since the definition could be
hiding in numerous ".h" files, it's really difficult to find. This personally
took me several hours to figure out.
2018-03-09 03:16:33 +00:00
Eelco Dolstra
16551f54c9
Fix #1957 2018-03-07 11:40:50 +01:00
Eelco Dolstra
4452f6e855
Merge pull request #1955 from dtzWill/fix/rand-random
rand() -> random(), since we use srandom().
2018-03-07 11:00:08 +01:00
Will Dietz
56253bb08f rand() -> random(), since we use srandom().
rand() requires we call srand() instead,
but might as well use random().
2018-03-06 17:44:25 -06:00
Will Dietz
e917c05628 logging: Don't throw from Activity destructor
Move definition of destructor to logging.cc for
access to util.hh's ignoreException.
2018-03-06 10:42:58 -06:00
Shea Levy
088ef81759
ssh-ng: Don't forward options to the daemon.
This can be iterated on and currently leaves out settings we know we
want to forward, but it fixes #1713 and fixes #1935 and isn't
fundamentally broken like the status quo. Future changes are suggested
in a comment.
2018-03-05 07:42:15 -05:00
Will Dietz
6b9a03f5d8 hoist vector out of loop just in case 2018-03-02 10:52:05 -06:00
Will Dietz
c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Shea Levy
3748a0ca1e
Merge branch 'improve-search-algorithm' of git://github.com/Ma27/nix 2018-03-02 11:41:08 -05:00
Shea Levy
aa8bbbf69d
Merge branch 'write-failure-fixes' of git://github.com/lheckemann/nix 2018-03-02 10:59:59 -05:00
Tuomas Tynkkynen
a0e38c16bc libexpr: Recognize newline in more places in lexer
Flex's regexes have an annoying feature: the dot matches everything
except a newline. This causes problems for expressions like:

"${0}\
"

where the backslash-newline combination matches this rule instead of the
intended one mentioned in the comment:

    <STRING>\$|\\|\$\\ {
                    /* This can only occur when we reach EOF, otherwise the above
                    (...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered.
                    This is technically invalid, but we leave the problem to the
                    parser who fails with exact location. */
                    return STR;
                }
However, the parser actually accepts the resulting token sequence
('"' DOLLAR_CURLY 0 '}' STR '"'), which is a problem because the lexer
rule didn't assign anything to yylval. Ultimately this leads to a crash
when dereferencing a NULL pointer in ExprConcatStrings::bindVars().

The fix does change the syntax of the language in some corner cases
but I think it's only turning previously invalid (or crashing) syntax
to valid syntax. E.g.

"a\
b"

and

''a''\
b''

were previously syntax errors but now both result in "a\nb".

Found by afl-fuzz.
2018-03-02 17:30:48 +02:00
Eelco Dolstra
939cf4cceb
Fix error message 2018-03-02 14:32:00 +01:00
Will Dietz
e89d02bf03 fetchGit: use "HEAD" as default ref 2018-02-28 16:34:34 -06:00
Shea Levy
14ca85688c
Actually fix nixDataDir in non-canonical path 2018-02-28 06:19:40 -05:00
Linus Heckemann
919c3c20b3 seccomp: add 32-bit ARM on aarch64-linux
This allows building armv[67]l-linux derivations on compatible aarch64
machines. Failure to add the architecture may result from missing
hardware support, in which case we can't run 32-bit binaries and don't
need to restrict them with seccomp anyway,
2018-02-27 00:32:15 +00:00
Linus Heckemann
eb03a296c1 Add build-extra-platforms setting
This allows specifying additional systems that a machine is able to
build for. This may apply on some armv7-capable aarch64 processors, or
on systems using qemu-user with binfmt-misc to support transparent
execution of foreign-arch programs.

This removes the previous hard-coded assumptions about which systems are
ABI-compatible with which other systems, and instead relies on the user
to specify any additional platforms that they have ensured compatibility
for and wish to build for locally.

NixOS should probably add i686-linux on x86_64-linux systems for this
setting by default.
2018-02-27 00:27:58 +00:00
Tuomas Tynkkynen
77e9e1ed91 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.

First attempt of this was reverted in e2d71bd186 because it caused
another infinite loop, which is fixed now and a test added.
2018-02-26 19:49:13 +02:00
Eelco Dolstra
24ec750003
nix run: Fix segfault on macOS
Note that clearenv() is not available on macOS.

Fixes #1907.
2018-02-26 18:29:40 +01:00
Will Dietz
009cf9cd23 nix search: explicitly handle empty search string, fixes #1893
This is important since this is given as an example.
Other patterns containing "empty search string" will still
be handled differently on different platforms ("asdf|")
but that's less of an issue.
2018-02-25 16:40:56 -06:00
Will Dietz
3cac8734ac nix search: fix bug where we wrote to cache when shouldn't, breaking
This is exposed by the tests added previously,
and resolves the error reported in #1892:
"expected JSON value".
2018-02-25 16:40:05 -06:00
Eelco Dolstra
64e486ab63
Fix downloadCached() with a chroot store
E.g.

  nix run --store ~/my-nix -f channel:nixos-17.03 hello -c hello

This problem was mentioned in #1897.
2018-02-23 13:00:42 +01:00
Eelco Dolstra
179b896acb
Merge branch 'data-dir-non-canon' of https://github.com/shlevy/nix 2018-02-22 14:20:43 +01:00
Eelco Dolstra
e8d53bfdc9
Revert "Enable sandbox builds on Linux by default"
This reverts commit ddc58e7896.

https://hydra.nixos.org/eval/1435322
2018-02-22 14:20:07 +01:00
Shea Levy
ddbcd01c83
Fix restricted mode when installing in non-canonical data dir 2018-02-22 07:18:14 -05:00
Eelco Dolstra
ddc58e7896
Enable sandbox builds on Linux by default
The overhead of sandbox builds is a problem on NixOS (since building a
NixOS configuration involves a lot of small derivations) but not for
typical non-NixOS use cases. So outside of NixOS we can enable it.

Issue #179.
2018-02-22 12:27:25 +01:00
Eelco Dolstra
de4c03d201
Merge branch 'fix/dry-run-partially' of https://github.com/dtzWill/nix 2018-02-22 12:18:20 +01:00
Eelco Dolstra
0d54671b7b
Manual: Update chapter on remote builds
Alos add a command "nix ping-store" to make it easier to see if Nix
can connect to a remote builder (e.g. 'nix ping-store --store
ssh://mac').
2018-02-21 16:24:26 +01:00
Eelco Dolstra
e2d71bd186
Revert "libexpr: Fix prim_replaceStrings() to work on an empty source string"
This reverts commit 4ea9707591.

It causes an infinite loop in Nixpkgs evaluation,
e.g. "nix-instantiate -A hello" hung.

PR #1886.
2018-02-21 15:35:28 +01:00
Shea Levy
a6c497f526
Merge branch 'nix-copy' of git://github.com/Mic92/nix-1 2018-02-20 21:05:17 -05:00
Jörg Thalheim
fa7fd76c5e nix-copy: fix examples
maybe a left-over from nix-store -r ?
2018-02-21 01:13:46 +00:00
Tuomas Tynkkynen
546f98dace libutil: Fix invalid assert on decoding base64 hashes
The assertion is broken because there is no one-to-one mapping from
length of a base64 string to the length of the output.

E.g.

"1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9q=" results in a 32-byte output.
"1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9qy" results in a 33-byte output.

To reproduce, evaluate:

builtins.derivationStrict {
    name = "0";
    builder = "0";
    system = "0";
    outputHashAlgo = "sha256";
    outputHash = "1q69lz7Empb06nzfkj651413n9icx0njmyr3xzq1j9qy";
}

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen
4ea9707591 libexpr: Fix prim_replaceStrings() to work on an empty source string
Otherwise, running e.g.

nix-instantiate --eval -E --strict 'builtins.replaceStrings [""] ["X"] "abc"'

would just hang in an infinite loop.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen
1d0e42879f libutil: Fix infinite loop in filterANSIEscapes on '\r'
E.g. nix-instantiate --eval -E 'abort "\r"' hangs.

Found by afl-fuzz.
2018-02-19 23:20:26 +02:00
Tuomas Tynkkynen
056d28a601 libexpr: Don't create lots of temporary strings in Bindings::lexicographicOrder
Avoids ~180,000 string temporaries created when evaluating a headless
NixOS system.
2018-02-19 22:47:25 +02:00
Eelco Dolstra
d4e93532e2
Fix incorrect (and unnecessary) format string
https://hydra.nixos.org/eval/1434547#tabs-now-fail
2018-02-19 20:46:39 +01:00
Eelco Dolstra
70eb64147e
Update release notes
Also add some examples to nix --help.
2018-02-19 20:38:53 +01:00
Eelco Dolstra
623fcb071e
Merge pull request #1882 from shlevy/no-seccomp-no-filterSyscalls
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
2018-02-19 17:39:46 +01:00
Shea Levy
e1eb63a586
Merge branch 'perf-fixes' of git://github.com/dezgeg/nix 2018-02-19 10:11:52 -05:00
Shea Levy
e59a8a63e1
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
Running Nix with build users without seccomp on Linux is dangerous,
and administrators should very explicitly opt-in to it.
2018-02-19 09:56:24 -05:00
Eelco Dolstra
ed73d40c3b
Config::handleUnknownSettings(): Remove unused 'fatal' argument 2018-02-19 14:00:34 +01:00
Eelco Dolstra
75a1d96cfd
Merge branch 'register-settings' of https://github.com/shlevy/nix 2018-02-19 13:58:34 +01:00
Eelco Dolstra
7fe5910bf8
Merge pull request #1857 from dtzWill/fix/check-for-lzma-mt
configure.ac: check if lzma has MT support, fix deb build/etc.
2018-02-19 13:06:14 +01:00
Shea Levy
690ac7c90b
configure: Add a flag to disable seccomp.
This is needed for new arches where libseccomp support doesn't exist
yet.

Fixes #1878.
2018-02-18 02:35:01 -05:00
Tuomas Tynkkynen
37264ed0ad libexpr: Avoid an unnecessary string copy in prim_derivationStrict 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
66eeff3345 libexpr: Remove unnecessary drvName assignment in prim_derivationStrict
drvName is already assigned to the same value right at the start of the
function.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
7e0360504d libexpr: Optimize prim_derivationStrict by using more symbol comparisons 2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
0845cdf944 libexpr: Rely on Boehm returning zeroed memory in EvalState::allocEnv()
Boehm guarantees that memory returned by GC_malloc() is zeroed, so take
advantage of that.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
b8bed7da14 libexpr: Optimize prim_attrNames a bit
Instead of having lexicographicOrder() create a temporary sorted array
of Attr*:s and copying attr names from that, copy the attr names
first and then sort that.
2018-02-17 16:54:21 +02:00
Tuomas Tynkkynen
f67a7007a2 libexpr: Pre-reserve space in string in unescapeStr()
Avoids some malloc() traffic.
2018-02-16 04:39:43 +02:00