Since SubstitutionGoal::finished() in build.cc computes the hash
anyway, we can prevent the inefficiency of computing the hash twice by
letting the substituter tell Nix about the expected hash, which can
then verify it.
The generated attrset has drvPath and outPath with the right string context, type 'derivation', outputName with
the right name, all with a list of outputs, and an attribute for each output.
I see three uses for this (though certainly there may be more):
* Using derivations generated by something besides nix-instantiate (e.g. guix)
* Allowing packages provided by channels to be used in nix expressions. If a channel installed a valid deriver
for each package it provides into the store, then those could be imported and used as dependencies or installed
in environment.systemPackages, for example.
* Enable hydra to be consistent in how it treats inputs that are outputs of another build. Right now, if an
input is passed as an argument to the job, it is passed as a derivation, but if it is accessed via NIX_PATH
(i.e. through the <> syntax), then it is a path that can be imported. This is problematic because the build
being depended upon may have been built with non-obvious arguments passed to its jobset file. With this
feature, hydra can just set the name of that input to the path to its drv file in NIX_PATH
Incremental optimisation requires creating links in /nix/store/.links
to all files in the store. However, this means that if we delete a
store path, no files are actually deleted because links in
/nix/store/.links still exists. So we need to check /nix/store/.links
for files with a link count of 1 and delete them.
optimiseStore() now creates persistent, content-addressed hard links
in /nix/store/.links. For instance, if it encounters a file P with
hash H, it will create a hard link
P' = /nix/store/.link/<H>
to P if P' doesn't already exist; if P' exist, then P is replaced by a
hard link to P'. This is better than the previous in-memory map,
because it had the tendency to unnecessarily replace hard links with a
hard link to whatever happened to be the first file with a given hash
it encountered. It also allows on-the-fly, incremental optimisation.
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5"). (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.) However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory. queryPathFromHashPart() prevents this by doing a cheap
database lookup.
queryValidPaths() combines multiple calls to isValidPath() in one.
This matters when using the Nix daemon because it reduces latency.
For instance, on "nix-env -qas \*" it reduces execution time from 5.7s
to 4.7s (which is indistinguishable from the non-daemon case).
Instead make a single call to querySubstitutablePathInfo() per
derivation output. This is faster and prevents having to implement
the "have" function in the binary cache substituter.
Getting substitute information using the binary cache substituter has
non-trivial latency overhead. A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these. If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.
To fix this problem, the binary cache substituter can now perform
requests in parallel. This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism. (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)
For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s. It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9. So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.
Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit. Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
In a private PID namespace, processes have PIDs that are separate from
the rest of the system. The initial child gets PID 1. Processes in
the chroot cannot see processes outside of the chroot. This improves
isolation between builds. However, processes on the outside can see
processes in the chroot and send signals to them (if they have
appropriate rights).
Since the builder gets PID 1, it serves as the reaper for zombies in
the chroot. This might turn out to be a problem. In that case we'll
need to have a small PID 1 process that sits in a loop calling wait().
In chroot builds, set the host name to "localhost" and the domain name
to "(none)" (the latter being the kernel's default). This improves
determinism a bit further.
P.S. I have to idea what UTS stands for.
This improves isolation a bit further, and it's just one extra flag in
the unshare() call.
P.S. It would be very cool to use CLONE_NEWPID (to put the builder in
a private PID namespace) as well, but that's slightly more risky since
having a builder start as PID 1 may cause problems.
On Linux it's possible to run a process in its own network namespace,
meaning that it gets its own set of network interfaces, disjunct from
the rest of the system. We use this to completely remove network
access to chroot builds, except that they get a private loopback
interface. This means that:
- Builders cannot connect to the outside network or to other processes
on the same machine, except processes within the same build.
- Vice versa, other processes cannot connect to processes in a chroot
build, and open ports/connections do not show up in "netstat".
- If two concurrent builders try to listen on the same port (e.g. as
part of a test), they no longer conflict with each other.
This was inspired by the "PrivateNetwork" flag in systemd.
Systemd can start the Nix daemon on demand when the Nix daemon socket
is first accessed. This is signalled through the LISTEN_FDS
environment variable, so all we need to do is check for that and then
use file descriptor 3 as the listen socket instead of creating one
ourselves.
We can't open a SQLite database if the disk is full. Since this
prevents the garbage collector from running when it's most needed, we
reserve some dummy space that we can free just before doing a garbage
collection. This actually revives some old code from the Berkeley DB
days.
Fixes#27.
There is a race condition when doing parallel builds with chroots and
the immutable bit enabled. One process may call makeImmutable()
before the other has called link(), in which case link() will fail
with EPERM. We could retry or wrap the operation in a lock, but since
this condition is rare and I'm lazy, we just use the existing copy
fallback.
Fixes#9.
This should fix rare Hydra errors of the form:
error: symlinking `/nix/var/nix/gcroots/per-user/hydra/hydra-roots/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795' to `/nix/store/7sfhs5fdmjxm8sqgcpd0pgcsmz1kq0l0-nixos-iso-0.1pre33785-33795': File exists
Setting the UNAME26 personality causes "uname" to return "2.6.x",
regardless of the kernel version. This improves determinism in
a few misbehaved packages.
Make the garbage collector more concurrent by deleting valid paths
outside the region where we're holding the global GC lock. This
should greatly reduce the time during which new builds are blocked,
since the deletion accounts for the vast majority of the time spent in
the GC.
To ensure that this is safe, the valid paths are invalidated and
renamed to some arbitrary path while we're holding the lock. This
ensures that we when we finally delete the path, it's not a (newly)
valid or locked path.
Nix now requires SQLite and bzip2 to be pre-installed. SQLite is
detected using pkg-config. We required DBD::SQLite anyway, so
depending on SQLite is not a big problem.
The --with-bzip2, --with-openssl and --with-sqlite flags are gone.
By moving the destructor object to libstore.so, it's also run when
download-using-manifests and nix-prefetch-url exit. This prevents
them from cluttering /nix/var/nix/temproots with stale files.
Not all SQLite builds have the function sqlite3_table_column_metadata.
We were only using it in a schema upgrade check for compatibility with
databases that were probably never seen in the wild. So remove it.
The variable ‘useChroot’ was not initialised properly. This caused
random failures if using the build hook. Seen on Mac OS X 10.7 with Clang.
Thanks to KolibriFX for finding this :-)
Chroots are initialised by hard-linking inputs from the Nix store to
the chroot. This doesn't work if the input has its immutable bit set,
because it's forbidden to create hard links to immutable files. So
temporarily clear the immutable bit when creating and destroying the
chroot.
Note that making regular files in the Nix store immutable isn't very
reliable, since the bit can easily become cleared: for instance, if we
run the garbage collector after running ‘nix-store --optimise’. So
maybe we should only make directories immutable.
I was bitten one time too many by Python modifying the Nix store by
creating *.pyc files when run as root. On Linux, we can prevent this
by setting the immutable bit on files and directories (as in ‘chattr
+i’). This isn't supported by all filesystems, so it's not an error
if setting the bit fails. The immutable bit is cleared by the garbage
collector before deleting a path. The only tricky aspect is in
optimiseStore(), since it's forbidden to create hard links to an
immutable file. Thus optimiseStore() temporarily clears the immutable
bit before creating the link.
environment of the given derivation in a format that can be sourced
by the shell, e.g.
$ eval "$(nix-store --print-env $(nix-instantiate /etc/nixos/nixpkgs -A pkg))"
$ NIX_BUILD_TOP=/tmp
$ source $stdenv/setup
This is especially useful to reproduce the environment used to build
a package outside of its builder for development purposes.
TODO: add a nix-build option to do the above and fetch the
dependencies of the derivation as well.
other simplifications.
* Use <nix/...> to locate the corepkgs. This allows them to be
overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
to create a temporary file.
unreachable paths. This matters when using --max-freed etc.:
unreachable paths could become reachable again, so it's nicer to
keep them if there is "real" garbage to be deleted. Also, don't use
readDirectory() but read the Nix store and delete invalid paths in
parallel. This reduces GC latency on very large Nix stores.
* Buffer the HashSink. This speeds up hashing a bit because it
prevents lots of calls to the hash update functions (e.g. nix-hash
went from 9.3s to 8.7s of user time on the closure of my
/var/run/current-system).
significantly cuts down the number of syscalls (e.g., for "nix-store
-qR /var/run/current-system" via the daemon, it reduced the number
of syscalls in the client from 29134 to 4766 and in the daemon from
44266 to 20666).
daemon (which is an error), print a nicer error message than
"Connection reset by peer" or "broken pipe".
* In the daemon, log errors that occur during request parameter
processing.
‘nix-store --export’.
* Add a Perl module that provides the functionality of
‘nix-copy-closure --to’. This is used by build-remote.pl so it no
longer needs to start a separate nix-copy-closure process. Also, it
uses the Perl API to do the export, so it doesn't need to start a
separate nix-store process either. As a result, nix-copy-closure
and build-remote.pl should no longer fail on very large closures due
to an "Argument list too long" error. (Note that having very many
dependencies in a single derivation can still fail because the
environment can become too large. Can't be helped though.)
libstore so that the Perl bindings can use it as well. It's vital
that the Perl bindings use the configuration file, because otherwise
nix-copy-closure will fail with a ‘database locked’ message if the
value of ‘use-sqlite-wal’ is changed from the default.
For each output, this adds a corresponding attribute to the derivation that is
the same as the derivation except for outPath, which is set to the path specific
to that output. Additionally, an "all" attribute is added that is a list of all
of the output derivations. This has to be done outside of derivationStrict as
each output is itself a derivation that contains itself (and all other outputs)
as an attribute. The derivation itself is equivalent to the first output in the
outputs list (or "out" if that list isn't set).
the contents of any of the given store paths have been modified.
E.g.
$ nix-store --verify-path $(nix-store -qR /var/run/current-system)
path `/nix/store/m2smyiwbxidlprfxfz4rjlvz2c3mg58y-etc' was modified! expected hash `fc87e271c5fdf179b47939b08ad13440493805584b35e3014109d04d8436e7b8', got `20f1a47281b3c0cbe299ce47ad5ca7340b20ab34246426915fce0ee9116483aa'
All paths are checked; the exit code is 1 if any path has been
modified, 0 otherwise.
This should also fix:
nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.
which was caused by hashDerivationModulo() calling the ‘store’
object (during store upgrades) before openStore() assigned it.
prevents files from being evaluated and stored as values multiple
times. For instance, evaluation of the ‘system’ attribute in NixOS
causes ‘nixpkgs/pkgs/lib/lists.nix’ to be evaluated 2019 times.
Caching gives a modest speedup and a decent memory footprint
reduction (e.g., from 1.44s to 1.28s, and from 81 MiB to 59 MiB with
GC_INITIAL_HEAP_SIZE=100000 on my system).
directory
/home/eelco/src/stdenv-updates
that you want to use as the directory for import such as
with (import <nixpkgs> { });
then you can say
$ nix-build -I nixpkgs=/home/eelco/src/stdenv-updates
brackets, e.g.
import <nixpkgs/pkgs/lib>
are resolved by looking them up relative to the elements listed in
the search path. This allows us to get rid of hacks like
import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"
The search path can be specified through the ‘-I’ command-line flag
and through the colon-separated ‘NIX_PATH’ environment variable,
e.g.,
$ nix-build -I /etc/nixos ...
If a file is not found in the search path, an error message is
lazily thrown.
derivations added to the store by clients have "correct" output
paths (meaning that the output paths are computed by hashing the
derivation according to a certain algorithm). This means that a
malicious user could craft a special .drv file to build *any*
desired path in the store with any desired contents (so long as the
path doesn't already exist). Then the attacker just needs to wait
for a victim to come along and install the compromised path.
For instance, if Alice (the attacker) knows that the latest Firefox
derivation in Nixpkgs produces the path
/nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1
then (provided this path doesn't already exist) she can craft a .drv
file that creates that path (i.e., has it as one of its outputs),
add it to the store using "nix-store --add", and build it with
"nix-store -r". So the fake .drv could write a Trojan to the
Firefox path. Then, if user Bob (the victim) comes along and does
$ nix-env -i firefox
$ firefox
he executes the Trojan injected by Alice.
The fix is to have the Nix daemon verify that derivation outputs are
correct (in addValidPath()). This required some refactoring to move
the hash computation code to libstore.
by setting the ‘outputs’ attribute. For example:
stdenv.mkDerivation {
name = "aterm-2.5";
src = ...;
outputs = [ "out" "tools" "dev" ];
configureFlags = "--bindir=$(tools)/bin --includedir=$(dev)/include";
}
This derivation creates three outputs, named like this:
/nix/store/gcnqgllbh01p3d448q8q6pzn2nc2gpyl-aterm-2.5
/nix/store/gjf1sgirwfnrlr0bdxyrwzpw2r304j02-aterm-2.5-tools
/nix/store/hp6108bqfgxvza25nnxfs7kj88xi2vdx-aterm-2.5-dev
That is, the symbolic name of the output is suffixed to the store
path (except for the ‘out’ output). Each path is passed to the
builder through the corresponding environment variable, e.g.,
${tools}.
The main reason for multiple outputs is to allow parts of a package
to be distributed and garbage-collected separately. For instance,
most packages depend on Glibc for its libraries, but don't need its
header files. If these are separated into different store paths,
then a package that depends on the Glibc libraries only causes the
libraries and not the headers to be downloaded.
The main problem with multiple outputs is that if one output exists
while the others have been garbage-collected (or never downloaded in
the first place), and we want to rebuild the other outputs, then
this isn't possible because we can't clobber a valid output (it
might be in active use). This currently gives an error message
like:
error: derivation `/nix/store/1s9zw4c8qydpjyrayxamx2z7zzp5pcgh-aterm-2.5.drv' is blocked by its output paths
There are two solutions: 1) Do the build in a chroot. Then we don't
need to overwrite the existing path. 2) Use hash rewriting (see the
ASE-2005 paper). Scary but it should work.
This is not finished yet. There is not yet an easy way to refer to
non-default outputs in Nix expressions. Also, mutually recursive
outputs aren't detected yet and cause the garbage collector to
crash.
write ‘attrs ? a.b’ to test whether ‘attrs’ has an attribute ‘a’
containing an attribute ‘b’. This is more convenient than ‘attrs ?
a && attrs.a ? b’.
Slight change in the semantics: it's no longer an error if the
left-hand side of ‘?’ is not an attribute set. In that case it just
returns false. So, ‘null ? foo’ no longer throws an error.
checked too soon whether substitutes are available. That is, it did
so for every available package, rather than those matching installed
packages. This was very slow and subject to assertion failures. So
do the check much later. Idem for `nix-env -qab' and `nix-env -ib'.
little RAM. Even if the memory isn't actually used, it can cause
problems with the overcommit heuristics in the kernel. So use a VM
space of 25% of RAM, up to 384 MB.
while checking the contents, since this operation can take a very
long time to finish. Also, fill in missing narSize fields in the DB
while doing this.
even with a very long busy timeout, because SQLITE_BUSY is also
returned to resolve deadlocks. This should get rid of random
"database is locked" errors. This is kind of hard to test though.
* Fix a horrible bug in deleteFromStore(): deletePathWrapped() should
be called after committing the transaction, not before, because the
commit might not succeed.
tree). This saves a lot of memory. The vector should be sorted so
that names can be looked up using binary search, but this is not the
case yet. (Surprisingly, looking up attributes using linear search
doesn't have a big impact on performance.)
Memory consumption for
$ nix-instantiate /etc/nixos/nixos/tests -A bittorrent.test --readonly-mode
on x86_64-linux with GC enabled is now 185 MiB (compared to 946
MiB on the trunk).
improves GC effectiveness a bit more (because a live value doesn't
keep other values in the environment plus the parent environments
alive), and removes the need for copy nodes.
a pointer to a Value, rather than the Value directly. This improves
the effectiveness of garbage collection a lot: if the Value is
stored inside the set directly, then any live pointer to the Value
causes all other attributes in the set to be live as well.
race with other processes that add new referrers to a path,
resulting in the garbage collector crashing with "foreign key
constraint failed". (Nix/4)
* Make --gc --print-dead etc. interruptible.
because it defines _FILE_OFFSET_BITS. Without this, on
OpenSolaris the system headers define it to be 32, and then
the 32-bit stat() ends up being called with a 64-bit "struct
stat", or vice versa.
This also ensures that we get 64-bit file sizes everywhere.
* Remove the redundant call to stat() in parseExprFromFile().
The file cannot be a symlink because that's the exit condition
of the loop before.
exception handler, otherwise throw an exception. We need to ignore
write errors in exception handlers to ensure that cleanup code runs
to completion if the other side of stderr has been closed
unexpectedly.
* If a path has disappeared, check its referrers first, and don't try
to invalidate paths that have valid referrers. Otherwise we get a
foreign key constraint violation.
* Read the whole Nix store directory instead of statting each valid
path, which is slower.
* Acquire the global GC lock.
hook script proper, and the stdout/stderr of the builder. Only the
latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
it requires a certain feature on the build machine, e.g.
requiredSystemFeatures = [ "kvm" ];
We need this in Hydra to make sure that builds that require KVM
support are forwarded to machines that have KVM support. Probably
this should also be enforced for local builds.
the hook every time we want to ask whether we can run a remote build
(which can be very often), we now reuse a hook process for answering
those queries until it accepts a build. So if there are N
derivations to be built, at most N hooks will be started.
faster than the old mode when fsyncs are enabled, because it only
performs an fsync() when doing a checkpoint, rather than at every
commit. Some timings for doing a "nix-instantiate /etc/nixos/nixos
-A system" after modifying the stdenv setup script:
42.5s - SQLite 3.6.23 with truncate mode and fsync
3.4s - SQLite 3.6.23 with truncate mode and no fsync
32.1s - SQLite 3.7.0 with truncate mode and fsync
16.8s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
every 1000 pages
8.3s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
every 8192 pages
1.7s - SQLite 3.7.0 with WAL mode and no fsync
The default is now to use WAL mode with fsyncs. Because WAL doesn't
work on remote filesystems such as NFS (as it uses shared memory),
truncate mode can be re-enabled by setting the "use-sqlite-wal"
option to false.
using the build hook mechanism, by setting the derivation attribute
"preferLocalBuild" to true. This has a few use cases:
- The user environment builder. Since it just creates a bunch of
symlinks without much computation, there is no reason to do it
remotely. In fact, doing it remotely requires the entire closure
of the user environment to be copied to the remote machine, which
is extremely wasteful.
- `fetchurl'. Performing the download on a remote machine and then
copying it to the local machine involves twice as much network
traffic as performing the download locally, and doesn't save any
CPU cycles on the local machine.
instance) "nix-env -i wine" works on x86_64-linux, even though Wine
is built on i686-linux. In the event that there are multiple
matching derivations, prefer those built for the current system.
An "using namespace std" was added locally in those functions that refer to
names from <cstring>. That is not pretty, but it's a very portable solution,
because strcpy() and friends will be found in both the 'std' and in the global
namespace.
This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.
The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
doesn't work because the garbage collector doesn't actually look at
locks. So r22253 was stupid. Use addTempRoot() instead. Also,
locking the temporary directory in exportPath() was silly because it
isn't even in the store.
errors with position info.
* For all positions, use the position of the first character of the
first token, rather than the last character of the first token plus
one.
check' now succeeds :-)
* An attribute set such as `{ foo = { enable = true; };
foo.port = 23; }' now parses. It was previously rejected, but I'm
too lazy to implement the check. (The only reason to reject it is
that the reverse, `{ foo.port = 23; foo = { enable = true; }; }', is
rejected, which is kind of ugly.)
values. This improves sharing and gives another speed up.
Evaluation of the NixOS system attribute is now almost 7 times
faster than the old evaluator.
use site, allowing environments to be stores as vectors of values
rather than maps. This should speed up evaluation and reduce the
number of allocations.
precedence, i.e. `with {x=1;}; with {x=2;}; x' evaluates to 2'.
This has a simpler implementation and seems more natural. There
doesn't seem to be any code in Nixpkgs or NixOS that relies on the
old behaviour.
efficiently. The symbol table ensures that there is only one copy
of each symbol, thus allowing symbols to be compared efficiently
using a pointer equality test.
This fixes a regression introduced in r20882 ("Add source location
information to the XML output.").
* src/libexpr/expr-to-xml.cc (nix::printTermAsXML): Dereference the
attribute RHS from "drvPath" and "outPath".
then the blackhole has to be removed to ensure that repeated
evaluation of the same value gives an assertion failure again rather
than an "infinite recursion" error.
that there are some places in Nixpkgs (php_configurable /
composableDerivation, it seems) that call `derivation' with
incorrect arguments (namely, the `name' attribute missing) but get
away with it because of laziness.
* Removed exprToString and stringToExpr because there is no ATerm
representation to work on anymore (and exposing the internals of the
evaluator like this is not a good idea anyway).
* src/libexpr/expr-to-xml.cc (nix::showAttrs): Add `location'
parameter. Provide location XML attributes when it's true. Update
callers.
(nix::printTermAsXML): Likewise.
* src/libexpr/expr-to-xml.hh (nix::printTermAsXML): Update prototype;
have `location' default to `false'.
* src/nix-instantiate/nix-instantiate.cc (printResult, processExpr): Add
`location' parameter; update callers.
(run): Add support for `--no-location'.
* src/nix-instantiate/help.txt: Update accordingly.
* tests/lang.sh: Invoke `nix-instantiate' with `--no-location' for the
XML tests.
* tests/lang/eval-okay-toxml.exp, tests/lang/eval-okay-to-xml.nix: New
files.
* src/libexpr/expr-to-xml.cc (nix::showAttrs): Dereference the attribute
RHS. Add "path", "line", and "column" XML attributes to the node when
source location information is available.
(nix::printTermAsXML): Likewise for functions.
allowed. So `name1@name2', `{attrs1}@{attrs2}' and so on are now no
longer legal. This is no big loss because they were not useful
anyway.
This also changes the output of builtins.toXML for @-patterns
slightly.
changed. This prevents corrupt paths from spreading to other
machines. Note that checking the hash is cheap because we're
hashing anyway (because of the --sign feature).
to make the Refs table more space-efficient. For instance, this
reduces the size of the database on my laptop from 93 MiB to 18
MiB. (It was 72 MiB with the old schema on an ext3 disk with a 1
KiB block size.)
This prevents remote builders from being killed by the
`max-silent-time' inactivity monitor while they are waiting for a
long garbage collection to finish. This happens fairly often in the
Hydra build farm.
_FILE_OFFSET_BITS=64. Without it, functions like stat() fail on
large file sizes. This happened with a Nix store on squashfs:
$ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null
error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type
$ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds
File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds'
Size: 0 Blocks: 36028797018963968 IO Block: 1024 regular empty file
(This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix
to fail.)
complete set of live and dead paths before starting the actual
deletion, but determines liveness on demand. I.e. for any path in
the store, it first tries to delete all the referrers, and then the
path itself. This means that the collector can start deleting paths
almost immediately.
(Linux) machines no longer maintain the atime because it's too
expensive, and on the machines where --use-atime is useful (like the
buildfarm), reading the atimes on the entire Nix store takes way too
much time to make it practical.
intersectAttrs returns the (right-biased) intersection between two
attribute sets, e.g. every attribute from the second set that also
exists in the first. functionArgs returns the set of attributes
expected by a function.
The main goal of these is to allow the elimination of most of
all-packages.nix. Most package instantiations in all-packages.nix
have this form:
foo = import ./foo.nix {
inherit a b c;
};
With intersectAttrs and functionArgs, this can be written as:
foo = callPackage (import ./foo.nix) { };
where
callPackage = f: args:
f ((builtins.intersectAttrs (builtins.functionArgs f) pkgs) // args);
I.e., foo.nix is called with all attributes from "pkgs" that it
actually needs (e.g., pkgs.a, pkgs.b and pkgs.c). (callPackage can
do any other generic package-level stuff we might want, such as
applying makeOverridable.) Of course, the automatically supplied
arguments can be overriden if needed, e.g.
foo = callPackage (import ./foo.nix) {
c = c_version_2;
};
but for the vast majority of packages, this won't be needed.
The advantages are to reduce the amount of typing needed to add a
dependency (from three sites to two), and to reduce the number of
trivial commits to all-packages.nix. For the former, there have
been two previous attempts:
- Use "args: with args;" in the package's function definition.
This however obscures the actual expected arguments of a
function, which is very bad.
- Use "{ arg1, arg2, ... }:" in the package's function definition
(i.e. use the ellipis "..." to allow arbitrary additional
arguments), and then call the function with all of "pkgs" as an
argument. But this inhibits error detection if you call it with
an misspelled (or obsolete) argument.
NixOS evaluation errors in particular look intimidating and
generally aren't very useful. Ideally the builtins.throw messages
should be self-contained.
UTC) rather than 0 (00:00:00). 1 is a better choice because some
programs use 0 as a special value. For instance, the Template
Toolkit uses a timestamp of 0 to denote the non-existence of a file,
so it barfs on files in the Nix store (see
template-toolkit-nix-store.patch in Nixpkgs). Similarly, Maya 2008
fails to load script directories with a timestamp of 0 and can't be
patched because it's closed source.
This will also shut up those "implausibly old time stamp" GNU tar
warnings.
attributes of the rec are in scope of `e'. This is useful in
expressions such as
rec {
lib = import ./lib;
inherit (lib) concatStrings;
}
It does change the semantics of expressions such as
let x = {y = 1;}; in rec { x = {y = 2;}; inherit (x) y; }.y
This now returns 2 instead of 1. However, no code in Nixpkgs or
NixOS seems to rely on the old behaviour.
shorthand for {x = {y = {z = ...;};};}. This is especially useful
for NixOS configuration files, e.g.
{
services = {
sshd = {
enable = true;
port = 2022;
};
};
}
can now be written as
{
services.sshd.enable = true;
services.sshd.port = 2022;
}
However, it is currently not permitted to write
{
services.sshd = {enable = true;};
services.sshd.port = 2022;
}
as this is considered a duplicate definition of `services.sshd'.
broken, but now the evaluator checks for it to prevent Nix
expressions from relying on undefined behaviour. Equality tests are
implemented using a shallow pointer equality test between ATerms.
However, because attribute sets are lazy and contain position
information, this can give false positives. For instance,
previously
let y = {x = 1;}; in y == y
evaluated to true, while the equivalent expression
{x = 1;} == {x = 1;}
evaluated to false. So disallow these tests for now. (Eventually
we may want to implement deep equality tests for attribute sets,
like lib.eqStrict.)
* Idem: disallow comparisons between functions.
* Implemented deep comparisons of lists. This had the same problem as
attribute sets - the elements in the list weren't evaluated. For
instance,
["xy"] == [("x" + "y")]
evaluated to false. Now it works properly.
(that is, call the build hook with a certain interval until it
accepts the build).
* build-remote.pl was totally broken: for all system types other than
the local system type, it would send all builds to the *first*
machine of the appropriate type.
poll for it (i.e. if we can't acquire the lock, then let the main
select() loop wait for at most a few seconds and then try again).
This improves parallelism: if two nix-store processes are both
trying to build a path at the same time, the second one shouldn't
block; it should first see if it can build other goals. Also, it
prevents the deadlocks that have been occuring in Hydra lately,
where a process waits for a lock held by another process that's
waiting for a lock held by the first.
The downside is that polling isn't really elegant, but POSIX doesn't
provide a way to wait for locks in a select() loop. The only
solution would be to spawn a thread for each lock to do a blocking
fcntl() and then signal the main thread, but that would require
pthreads.
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
isn't a NAR archive.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
hook. This fixes a problem with log files being partially or
completely filled with 0's because another nix-store process
truncates the log file. It should also be more efficient.
the DerivationGoal runs. Otherwise, if a goal is a top-level goal,
then the lock won't be released until nix-store finishes. With
--keep-going and lots of top-level goals, it's possible to run out
of file descriptors (this happened sometimes in the build farm for
Nixpkgs). Also, for failed derivation, it won't be possible to
build it again until the lock is released.
* Idem for locks on build users: these weren't released in a timely
manner for failed top-level derivation goals. So if there were more
than (say) 10 such failed builds, you would get an error about
having run out of build users.
scan for runtime dependencies (i.e. the local machine shouldn't do a
scan that the remote machine has already done). Also pipe directly
into `nix-store --import': don't use a temporary file.
(e.g. an SSH connection problem) and permanent failures (i.e. the
builder failed). This matters to Hydra (it wants to know whether it
makes sense to retry a build).
allocate memory, which is verboten in signal handlers. This caused
random failures in the test suite on Mac OS X (triggered by the spurious
SIGPOLL signals on Mac OS X, which should also be fixed).
closure of the inputs. This really enforces that there can't be any
undeclared dependencies on paths in the store. This is done by
creating a fake Nix store and creating bind-mounts or hard-links in
the fake store for all paths in the closure. After the build, the
build output is moved from the fake store to the real store. TODO:
the chroot has to be on the same filesystem as the Nix store for
this to work, but this isn't enforced yet. (I.e. it only works
currently if /tmp is on the same FS as /nix/store.)
bind-mounts we do are only visible to the builder process and its
children. So accidentally doing "rm -rf" on the chroot directory
won't wipe out /nix/store and other bind-mounted directories
anymore. Also, the bind-mounts in the private namespace disappear
automatically when the builder exits.
necessary that at least one build hook doesn't return "postpone",
otherwise nix-store will barf ("waiting for a build slot, yet there
are no running children"). So inform the build hook when this is
the case, so that it can start a build even when that would exceed
the maximum load on a machine.
nix-store -r (or some other operation) is started via ssh, it will
at least have a chance of terminating quickly when the connection is
killed. Right now it just runs to completion, because it never
notices that stderr is no longer connected to anything. Of course
it would be better if sshd would just send a SIGHUP, but it doesn't
(https://bugzilla.mindrot.org/show_bug.cgi?id=396).
derivation should be a source rather than a derivation dependency of
the call to the NAR derivation. Otherwise the derivation (and all
its dependencies) will be built as a side-effect, which may not even
succeed.
SHA-256 outputs of fixed-output derivations. I.e. they now produce
the same store path:
$ nix-store --add x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
$ nix-store --add-fixed --recursive sha256 x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
the latter being the same as the path that a derivation
derivation {
name = "x";
outputHashAlgo = "sha256";
outputHashMode = "recursive";
outputHash = "...";
...
};
produces.
This does change the output path for such fixed-output derivations.
Fortunately they are quite rare. The most common use is fetchsvn
calls with SHA-256 hashes. (There are a handful of those is
Nixpkgs, mostly unstable development packages.)
* Documented the computation of store paths (in store-api.cc).
dependency. `storePath /nix/store/bla' gives exactly the same
result as `toPath /nix/store/bla', except that the former includes
/nix/store/bla in the dependency context of the string.
Useful in some generated Nix expressions like nix-push, which now
finally does the right thing wrt distributed builds. (Previously
the path to be packed wasn't an explicit dependency, so it wouldn't
be copied to the remote machine.)
subtle and often hard-to-reproduce bugs where programs in pipes
either barf with a "Broken pipe" message or not, depending on the
exact timing conditions. This particularly happened in GNU M4 (and
Bison, which uses M4).
disasters involving `rm -rf' on bind mounts. Will try the
definitive fix (per-process mounts, apparently possible via the
CLONE_NEWNS flag in clone()) some other time.
accessed time of paths that may be deleted. Anything more recently
used won't be deleted. The time is specified in time_t,
e.g. seconds since 1970-01-01 00:00:00 UTC; use `date +%s' to
convert to time_t from the command line.
Example: to delete everything that hasn't been used in the last two
months:
$ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")
order of ascending last access time. This is useful in conjunction
with --max-freed or --max-links to prefer deleting non-recently used
garbage, which is good (especially in the build farm) since garbage
may become live again.
The code could easily be modified to accept other criteria for
ordering garbage by changing the comparison operator used by the
priority queue in collectGarbage().
attributes from the meta attribute. Not doing so caused nix-env to
barf on the "psi" package, which has a meta.function attribute,
the textual serialisation of which causes a gigantic string to be
produced --- so big that it causes nix-env to run out of memory.
Note however that "meta" really only should contain strings.
meta.function should be passthru.function.
particular, dietlibc cannot figure out the cwd because the inode of
the current directory doesn't appear in .. (because getdents returns
the inode of the mount point).
(which means it can only be defined via "inherit"), otherwise we get
scoping bugs, since __overrides can't be recursive (or at least, it
would be hard).
need /etc in the chroot (in particular, /etc/resolv.conf for
fetchurl). Not having /etc/resolv.conf in the chroot is a good
thing, since we don't want normal derivations to download files.
~/.nix-defexpr, otherwise the attribute cannot be selected with the
`-A' option. Useful if you want to stick a Nix expression directly
in ~/.nix-defexpr.
a rec. This will be very useful to allow end-user customisation of
all-packages.nix, for instance globally overriding GCC or some other
dependency. The // operator doesn't cut it: you could replace the
"gcc" attribute, but all other attributes would continue to
reference the original value due to the substitution semantics of
rec.
The syntax is a bit hacky but this is to allow backwards
compatibility.
in attribute set pattern matches. This allows defining a function
that takes *at least* the listed attributes, while ignoring
additional attributes. For instance,
{stdenv, fetchurl, fuse, ...}:
stdenv.mkDerivation {
...
};
defines a function that requires an attribute set that contains the
specified attributes but ignores others. The main advantage is that
we can then write in all-packages.nix
aefs = import ../bla/aefs pkgs;
instead of
aefs = import ../bla/aefs {
inherit stdenv fetchurl fuse;
};
This saves a lot of typing (not to mention not having to update
all-packages.nix with purely mechanical changes). It saves as much
typing as the "args: with args;" style, but has the advantage that
the function arguments are properly declared (not implicit in what
the body of the "with" uses).
functions that take a single argument (plain lambdas) into one AST
node (Function) that contains a Pattern node describing the
arguments. Current patterns are single lazy arguments (VarPat) and
matching against an attribute set (AttrsPat).
This refactoring allows other kinds of patterns to be added easily,
such as Haskell-style @-patterns, or list pattern matching.
`-u'. Instead of acquiring an exclusive lock on the profile for the
entire duration of the operation, we just perform the operation
optimistically (without an exclusive lock), and check at the end
whether the profile changed while we were busy (i.e., the symlink
target changed). If so, the operation is restarted. Restarting is
generally cheap, since the build results are still in the Nix store.
Most of the time, only the user environment has to be rebuilt.
again. (After the previous substituter mechanism refactoring I
didn't update the code that obtains the references of substitutable
paths.) This required some refactoring: the substituter programs
are now kept running and receive/respond to info requests via
stdin/stdout.
it only does something if $NIX_OTHER_STORES (not really a good
name...) is set.
* Do globbing on the elements of $NIX_OTHER_STORES. E.g. you could
set it to /mnts/*/nix or something.
* Install substituters in libexec/nix/substituters.
logic through the `parseDrvName' and `compareVersions' primops.
This will allow expressions to easily check whether some dependency
is a specific needed version or falls in some version range. See
tests/lang/eval-okay-versions.nix for examples.
bytes have been freed, `--max-links' to stop when the Nix store
directory has fewer than N hard links (the latter being important
for very large Nix stores on filesystems with a 32000 subdirectories
limit).
store under the reference relation, since that means that the
garbage collector will need a long time to start deleting paths.
Instead just delete the referrers of a path first.
This isn't usually a problem, except that it causes tests to fail
when performed in a directory with a very long path name. So chdir
to the socket directory and use a relative path name.
/tmp/nix-<pid>-<counter> for temporary build directories. This
increases purity a bit: many packages store the temporary build path
in their output, causing (generally unimportant) binary differences.
undefined variables by definition. This matters for the
implementation of "with", which does a call to checkVarDefs to see
if the body of the with has no undefined variables. (It can't be
checked at parse time because you don't know which variables are in
the "with" attribute set.) If we check closed terms, then we check
not just the with body but also the substituted terms, which are
typically very large. This is the cause of the poor nix-env
performance on Nixpkgs lately. It didn't happen earlier because
"with" wasn't used very often in the past.
This fix improves nix-env performance roughly 60x on current Nixpkgs.
nix-env -qa is down from 29.3s to 0.5s on my laptop, and nix-env -qa
--out-path is down from 229s to 3.39s. Not bad for a 1-line fix :-)
