instantiation, e.g. "nix-env -i" and "nix-env -qas" (but not
"nix-env -qa"). It turns out that many redundant calls to
addToStore(path) were made, which reads and hashes the entire path.
For instance, the bash bootstrap binary in Nixpkgs would be read and
hashed many times. As a result nix-env would spend around 92% of
its time in the function sha256_block (according to callgrind).
Some simple memoization fixes this.
expressions that cause an assertion failure (like `assert system ==
"i686-linux"'). This allows all-packages.nix in Nixpkgs to be used
on all platforms, even if some Nix expressions don't work on all
platforms.
Not sure if this is a good idea; it's a bit hacky. In particular,
due to laziness some derivations might appear in `nix-env -qa' but
disappear in `nix-env -qas' or `nix-env -i'.
Commit 5000!
derivation(s) we're interested, e.g.,
$ nix-instantiate ./all-packages.nix --attr xlibs.libX11
List elements can also be selected:
$ nix-instantiate ./build-for-release.nix --attr 0.subversion
This allows a non-ambiguous specification of a derivation. Of
course, this should also be added to nix-env and nix-build.
dependencyClosure { ... searchPath = [ ../foo ../bar ]; ... }
* Primop `dirOf' to return the directory part of a path (e.g., dirOf
/a/b/c == /a/b).
* Primop `relativise' (according to Webster that's a real word!) that
given paths A and B returns a string representing path B relative
path to A; e.g., relativise /a/b/c a/b/x/y => "../x/y".
determination (e.g., finding the header files dependencies of a C
file) in Nix low-level builds automatically.
For instance, in the function `compileC' in make/lib/default.nix, we
find the header file dependencies of C file `main' as follows:
localIncludes =
dependencyClosure {
scanner = file:
import (findIncludes {
inherit file;
});
startSet = [main];
};
The function works by "growing" the set of dependencies, starting
with the set `startSet', and calling the function `scanner' for each
file to get its dependencies (which should yield a list of strings
representing relative paths). For instance, when `scanner' is
called on a file `foo.c' that includes the line
#include "../bar/fnord.h"
then `scanner' should yield ["../bar/fnord.h"]. This list of
dependencies is absolutised relative to the including file and added
to the set of dependencies. The process continues until no more
dependencies are found (hence its a closure).
`dependencyClosure' yields a list that contains in alternation a
dependency, and its relative path to the directory of the start
file, e.g.,
[ /bla/bla/foo.c
"foo.c"
/bla/bar/fnord.h
"../bar/fnord.h"
]
These relative paths are necessary for the builder that compiles
foo.c to reconstruct the relative directory structure expected by
foo.c.
The advantage of `dependencyClosure' over the old approach (using
the impure `__currentTime') is that it's completely pure, and more
efficient because it only rescans for dependencies (i.e., by
building the derivations yielded by `scanner') if sources have
actually changed. The old approach rescanned every time.
(closed(closed(closed(...)))) since this reduces performance by
producing bigger terms and killing caching (which incidentally also
prevents useful infinite recursion detection).
`removeAttrs attrs ["x", "y"]' returns the set `attrs' with the
attributes named `x' and `y' removed. It is not an error for the
named attributes to be missing from the input set.
* Make the `derivation' primitive much more lazy. The expression
`derivation attrs' now evaluates to (essentially)
attrs // {
type = "derivation";
outPath = derivation! attrs;
drvPath = derivation! attrs;
}
where `derivation!' is a primop that does the actual derivation
instantiation (i.e., it does what `derivation' used to do). The
advantage is that it allows commands such as `nix-env -qa' and
`nix-env -i' to be much faster since they no longer need to
instantiate all derivations, just the `name' attribute. (However,
`nix-env' doesn't yet take advantage of this since it still always
evaluates the `outPath' and `drvPath' attributes).
Also, this allows derivations to cyclically reference each other,
for example,
webServer = derivation {
...
hostName = "svn.cs.uu.nl";
services = [svnService];
};
svnService = derivation {
...
hostName = webServer.hostName;
};
Previously, this would yield a black hole (infinite recursion).
derivations. This is mostly to simplify the implementation of
nix-prefetch-{url, svn}, which now work properly in setuid
installations.
* Enforce valid store names in `nix-store --add / --add-fixed'.
* Set the references for the user environment manifest properly.
* Don't copy the manifest (this was accidental).
* Don't store derivation paths in the manifest (maybe this should be
made optional). This cleans up the semantics of nix-env, which were
weird.
* Hash on the output paths of activated components, not on derivation
paths. This is because we don't know the derivation path of already
installed components anymore, and it allows the installation of
components by store path (skipping Nix expressions entirely).
* Query options `--out-path' and `--drv-path' to show the output and
derivation paths of components, respectively (the latter replaces
the `--expr' query).
promise :-) This allows derivations to specify on *what* output
paths of input derivations they are dependent. This helps to
prevent unnecessary downloads. For instance, a build might be
dependent on the `devel' and `lib' outputs of some library
component, but not the `docs' output.
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
they have been built/added. This is so that we can check whether
the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
holds.
representation of closures as ATerms in the Nix store. Instead, the
file system pointer graph is now stored in the Nix database. This
has many advantages:
- It greatly simplifies the implementation (we can drop the notion
of `successors', and so on).
- It makes registering roots for the garbage collector much easier.
Instead of specifying the closure expression as a root, you can
simply specify the store path that must be retained as a root.
This could not be done previously, since there was no way to find
the closure store expression containing a given store path.
- Better traceability: it is now possible to query what paths are
referenced by a path, and what paths refer to a path.
* Formalise the notion of fixed-output derivations, i.e., derivations
for which a cryptographic hash of the output is known in advance.
Changes to such derivations should not propagate upwards through the
dependency graph. Previously this was done by specifying the hash
component of the output path through the `id' attribute, but this is
insecure since you can lie about it (i.e., you can specify any hash
and then produce a completely different output). Now the
responsibility for checking the output is moved from the builder to
Nix itself.
A fixed-output derivation can be created by specifying the
`outputHash' and `outputHashAlgo' attributes, the latter taking
values `md5', `sha1', and `sha256', and the former specifying the
actual hash in hexadecimal or in base-32 (auto-detected by looking
at the length of the attribute value). MD5 is included for
compatibility but should be considered deprecated.
* Removed the `drvPath' pseudo-attribute in derivation results. It's
no longer necessary.
* Cleaned up the support for multiple output paths in derivation store
expressions. Each output now has a unique identifier (e.g., `out',
`devel', `docs'). Previously there was no way to tell output paths
apart at the store expression level.
* `nix-hash' now has a flag `--base32' to specify that the hash should
be printed in base-32 notation.
* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
`md5'.
* `nix-prefetch-url' now prints out a SHA-1 hash in base-32. (TODO: a
flag to specify the hash.)
bits, then encode them in a radix-32 representation (using digits
and letters except e, o, u, and t). This produces store paths like
/nix/store/4i0zb0z7f88mwghjirkz702a71dcfivn-aterm-2.3.1. The nice
thing about this is that the hash part of the file name is still 32
characters, as before with MD5.
(Of course, shortening SHA-256 to 160 bits makes it no better than
SHA-160 in theory, but hopefully it's a bit more resistant to
attacks; it's certainly a lot slower.)
* Start cleaning up unique store path generation (they weren't always
unique; in particular the suffix ("-aterm-2.2", "-builder.sh") was
not part of the hash, therefore changes to the suffix would cause
multiple store objects with the same hash).
Instead we generate data bindings (build and match functions) for
the constructors specified in `constructors.def'. In particular
this removes the conversions between AFuns and strings, and Nix
expression evaluation now seems 3 to 4 times faster.
The expression `with E1; E2' evaluates to E2 with all bindings in
the attribute set E1 substituted. E.g.,
with {x = 123;}; x
evaluates to 123. That is, the attribute set E1 is in scope in E2.
This is particularly useful when importing files containing lots
definitions. E.g., instead of
let {
inherit (import ./foo.nix) a b c d e f;
body = ... a ... f ...;
}
we can now say
with import ./foo.nix;
... a ... f ...
I.e., we don't have to say what variables should be brought into scope.
permission to the Nix store or database. E.g., `nix-env -qa' will
work, but `nix-env -qas' won't (the latter needs DB access). The
option `--readonly-mode' forces this mode; otherwise, it's only
activated when the database cannot be opened.
Previously there was the problem that all files read by nix-env
etc. should be reachable and readable by the Nix user. So for
instance building a Nix expression in your home directory meant that
the home directory should have at least g+x or o+x permission so
that the Nix user could reach the Nix expression. Now we just
switch back to the original user just prior to reading sources and
the like. The places where this happens are somewhat arbitrary,
however. Any scope that has a live SwitchToOriginalUser object in
it is executed as the original user.
* Back out r1385. setreuid() sets the saved uid to the new
real/effective uid, which prevents us from switching back to the
original uid. setresuid() doesn't have this problem (although the
manpage has a bug: specifying -1 for the saved uid doesn't leave it
unchanged; an explicit value must be specified).
print a nice backtrace of the stack, rather than vomiting a gigantic
(and useless) aterm on the screen. Example:
error: while evaluating file `.../pkgs/system/test.nix':
while evaluating attribute `subversion' at `.../pkgs/system/all-packages-generic.nix', line 533:
while evaluating function at `.../pkgs/applications/version-management/subversion/default.nix', line 1:
assertion failed at `.../pkgs/applications/version-management/subversion/default.nix', line 13
Since the Nix expression language is lazy, the trace may be
misleading. The purpose is to provide a hint as to the location of
the problem.
instead of `derivation' triggered a huge slowdown in the Nix
expression evaluator. Total execution time of `nix-env -qa' went up
by a factor of 60 or so.
This scalability problem was caused by expressions such as
(x: y: ... x ...) a b
where `a' is a large term (say, the one in
`all-packages-generic.nix'). Then the first beta-reduction would
produce
(y: ... a ...) b
by substituting `a' for `x'. The second beta-reduction would then
substitute `b' for `y' into the body `... a ...', which is a large
term due to `a', and thus causes a large traversal to be performed
by substitute() in the second reduction. This is however entirely
redundant, since `a' cannot contain free variables (since we never
substitute below a weak head normal form).
The solution is to wrap substituted terms into a `Closed'
constructor, i.e.,
subst(subs, Var(x)) = Closed(e) iff subs[x] = e
have substitution not descent into closed terms,
subst(subs, Closed(x)) = Closed(x)
and otherwise ignore them for evaluation,
eval(Closed(x)) = eval(x).
* Fix a typo that caused incorrect substitutions to be performed in
simple lambdas, e.g., `(x: x: x) a' would reduce to `(x: a)'.
`bla:' is now no longer parsed as a URL.
* Re-enabled support for the `args' attribute in derivations to
specify command line arguments to the builder, e.g.,
...
builder = /usr/bin/python;
args = ["-c" ./builder.py];
...
other attribute sets, rather than the current scope. E.g.,
{inherit (pkgs) gcc binutils;}
is equivalent to
{gcc = pkgs.gcc; binutils = pkgs.binutils;}
I am not so happy about the syntax.
parser (roughly 80x faster).
The absolutely latest version of Bison (1.875c) is required for
reentrant GLR support, as well as a recent version of Flex (say,
2.5.31). Note that most Unix distributions ship with the
prehistoric Flex 2.5.4, which doesn't support reentrancy.
Nix. This is to prevent Berkeley DB from becoming wedged.
Unfortunately it is not possible to throw C++ exceptions from a
signal handler. In fact, you can't do much of anything except
change variables of type `volatile sig_atomic_t'. So we set an
interrupt flag in the signal handler and check it at various
strategic locations in the code (by calling checkInterrupt()).
Since this is unlikely to cover all cases (e.g., (semi-)infinite
loops), sometimes SIGTERM may now be required to kill Nix.
path of the Nix expression to be used with the import, upgrade, and
query commands. For instance,
$ nix-env -I ~/nixpkgs/pkgs/system/i686-linux.nix
$ nix-env --query --available [aka -qa]
sylpheed-0.9.7
bison-1.875
pango-1.2.5
subversion-0.35.1
...
$ nix-env -i sylpheed
$ nix-env -u subversion
There can be only one default at a time.
* If the path to a Nix expression is a symlink, follow the symlink
prior to resolving relative path references in the expression.
