SQLite manifest cache. The DBI AutoCommit feature caused every
process to have an active transaction at all times, which could
indefinitely block processes wanting to update the manifest cache.
* Disable fsync() in the manifest cache because we don't need
integrity (the cache can always be recreated if it gets corrupted).
them into memory. This brings memory use down to (more or less)
O(1). For instance, on my test case, the maximum resident size of
download-using-manifests while filling the DB went from 142 MiB to
11 MiB.
This significantly speeds up the download-using-manifests
substituter, especially if manifests are very large. For instance,
one "nix-build -A geeqie" operation that updated four packages using
binary patches went from 18.5s to 1.6s. It also significantly
reduces memory use.
The cache is kept in /nix/var/nix/manifests/cache.sqlite. It's
updated automatically when manifests are added to or removed from
/nix/var/nix/manifests. It might be interesting to have nix-pull
store manifests directly in the DB, rather than storing them as
separate flat files, but then we would need a command line interface
to delete manifests from the DB.
`+' and `?' in filenames. This is very slow if /nix/store is very
large. (This is a quick hack - a cleaner solution would be to
bypass the shell entirely.)
`nix-store -q --hash' to get the hash of the base path rather than
`nix-hash'. However, only do this for estimating the size of a
download, not for the actual substitution, because sometimes the
contents of store paths are modified (which they shouldn't, of
course).
hook script proper, and the stdout/stderr of the builder. Only the
latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
it requires a certain feature on the build machine, e.g.
requiredSystemFeatures = [ "kvm" ];
We need this in Hydra to make sure that builds that require KVM
support are forwarded to machines that have KVM support. Probably
this should also be enforced for local builds.
the hook every time we want to ask whether we can run a remote build
(which can be very often), we now reuse a hook process for answering
those queries until it accepts a build. So if there are N
derivations to be built, at most N hooks will be started.
load on the Hydra build farm (where it's unnecessary anyway because
it has a fast connection to the build machines). In any case,
compression can be enabled by using the `-C' option to ssh.
doesn't exist. The Debian packages don't include the manifests
directory, so nix-channel would silently skip doing a nix-pull,
resulting in everything being built from source. Thanks to Juan
Pedro Bolívar Puente.
giving jobs to the first machine until it hits its job limit, then
the second machine and so on. This should improve utilisation of
the Hydra build farm a lot. Also take an optional speed factor
into account to cause fast machines to be preferred over slower
machines with a similar load.
(that is, call the build hook with a certain interval until it
accepts the build).
* build-remote.pl was totally broken: for all system types other than
the local system type, it would send all builds to the *first*
machine of the appropriate type.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
downloaded files; rather, check the hash of the unpacked store
path.
When the server produces bzipped NAR archives on demand (like Hydra
does), the hash of the file is not known in advance; it's streamed
from the server. Thus the manifest doesn't contain a hash for the
bzipped NAR archive. However, the server does know the hash of the
*uncompressed* NAR archive (the "NarHash" field), since it's stored
in the Nix database (nix-store -q --hash /nix/store/bla). So we use
that instead for checking the integrity of the download.
scan for runtime dependencies (i.e. the local machine shouldn't do a
scan that the remote machine has already done). Also pipe directly
into `nix-store --import': don't use a temporary file.
(e.g. an SSH connection problem) and permanent failures (i.e. the
builder failed). This matters to Hydra (it wants to know whether it
makes sense to retry a build).
makes more sense for the build farm, otherwise every nix-store
invocation will lead to at least one local build. Will come up with
a better solution later...
necessary that at least one build hook doesn't return "postpone",
otherwise nix-store will barf ("waiting for a build slot, yet there
are no running children"). So inform the build hook when this is
the case, so that it can start a build even when that would exceed
the maximum load on a machine.
list like
root@example.org x86_64-linux /root/.ssh/id_buildfarm 1
root@example.org i686-darwin /root/.ssh/id_buildfarm 1
This is possible when the Nix installation on example.org itself has
remote builds enabled.
derivation should be a source rather than a derivation dependency of
the call to the NAR derivation. Otherwise the derivation (and all
its dependencies) will be built as a side-effect, which may not even
succeed.
dependency. `storePath /nix/store/bla' gives exactly the same
result as `toPath /nix/store/bla', except that the former includes
/nix/store/bla in the dependency context of the string.
Useful in some generated Nix expressions like nix-push, which now
finally does the right thing wrt distributed builds. (Previously
the path to be packed wasn't an explicit dependency, so it wouldn't
be copied to the remote machine.)
Common code in local build package sources refactored out in a function; before building the real set of derivations needed is found (slightly slower for only one build strategy, but less garbage on output and better performance for multiple build strategies).
Now you have full choice of best-effort build regardless of method (substituters or actual build), using substituters, building only fixed derivations (should get you all the downloads) and local build without even trying substituters.
Some minor fix in the help text about behavior with no package sources.
again. (After the previous substituter mechanism refactoring I
didn't update the code that obtains the references of substitutable
paths.) This required some refactoring: the substituter programs
are now kept running and receive/respond to info requests via
stdin/stdout.
it only does something if $NIX_OTHER_STORES (not really a good
name...) is set.
* Do globbing on the elements of $NIX_OTHER_STORES. E.g. you could
set it to /mnts/*/nix or something.
* Install substituters in libexec/nix/substituters.
stores (typically remote Nix stores mounted via e.g. NFS, or the Nix
store on the NixOS installation CD). Example use:
$ sshfs foo@example.org:/ /mnt
$ NIX_OTHER_STORES=/mnt/nix \
NIX_SUBSTITUTERS=.../copy-from-other-stores.pl \
nix-env -i foo
This will be especially useful for the installation CD since it
doesn't require a manifest for the CD contents.
usage by finding identical files in the store and hard-linking them
to each other. It typically reduces the size of the store by
something like 25-35%. This is what the optimise-store.pl script
did, but the new command is faster and more correct (it's safe wrt
garbage collection and concurrent builds).
Nix expressions in that directory are combined into an attribute set
{file1 = import file1; file2 = import file2; ...}, i.e. each Nix
expression is an attribute with the file name as the attribute
name. Also recurses into directories.
* nix-env: removed the "--import" (-I) option which set the
~/.nix-defexpr symlink.
* nix-channel: don't use "nix-env --import", instead symlink
~/.nix-defexpr/channels. So finally nix-channel --update doesn't
override any default Nix expressions but combines with them.
This means that you can have (say) a local Nixpkgs SVN tree and use
it as a default for nix-env:
$ ln -s .../path-to-nixpkgs-tree ~/.nix-defexpr/nixpkgs_svn
and be subscribed to channels (including Nixpkgs) at the same time.
(If there is any ambiguity, the -A flag can be used to
disambiguate, e.g. "nix-env -i -A nixpkgs_svn.pan".)
manifests directory. In that case, we don't do a nix-pull, so the
user gets pure source deployment.
The directory /nix/var/nix/gcroots/per-user/$USER should be
writable. (It's created automatically if
/nix/var/nix/gcroots/per-user is writable, e.g. if it has 1777
permission.)
need any info on substitutable paths, we just call the substituters
(such as download-using-manifests.pl) directly. This means that
it's no longer necessary for nix-pull to register substitutes or for
nix-channel to clear them, which makes those operations much faster
(NIX-95). Also, we don't have to worry about keeping nix-pull
manifests (in /nix/var/nix/manifests) and the database in sync with
each other.
The downside is that there is some overhead in calling an external
program to get the substitutes info. For instance, "nix-env -qas"
takes a bit longer.
Abolishing the substitutes table also makes the logic in
local-store.cc simpler, as we don't need to store info for invalid
paths. On the downside, you cannot do things like "nix-store -qR"
on a substitutable but invalid path (but nobody did that anyway).
* Never catch interrupts (the Interrupted exception).
NIX_DOWNLOAD_CACHE is set, then nix-prefetch-url will store the hash
and timestamp of downloaded files in the directory
$NIX_DOWNLOAD_CACHE. This allows it to figure out if the file is
still in the Nix store.
