Commit graph

123 commits

Author SHA1 Message Date
Eelco Dolstra
28e7e29abd Eliminate reserveSpace flag 2016-02-24 17:44:55 +01:00
Eelco Dolstra
c8f4d89a34 Expose the export magic value and move LocalStore::queryReferences to Store 2016-02-15 15:01:26 +01:00
Eelco Dolstra
fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra
c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra
9e7c1a4bbd Use the daemon when we don't have write access to the Nix database 2016-01-31 10:28:48 +01:00
Eelco Dolstra
4ba6bc184c Shut up clang warnings 2015-09-18 01:22:35 +02:00
Eelco Dolstra
1511aa9f48 Allow remote builds without sending the derivation closure
Previously, to build a derivation remotely, we had to copy the entire
closure of the .drv file to the remote machine, even though we only
need the top-level derivation. This is very wasteful: the closure can
contain thousands of store paths, and in some Hydra use cases, include
source paths that are very large (e.g. Git/Mercurial checkouts).

So now there is a new operation, StoreAPI::buildDerivation(), that
performs a build from an in-memory representation of a derivation
(BasicDerivation) rather than from a on-disk .drv file. The only files
that need to be in the Nix store are the sources of the derivation
(drv.inputSrcs), and the needed output paths of the dependencies (as
described by drv.inputDrvs). "nix-store --serve" exposes this
interface.

Note that this is a privileged operation, because you can construct a
derivation that builds any store path whatsoever. Fixing this will
require changing the hashing scheme (i.e., the output paths should be
computed from the other fields in BasicDerivation, allowing them to be
verified without access to other derivations). However, this would be
quite nice because it would allow .drv-free building (e.g. "nix-env
-i" wouldn't have to write any .drv files to disk).

Fixes #173.
2015-07-17 17:57:40 +02:00
Eelco Dolstra
7ea6ecf855 addToStore(): Take explicit name argument 2015-03-25 17:06:12 +01:00
Eelco Dolstra
1256ab3b44 Clean up temp roots in a more C++ way 2014-11-19 17:07:29 +01:00
Ludovic Courtès
8fb8c26b6d Add an 'optimiseStore' remote procedure call. 2014-09-01 23:53:26 +02:00
Eelco Dolstra
54a34119f3 Use std::unordered_set 2014-05-26 17:53:17 +02:00
Eelco Dolstra
84813af5b9 nix-store --optimise: Remove bogus statistics 2014-05-15 11:33:46 +02:00
Wout Mertens
3b9ea8452f Shortcut store files before lstat
readdir() already returns the inode numbers, so we don't need to call
lstat to know if a file was already linked or not.
2014-05-15 09:02:22 +02:00
Wout Mertens
d73ffc552f Use the inodes given by readdir directly 2014-05-14 22:52:10 +02:00
Wout Mertens
e974f20c98 Preload linked hashes to speed up lookups
By preloading all inodes in the /nix/store/.links directory, we can
quickly determine of a hardlinked file was already linked to the hashed
links.
This is tolerant of removing the .links directory, it will simply
recalculate all hashes in the store.
2014-05-13 23:10:06 +02:00
Eelco Dolstra
1aa19b24b2 Add a flag ‘--check’ to verify build determinism
The flag ‘--check’ to ‘nix-store -r’ or ‘nix-build’ will cause Nix to
redo the build of a derivation whose output paths are already valid.
If the new output differs from the original output, an error is
printed.  This makes it easier to test if a build is deterministic.
(Obviously this cannot catch all sources of non-determinism, but it
catches the most common one, namely the current time.)

For example:

  $ nix-build '<nixpkgs>' -A patchelf
  ...
  $ nix-build '<nixpkgs>' -A patchelf --check
  error: derivation `/nix/store/1ipvxsdnbhl1rw6siz6x92s7sc8nwkkb-patchelf-0.6' may not be deterministic: hash mismatch in output `/nix/store/4pc1dmw5xkwmc6q3gdc9i5nbjl4dkjpp-patchelf-0.6.drv'

The --check build fails if not all outputs are valid.  Thus the first
call to nix-build is necessary to ensure that all outputs are valid.

The current outputs are left untouched: the new outputs are either put
in a chroot or diverted to a different location in the store using
hash rewriting.
2014-02-18 01:01:14 +01:00
Eelco Dolstra
a478e8a7bb Remove nix-setuid-helper
AFAIK, nobody uses it, it's not maintained, and it has no tests.
2013-11-14 11:57:37 +01:00
Eelco Dolstra
a737f51fd9 Retry all SQLite operations
To deal with SQLITE_PROTOCOL, we also need to retry read-only
operations.
2013-10-16 15:58:20 +02:00
Ivan Kozik
34bb806f74 Fix typos, especially those that end up in the Nix manual 2013-08-26 11:15:22 +02:00
Eelco Dolstra
22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra
1b6ee8f4c7 Allow hard links between the outputs of a derivation 2013-06-13 17:29:56 +02:00
Eelco Dolstra
5959c591a0 Process stderr from substituters while doing have/info queries 2013-06-07 15:02:14 +02:00
Eelco Dolstra
c5f9d0d080 Buffer reads from the substituter
This greatly reduces the number of system calls.
2013-06-07 14:00:23 +02:00
Eelco Dolstra
bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra
28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra
b008674e46 Refactoring: Split off the non-recursive canonicalisePathMetaData()
Also, change the file mode before changing the owner.  This prevents a
slight time window in which a setuid binary would be setuid root.
2013-02-27 16:42:19 +01:00
Eelco Dolstra
5526a282b5 Security: Don't allow builders to change permissions on files they don't own
It turns out that in multi-user Nix, a builder may be able to do

  ln /etc/shadow $out/foo

Afterwards, canonicalisePathMetaData() will be applied to $out/foo,
causing /etc/shadow's mode to be set to 444 (readable by everybody but
writable by nobody).  That's obviously Very Bad.

Fortunately, this fails in NixOS's default configuration because
/nix/store is a bind mount, so "ln" will fail with "Invalid
cross-device link".  It also fails if hard-link restrictions are
enabled, so a workaround is:

  echo 1 > /proc/sys/fs/protected_hardlinks

The solution is to check that all files in $out are owned by the build
user.  This means that innocuous operations like "ln
${pkgs.foo}/some-file $out/" are now rejected, but that already failed
in chroot builds anyway.
2013-02-26 02:30:19 +01:00
Eelco Dolstra
def5160b61 Clear any immutable bits in the Nix store
Doing this once makes subsequent operations like garbage collecting
more efficient since we don't have to call makeMutable() first.
2013-01-03 12:59:23 +01:00
Eelco Dolstra
2754a07ead nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settings
So if a path is not garbage solely because it's reachable from a root
due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store
-q --roots’ now shows that root.
2012-12-20 18:41:44 +01:00
Eelco Dolstra
06f62defe6 Yet another rewrite of the garbage collector
But this time it's *obviously* correct!  No more segfaults due to
infinite recursions for sure, etc.

Also, move directories to /nix/store/trash instead of renaming them to
/nix/store/bla-gc-<pid>.  Then we can just delete /nix/store/trash at
the end.
2012-12-20 17:32:15 +01:00
Eelco Dolstra
ea89df2b76 Use vfork() instead of fork() if available
Hopefully this reduces the chance of hitting ‘unable to fork: Cannot
allocate memory’ errors.  vfork() is used for everything except
starting builders.
2012-11-09 18:00:33 +01:00
Eelco Dolstra
0a7084567f Add a ‘--repair’ flag to nix-instantiate
This allows repairing corrupted derivations and other source files.
2012-10-03 15:09:18 -04:00
Eelco Dolstra
a3f205b249 When repairing a derivation, check and repair the entire output closure
If we find a corrupted path in the output closure, we rebuild the
derivation that produced that particular path.
2012-10-03 10:38:09 -04:00
Eelco Dolstra
2001895f3d Add a --repair flag to ‘nix-store -r’ to repair derivation outputs
With this flag, if any valid derivation output is missing or corrupt,
it will be recreated by using a substitute if available, or by
rebuilding the derivation.  The latter may use hash rewriting if
chroots are not available.
2012-10-02 17:13:46 -04:00
Eelco Dolstra
8e3a7bd712 nix-store --verify: Add an option ‘--repair’ to repair all missing/corrupt paths
Also, return a non-zero exit code if errors remain after
verifying/repairing.
2012-10-02 15:12:56 -04:00
Eelco Dolstra
9958bd6992 Add operation ‘nix-store --repair-path’
This operation allows fixing corrupted or accidentally deleted store
paths by redownloading them using substituters, if available.

Since the corrupted path cannot be replaced atomically, there is a
very small time window (one system call) during which neither the old
(corrupted) nor the new (repaired) contents are available.  So
repairing should be used with some care on critical packages like
Glibc.
2012-10-02 14:08:59 -04:00
Eelco Dolstra
b9124a5c33 Support having /nix/store as a read-only bind mount
It turns out that the immutable bit doesn't work all that well.  A
better way is to make the entire Nix store a read-only bind mount,
i.e. by doing

  $ mount --bind /nix/store /nix/store
  $ mount -o remount,ro,bind /nix/store

(This would typically done in an early boot script, before anything
from /nix/store is used.)

Since Nix needs to be able to write to the Nix store, it now detects
if /nix/store is a read-only bind mount and then makes it writable in
a private mount namespace.
2012-09-19 15:45:29 -04:00
Eelco Dolstra
e6e495649c Vacuum the SQLite DB after running the garbage collector 2012-09-13 14:33:41 -04:00
Eelco Dolstra
e94806d030 Merge branch 'master' into no-manifests 2012-08-27 11:09:07 -04:00
Eelco Dolstra
6763084ae5 Count bytes freed deleting unused links 2012-08-01 22:43:03 -04:00
Eelco Dolstra
01d56c1eec Drop the block count in the garbage collector 2012-08-01 22:34:46 -04:00
Eelco Dolstra
c770a2422a Report substituter errors to clients of the Nix daemon 2012-08-01 11:19:24 -04:00
Eelco Dolstra
97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra
f9613da180 Remove unused variables 2012-07-30 15:43:37 -04:00
Eelco Dolstra
8c79100839 Merge branch 'master' into no-manifests 2012-07-26 15:14:33 -04:00
Eelco Dolstra
680ab6f83d Garbage collect unused links in /nix/store/.links
Incremental optimisation requires creating links in /nix/store/.links
to all files in the store.  However, this means that if we delete a
store path, no files are actually deleted because links in
/nix/store/.links still exists.  So we need to check /nix/store/.links
for files with a link count of 1 and delete them.
2012-07-23 17:14:16 -04:00
Eelco Dolstra
6193105710 Automatically optimise the Nix store when a new path is added
Auto-optimisation is enabled by default.  It can be turned off by
setting auto-optimise-store to false in nix.conf.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
564fb7d9fa optimiseStore(): Use a content-addressed file store in /nix/store/.links
optimiseStore() now creates persistent, content-addressed hard links
in /nix/store/.links.  For instance, if it encounters a file P with
hash H, it will create a hard link

  P' = /nix/store/.link/<H>

to P if P' doesn't already exist; if P' exist, then P is replaced by a
hard link to P'.  This is better than the previous in-memory map,
because it had the tendency to unnecessarily replace hard links with a
hard link to whatever happened to be the first file with a given hash
it encountered.  It also allows on-the-fly, incremental optimisation.
2012-07-23 17:14:15 -04:00
Eelco Dolstra
b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra
fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00