Commit graph

568 commits

Author SHA1 Message Date
Eelco Dolstra
6f788880b6 * Re-enable dot graph generation. 2005-03-26 22:06:57 +00:00
Eelco Dolstra
298dd487bb * When finding live paths, the deriver need not be valid. 2005-03-25 14:31:12 +00:00
Eelco Dolstra
7eaf038763 * `nix-store --verify': repair bad referer mappings. 2005-03-25 14:21:49 +00:00
Eelco Dolstra
c6178f0b03 * Create missing log and temproots directories automatically (reported
by Rob).
2005-03-24 17:46:38 +00:00
Eelco Dolstra
cff6bc06df * Fix endianness bug. 2005-03-23 19:18:22 +00:00
Eelco Dolstra
0df9f08078 * Export the references graph to the build hook. 2005-03-23 13:16:36 +00:00
Eelco Dolstra
3f236f01ae * `nix-store --register-validity': allow a path to refer to a path
listed later in the list of new valid paths.
2005-03-23 13:07:28 +00:00
Eelco Dolstra
a04c62e0c4 * Canonicalise path meta-data in `nix-store --register-validity'. 2005-03-23 12:06:57 +00:00
Eelco Dolstra
f20f081560 * nix-store: --isvalid' -> --check-validity', `--validpath' ->
`--register-validity'.
* `nix-store --register-validity': read arguments from stdin, and
  allow the references and deriver to be set.
2005-03-23 11:25:20 +00:00
Eelco Dolstra
c757d16c8c * Bug in clearSubstitutes(). 2005-03-14 18:54:40 +00:00
Eelco Dolstra
08df443618 * Check for duplicate attributes and formal parameters in Nix
expressions.
2005-03-10 11:33:46 +00:00
Eelco Dolstra
4bbdcfbb45 * Don't use fork() in copyPath(), but a string buffer. 2005-03-03 13:58:02 +00:00
Eelco Dolstra
86cb3cc554 * Increase Berkeley DB limits a bit more.
* Maintain the cleanup invariant in clearSubstitutes().
2005-03-03 13:10:44 +00:00
Eelco Dolstra
07b4399fb6 * `nix-store -q --hash' to quickly query the hash of the contents of a
store path (which is stored in the database).
2005-03-02 15:57:06 +00:00
Eelco Dolstra
bfaf83a0fd * When multiple derivations are specified in `nix-store -r', don't
continue building when one fails unless `--keep-going' is
  specified.
* When `--keep-going' is specified, print out the set of failing
  derivations at the end (otherwise it can be hard to find out which
  failed).
2005-02-23 11:19:27 +00:00
Eelco Dolstra
3a2c3f0cf2 * Support for fixed-output hashes over directory trees (i.e., over the
NAR dump of the path).
2005-02-22 21:14:41 +00:00
Eelco Dolstra
eda2c3c253 * Compatibility hack so that Nixpkgs can continue to do hash checking
in `fetchurl' in Nix <= 0.7, but doesn't in Nix 0.8.
2005-02-22 15:23:24 +00:00
Eelco Dolstra
3c1630131e * Subtle bug in the builder: if a subgoal that is instantiated
multiple times is also a top-level goal, then the second and later
  instantiations would never be created because there would be a
  stable pointer to the first one that would keep it alive in the
  WeakGoalMap.
* Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
Eelco Dolstra
e0181f56be * `nix-store -q --tree' shows a tree representing the dependency graph
of the given derivation.  Useful for getting a quick overview of how
  something was built.  E.g., to find out how the `baffle' program in
  your user environment was built, you can do

    $ nix-store -q --tree $(nix-store -qd $(which baffle))

  Tree nesting depth is minimised (?) by topologically sorting paths
  under the relation A < B iff A \in closure(B).
2005-02-17 15:57:46 +00:00
Eelco Dolstra
74ab0695b5 * Compatibility hack with older user environments. 2005-02-17 15:48:50 +00:00
Eelco Dolstra
8a3a96dd5b * Switch to the calling user context for some more operations in a
setuid installation.
2005-02-17 13:55:18 +00:00
Eelco Dolstra
88273f9574 * Put build logs in $prefix/var/nix/log/drvs/. 2005-02-17 13:54:45 +00:00
Eelco Dolstra
202d5bbda5 * Compatibility with older GCCs. 2005-02-15 12:05:47 +00:00
Eelco Dolstra
e17910cfb5 * And yet another installation source: the ability to copy user
environment elements from one user environment to another, e.g.,

    $ nix-env -i --from-profile /nix/var/nix/profiles/other-profile aterm
    
  copies the `aterm' component installed in the `other-profile' to the
  user's current profile.
2005-02-15 10:49:31 +00:00
Eelco Dolstra
8992fce3da * It is now possible to add store derivations or paths directly to a
user environment, e.g.,

    $ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv

  or 

    $ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1

  This is useful because it allows Nix expressions to be bypassed
  entirely.  For instance, if only a nix-pull manifest is provided,
  plus the top-level path of some component, it can be installed
  without having to supply the Nix expression (e.g., for obfuscation,
  or to be independent of Nix expression language changes or context
  dependencies).
2005-02-14 17:35:10 +00:00
Eelco Dolstra
e446d342b7 * Added an installation source --from-expression' (or -E') to
install derivations from a Nix expression specified on the command
  line.  This is particularly useful for disambiguation if there are
  multiple derivations with the same name.  For instance, in Nixpkgs,
  to install the Firefox wrapper rather than the plain Firefox
  component:

    $ nix-env -f .../i686-linux.nix -i -E 'x: x.firefoxWrapper'

  The Nix expressions should be functions to which the default Nix
  expression (in this case, `i686-linux.nix') is passed, hence `x:
  ...'.

  This might also be a nice way to deal with high-level (user-level)
  variability, e.g.,

    $ nix-env -f ./server.nix -i -E 'x: x {port = 8080; ssl = false;}'
2005-02-14 17:07:43 +00:00
Eelco Dolstra
0cb016c209 * Refactoring. Hope this doesn't break the semantics of `-u' ;-) 2005-02-14 16:16:02 +00:00
Eelco Dolstra
a04a5de8f7 * Implement the `gc-keep-derivations' global configuretion flag. 2005-02-14 14:16:56 +00:00
Eelco Dolstra
6a8ef36fe6 * Global configuration option `env-keep-derivations' to store pointer
to derivations in user environments.  Nice for developers (since it
  prevents build-time-only dependencies from being GC'ed, in
  conjunction with `gc-keep-outputs').  Turned off by default.
2005-02-14 13:07:09 +00:00
Eelco Dolstra
b0aba6ec2a * Don't keep the derivation symlink when creating profile generations. 2005-02-14 10:44:57 +00:00
Eelco Dolstra
32429142cd * Type error in constructor call (caught by GCC 3.3, but not 3.4!). 2005-02-14 09:53:11 +00:00
Eelco Dolstra
20ce2642fc * Refactoring to support different installation sources in nix-env.
* Set the references for the user environment manifest properly.
* Don't copy the manifest (this was accidental).
* Don't store derivation paths in the manifest (maybe this should be
  made optional).  This cleans up the semantics of nix-env, which were
  weird.
* Hash on the output paths of activated components, not on derivation
  paths.  This is because we don't know the derivation path of already
  installed components anymore, and it allows the installation of
  components by store path (skipping Nix expressions entirely).
* Query options `--out-path' and `--drv-path' to show the output and
  derivation paths of components, respectively (the latter replaces
  the `--expr' query).
2005-02-11 16:56:45 +00:00
Eelco Dolstra
80870d9291 * Input sources should be in the set of all referenceable paths too. 2005-02-11 16:03:47 +00:00
Eelco Dolstra
3a99616968 * Commit more often to prevent out-of-memory errors. 2005-02-09 14:37:24 +00:00
Eelco Dolstra
98df735b51 * Propagate the deriver of a path through the substitute mechanism.
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
2005-02-09 12:57:13 +00:00
Eelco Dolstra
582e01c06f * Automatically upgrade <= 0.7 Nix stores to the new schema (so that
existing user environments continue to work).
* `nix-store --verify': detect incomplete closures.
2005-02-09 09:50:29 +00:00
Eelco Dolstra
c547439843 * Subflag in --verify': nix-store --verify --check-contents' checks
that the contents of store paths has not changed by comparing hashes
  of their current contents to the hashes stored in the database.
2005-02-08 13:48:53 +00:00
Eelco Dolstra
3d74274b37 * Updated `nix-store --verify' to the new schema. 2005-02-08 13:23:55 +00:00
Eelco Dolstra
fbc434ee4c * `nix-store -qb' to query derivation environment bindings. Useful
for finding build-time dependencies (possibly after a build).  E.g.,

    $ nix-store -qb aterm $(nix-store -qd $(which strc))
    /nix/store/jw7c7s65n1gwhxpn35j9rgcci6ilzxym-aterm-2.3.1

* Arguments to nix-store can be files within store objects, e.g.,
  /nix/store/jw7c...-aterm-2.3.1/bin/baffle.

* Idem for garbage collector roots.
2005-02-07 14:32:44 +00:00
Eelco Dolstra
450c358e20 * Maintain a database table (`derivers') that maps output paths to the
derivation that produced them.
* `nix-store -qd PATH' prints out the derivation that produced a path.
2005-02-07 13:40:40 +00:00
Eelco Dolstra
a37338815d * A GC setting `gc-keep-outputs' to specify whether output paths of
derivations should be kept.
2005-02-01 22:07:48 +00:00
Eelco Dolstra
65b6c8ab4c * Move root finding from nix-collect-garbage' to nix-store --gc'.
This was necessary becase root finding must be done after
  acquisition of the global GC lock.

  This makes `nix-collect-garbage' obsolete; it is now just a wrapper
  around `nix-store --gc'.

* Automatically remove stale GC roots (i.e., indirect GC roots that
  point to non-existent paths).
2005-02-01 15:05:32 +00:00
Eelco Dolstra
630ae0c9d7 * nix-build: use an indirection scheme to make it easier for users to
get rid of GC roots.  Nix-build places a symlink `result' in the
  current directory.  Previously, removing that symlink would not
  remove the store path being linked to as a GC root.  Now, the GC
  root created by nix-build is actually a symlink in
  `/nix/var/nix/gcroots/auto' to `result'.  So if that symlink is
  removed the GC root automatically becomes invalid (since it can no
  longer be resolved).  The root itself is not automatically removed -
  the garbage collector should delete dangling roots.
2005-02-01 13:48:46 +00:00
Eelco Dolstra
dcc37c236c * nix-store, nix-instantiate: added an option `--add-root' to
immediately add the result as a permanent GC root.  This is the only
  way to prevent a race with the garbage collector.  For instance, the
  old style

    ln -s $(nix-store -r $(nix-instantiate foo.nix)) \
      /nix/var/nix/gcroots/result

  has two time windows in which the garbage collector can interfere
  (by GC'ing the derivation and the output, respectively).  On the
  other hand,

    nix-store --add-root /nix/var/nix/gcroots/result -r \
      $(nix-instantiate --add-root /nix/var/nix/gcroots/drv \
        foo.nix)

  is safe.

* nix-build: use `--add-root' to prevent GC races.
2005-02-01 12:36:25 +00:00
Eelco Dolstra
06b4424286 * Add missing files to dist.
* Fix GC and substitute bugs related to self-references.  Add a
  regression test.
2005-02-01 09:23:38 +00:00
Eelco Dolstra
32fa82a56a * Acquire a global GC lock to prevent new temporary root files from
being created after the garbage collector has read the temproots
  directory.  This blocks the creation of new processes, but the
  garbage collector could periodically release the GC lock to allow
  them to run.
2005-01-31 22:23:49 +00:00
Eelco Dolstra
89c9bc11ab * Add a test for a more subtle race: a process starting after the
temporary root files have been read but creating outputs before the
  store directory has been read.
2005-01-31 22:01:55 +00:00
Eelco Dolstra
207bdcbe86 * Automatically remove temporary root files. 2005-01-31 21:20:59 +00:00
Eelco Dolstra
252c9c91ab * Topologically sort paths under the references relation to ensure
that they are deleted in an order that maintains the closure
  invariant.
* Presence of a path in a temporary roots file does not imply that all
  paths in its closure are also present, so add the closure.
2005-01-31 14:00:43 +00:00
Eelco Dolstra
33c5d23b81 * Don't delete active lock files. 2005-01-31 12:19:53 +00:00
Eelco Dolstra
1328aa3307 * Start of concurrent garbage collection. Processes write temporary
roots to a per-process temporary file in /nix/var/nix/temproots
  while holding a write lock on that file.  The garbage collector
  acquires read locks on all those files, thus blocking further
  progress in other Nix processes, and reads the sets of temporary
  roots.
2005-01-31 10:27:25 +00:00
Eelco Dolstra
22cfdfa246 * Use NIX_STORE environment variable to locate the store (in addition
to NIX_STORE_DIR) so that Nix invocations in builders in `make
  check' work correctly if the store doesn't exist.
2005-01-28 13:19:16 +00:00
Eelco Dolstra
0ea8b6993a * Only invalidate paths when they are in fact valid. 2005-01-28 11:05:46 +00:00
Eelco Dolstra
a85d1849af * Missing dependency; only a problem when building from Subversion. 2005-01-27 19:00:48 +00:00
Eelco Dolstra
8a3eef22e3 * Fix deadlock. 2005-01-27 17:48:14 +00:00
Eelco Dolstra
c60a4943ba * Update referers mappings when updating/clearing the references
mapping.
* Do things in the right order in invalidatePath().
2005-01-27 16:18:39 +00:00
Eelco Dolstra
c505702265 * Fix and simplify the garbage collector (it's still not concurrent,
though).  In particular it's now much easier to register a GC root.
  Just place a symlink to whatever store path it is that you want to
  keep in /nix/var/nix/gcroots.
2005-01-27 15:21:29 +00:00
Eelco Dolstra
59682e6188 * Make lock removal safe by signalling to blocked processes that the
lock they are waiting on has become stale (we do this by writing a
  meaningless token to the unlinked file).
2005-01-27 12:19:25 +00:00
Eelco Dolstra
a24b78e9f1 * Maintain the references/referers relation also for derivations.
This simplifies garbage collection and `nix-store --query
  --requisites' since we no longer need to treat derivations
  specially.

* Better maintaining of the invariants, e.g., setReferences() can only
  be called on a valid/substitutable path.
2005-01-25 21:28:25 +00:00
Eelco Dolstra
2a2756b856 * Simplification: registerSubstitutes -> registerSubstitute. We no
longer need the former since there we no longer have the
  substitutes-rev table (which triggered a O(n^2) cost in updating
  them).
2005-01-25 20:27:40 +00:00
Eelco Dolstra
066da4ab85 * Really fix the substitute mechanism, i.e., ensure the closure
invariant by registering references through the manifest.
* Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
Eelco Dolstra
c6290e42bc * Fix the `--fallback' switch.
* Fix the substitutes tests.
2005-01-25 13:00:12 +00:00
Eelco Dolstra
581fc47783 * Fix the build hook mechanism; pass the pointer graph to the hook. 2005-01-25 11:55:43 +00:00
Eelco Dolstra
52bf9b86bb * In nix-store: added query `--referers-closure' that returns the
closure of the referers relation rather than the references
  relation, i.e., the set of all paths that directly or indirectly
  refer to the given path.  Note that contrary to the references
  closure this set is not fixed; it can change as paths are added to
  or removed from the store.
2005-01-25 11:18:03 +00:00
Eelco Dolstra
80faa2f98a * In nix-store: change --build' back to --realise'. Also brought
back the query flag `--force-realise'.
* Fixed some of the tests.
2005-01-25 10:55:33 +00:00
Eelco Dolstra
6a0a2d5593 * Terminology fixes. 2005-01-20 16:01:07 +00:00
Eelco Dolstra
6bb5efadec * Ensure that derivation names and sources don't end in `.drv'. 2005-01-20 15:25:01 +00:00
Eelco Dolstra
05f0430de1 * Another change to low-level derivations. The last one this year, I
promise :-) This allows derivations to specify on *what* output
  paths of input derivations they are dependent.  This helps to
  prevent unnecessary downloads.  For instance, a build might be
  dependent on the `devel' and `lib' outputs of some library
  component, but not the `docs' output.
2005-01-20 14:10:19 +00:00
Eelco Dolstra
e0f4e587c3 * Nix-store queries --references' and referers' to query the pointer
graph.  That is, `nix-store --query --references PATH' shows the set
  of paths referenced by PATH, and `nix-store --query --referers PATH'
  shows the set of paths referencing PATH.
2005-01-19 16:59:56 +00:00
Eelco Dolstra
96de272b48 * Renamed normalise.cc' -> build.cc', `storeexprs.cc' ->
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
  they have been built/added.  This is so that we can check whether
  the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
  holds.
2005-01-19 16:39:47 +00:00
Eelco Dolstra
ef5f254a55 * `nix-store --build' now builds its arguments in parallel instead of
sequentially (within the limits set by `--jobs').  This should
  greatly improve the utilisation of the build farm when doing Nixpkgs
  builds.
2005-01-19 15:02:02 +00:00
Eelco Dolstra
06c77bf7a8 * Change extension .store' to .drv'.
* Re-enable `nix-store --query --requisites'.
2005-01-19 14:36:00 +00:00
Eelco Dolstra
863dcff6c5 * Started removing closure store expressions, i.e., the explicit
representation of closures as ATerms in the Nix store.  Instead, the
  file system pointer graph is now stored in the Nix database.  This
  has many advantages:

  - It greatly simplifies the implementation (we can drop the notion
    of `successors', and so on).

  - It makes registering roots for the garbage collector much easier.
    Instead of specifying the closure expression as a root, you can
    simply specify the store path that must be retained as a root.
    This could not be done previously, since there was no way to find
    the closure store expression containing a given store path.
    
  - Better traceability: it is now possible to query what paths are
    referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
Eelco Dolstra
e9762e2d10 * Support arities > 6. 2005-01-19 11:04:24 +00:00
Eelco Dolstra
6d493751c3 * Get --readonly-mode to work again. 2005-01-18 11:15:50 +00:00
Eelco Dolstra
32aac8748a * Actually check that the result of fixed-output derivations matches
the specified hash.
2005-01-17 19:01:48 +00:00
Eelco Dolstra
f3dc231250 * Removed the `id' attribute hack.
* Formalise the notion of fixed-output derivations, i.e., derivations
  for which a cryptographic hash of the output is known in advance.
  Changes to such derivations should not propagate upwards through the
  dependency graph.  Previously this was done by specifying the hash
  component of the output path through the `id' attribute, but this is
  insecure since you can lie about it (i.e., you can specify any hash
  and then produce a completely different output).  Now the
  responsibility for checking the output is moved from the builder to
  Nix itself.

  A fixed-output derivation can be created by specifying the
  `outputHash' and `outputHashAlgo' attributes, the latter taking
  values `md5', `sha1', and `sha256', and the former specifying the
  actual hash in hexadecimal or in base-32 (auto-detected by looking
  at the length of the attribute value).  MD5 is included for
  compatibility but should be considered deprecated.

* Removed the `drvPath' pseudo-attribute in derivation results.  It's
  no longer necessary.

* Cleaned up the support for multiple output paths in derivation store
  expressions.  Each output now has a unique identifier (e.g., `out',
  `devel', `docs').  Previously there was no way to tell output paths
  apart at the store expression level.

* `nix-hash' now has a flag `--base32' to specify that the hash should
  be printed in base-32 notation.

* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
  `md5'.

* `nix-prefetch-url' now prints out a SHA-1 hash in base-32.  (TODO: a
  flag to specify the hash.)
2005-01-17 16:55:19 +00:00
Eelco Dolstra
d58a11e019 * Shorten SHA-256 hashes used in store path name generation to 160
bits, then encode them in a radix-32 representation (using digits
  and letters except e, o, u, and t).  This produces store paths like
  /nix/store/4i0zb0z7f88mwghjirkz702a71dcfivn-aterm-2.3.1.  The nice
  thing about this is that the hash part of the file name is still 32
  characters, as before with MD5.

  (Of course, shortening SHA-256 to 160 bits makes it no better than
  SHA-160 in theory, but hopefully it's a bit more resistant to
  attacks; it's certainly a lot slower.)
2005-01-14 16:04:03 +00:00
Eelco Dolstra
9530cc3170 * Start move towards SHA-256 hashes instead of MD5.
* Start cleaning up unique store path generation (they weren't always
  unique; in particular the suffix ("-aterm-2.2", "-builder.sh") was
  not part of the hash, therefore changes to the suffix would cause
  multiple store objects with the same hash).
2005-01-14 13:51:38 +00:00
Eelco Dolstra
63791eb05b * Add SHA-256.
* Tests for the various hashes.
2005-01-14 12:03:04 +00:00
Eelco Dolstra
37b51a9aa6 * Removed some dead code. 2005-01-14 10:16:33 +00:00
Eelco Dolstra
7e8961f720 * Added SHA-1 support. nix-hash' now has an option --type sha1' to
select SHA-1 hashing.
2005-01-13 17:39:26 +00:00
Eelco Dolstra
73992371a3 * Refactoring to support SHA-1. 2005-01-13 15:44:44 +00:00
Eelco Dolstra
a03397be4c * Cygwin compatibility. 2005-01-04 17:38:26 +00:00
Eelco Dolstra
4f07ebc67e * Integrated bsdiff/bspatch 4.2 (from
http://www.daemonology.net/bsdiff/bsdiff-4.2.tar.gz) into the source
  tree.  The license is a bit peculiar, but it does allow verbatim
  copying, which is what we do here (i.e., so don't make any changes
  to the sources).
2004-12-29 22:08:48 +00:00
Eelco Dolstra
8b9697e575 * An operation `nix-store --clear-substitutes' to remove all
registered substitute mappings.
2004-12-20 14:16:55 +00:00
Eelco Dolstra
fa9259f5f5 * Simplify the substitute mechanism:
- Drop the store expression.  So now a substitute is just a
    command-line invocation (a program name + arguments).  If you
    register a substitute you are responsible for registering the
    expression that built it (if any) as a root of the garbage
    collector.
  - Drop the substitutes-rev DB table.
2004-12-20 13:43:32 +00:00
Eelco Dolstra
13f77276d1 * utime() follows symlinks, so don't change the mtime if the file is a
symlink.
2004-11-29 19:22:16 +00:00
Eelco Dolstra
f17553a212 * Remove debug statement. 2004-11-29 15:30:44 +00:00
Eelco Dolstra
4115d8d8ce * Canonicalise metadata of all files in store paths (set the mtime to
0, set the mode to either 444 or 555, set the group to the default).
2004-11-29 15:09:29 +00:00
Eelco Dolstra
54c7a870d5 * Document --delete-generations and other nix-env options. 2004-11-12 23:56:37 +00:00
Eelco Dolstra
c2b0d6b02f * Document --eval-only and --parse-only options in nix-instantiate. 2004-11-12 23:22:08 +00:00
Eelco Dolstra
92ee003dc9 * Fix broken format string. 2004-11-08 15:20:52 +00:00
Eelco Dolstra
cb7ccb528b * string2ATerm -> overloaded toATerm. 2004-11-03 18:12:03 +00:00
Eelco Dolstra
a69534fc21 * Drop ATmake / ATMatcher also in handling store expressions. 2004-10-29 11:22:49 +00:00
Eelco Dolstra
3277c9432a * Bug fix in parsing of /* ... */ comments; due to longest match
regexp there could be only one such comment per file.
2004-10-27 13:00:31 +00:00
Eelco Dolstra
f09618b63a * Turn on read-only mode in queries. This prevents redundant store I/O. 2004-10-27 10:24:44 +00:00
Eelco Dolstra
c7bea941b0 * Oops, I did it again. 2004-10-27 10:05:51 +00:00
Eelco Dolstra
210ab0296d * Add file to `make dist'. 2004-10-27 00:02:31 +00:00
Eelco Dolstra
c52dda95a6 * Bug: check that term is an application. 2004-10-26 23:30:18 +00:00