Commit graph

354 commits

Author SHA1 Message Date
Eelco Dolstra
9ac3f5df9c * Propagate --max-silent-time to remote machines. 2008-12-04 16:51:16 +00:00
Eelco Dolstra
bcfe98acff * Prefer building on a remote machine over a local machine. This
makes more sense for the build farm, otherwise every nix-store
  invocation will lead to at least one local build.  Will come up with
  a better solution later...
2008-12-04 16:35:47 +00:00
Eelco Dolstra
909fbb9de1 * When using build hooks, for any nix-store -r build operation, it is
necessary that at least one build hook doesn't return "postpone",
  otherwise nix-store will barf ("waiting for a build slot, yet there
  are no running children").  So inform the build hook when this is
  the case, so that it can start a build even when that would exceed
  the maximum load on a machine.
2008-12-04 14:29:41 +00:00
Eelco Dolstra
5dfba0b4db * Force allocation of a pseudo-terminal to clean up the remote
nix-store process when the connection is interrupted.
2008-12-04 13:36:52 +00:00
Eelco Dolstra
9fd9c4c635 * Support multiple system types per remote machine, e.g. a machine
list like

    root@example.org x86_64-linux /root/.ssh/id_buildfarm 1
    root@example.org i686-darwin  /root/.ssh/id_buildfarm 1

  This is possible when the Nix installation on example.org itself has
  remote builds enabled.
2008-12-04 12:20:06 +00:00
Eelco Dolstra
f8713e1287 * Dirty hack to make nix-push work properly on derivations: the
derivation should be a source rather than a derivation dependency of
  the call to the NAR derivation.  Otherwise the derivation (and all
  its dependencies) will be built as a side-effect, which may not even
  succeed.
2008-12-04 10:40:41 +00:00
Eelco Dolstra
6bbff48079 2008-11-20 17:22:42 +00:00
Eelco Dolstra
4213b8d8ec * Urgh. 2008-11-20 15:44:59 +00:00
Eelco Dolstra
aab530e971 * Primop builtins.storePath for declaring a store path as a
dependency.  `storePath /nix/store/bla' gives exactly the same
  result as `toPath /nix/store/bla', except that the former includes
  /nix/store/bla in the dependency context of the string.

  Useful in some generated Nix expressions like nix-push, which now
  finally does the right thing wrt distributed builds.  (Previously
  the path to be packed wasn't an explicit dependency, so it wouldn't
  be copied to the remote machine.)
2008-11-19 23:26:19 +00:00
Eelco Dolstra
fa791116a3 * Get rid of nix-pack-closure / nix-unpack-closure, they're redundant. 2008-11-18 14:43:40 +00:00
Eelco Dolstra
2b2aa8a820 * Doh. 2008-09-08 11:02:15 +00:00
Michael Raskin
2a01d06da6 Added nix-http-export.cgi to extra distributed scripts; so it can be installed from releases, not only from SVN. 2008-08-29 08:34:38 +00:00
Michael Raskin
b455c4c45c Updates to nix-reduce-build
Common code in local build package sources refactored out in a function; before building the real set of derivations needed is found (slightly slower for only one build strategy, but less garbage on output and better performance for multiple build strategies).

Now you have full choice of best-effort build regardless of method (substituters or actual build), using substituters, building only fixed derivations (should get you all the downloads) and local build without even trying substituters. 

Some minor fix in the help text about behavior with no package sources.
2008-08-06 19:43:53 +00:00
Eelco Dolstra
7592f48c83 * nix-build: `--dry-run' flag. 2008-08-04 13:46:01 +00:00
Eelco Dolstra
42043953c3 * Doh. 2008-08-04 13:15:47 +00:00
Michael Raskin
b3c26180e3 Updates to nix-reduce-build: only realize fixed derivations if user asks so, or only use substituters. Oh, and add possibility to use : for things like /etc/nixos/nixpkgs:-A:gnused 2008-08-02 16:43:25 +00:00
Eelco Dolstra
3c92ea399d * Make nix-env --dry-run print the paths to be substituted correctly
again.  (After the previous substituter mechanism refactoring I
  didn't update the code that obtains the references of substitutable
  paths.)  This required some refactoring: the substituter programs
  are now kept running and receive/respond to info requests via
  stdin/stdout.
2008-08-02 12:54:35 +00:00
Eelco Dolstra
096198d11f * A quick hack to make nix-prefetch-url support mirror:// URLs. It
requires that $NIXPKGS_ALL points at a Nixpkgs tree.
2008-07-23 16:02:58 +00:00
Eelco Dolstra
660244f65f * Make sure that copy-from-other-stores.pl is built. 2008-07-23 09:38:38 +00:00
Eelco Dolstra
989176c56e * Allow read-only access to the store (e.g., non-root users on NixOS
can do operations like "nix-store -qR <path>" even without the Nix
  daemon).
2008-07-18 15:34:46 +00:00
Eelco Dolstra
8bc591a6f0 * Use the copy-from-other-stores substituter by default. Of course,
it only does something if $NIX_OTHER_STORES (not really a good
  name...) is set.
* Do globbing on the elements of $NIX_OTHER_STORES.  E.g. you could
  set it to /mnts/*/nix or something.
* Install substituters in libexec/nix/substituters.
2008-07-18 13:05:10 +00:00
Eelco Dolstra
15f39aba8c * Quick prototype of a substituter that copies paths from other Nix
stores (typically remote Nix stores mounted via e.g. NFS, or the Nix
  store on the NixOS installation CD).  Example use:

  $ sshfs foo@example.org:/ /mnt
  $ NIX_OTHER_STORES=/mnt/nix \
    NIX_SUBSTITUTERS=.../copy-from-other-stores.pl \
    nix-env -i foo

  This will be especially useful for the installation CD since it
  doesn't require a manifest for the CD contents.
2008-07-12 18:58:24 +00:00
Michael Raskin
955b8841cd Also trying to build derivers in case we cannot get substituters 2008-06-14 08:48:40 +00:00
Michael Raskin
18e27629d3 Added local best-effort builds (i.e. one failure does not ruin all packages you would like to see built) 2008-06-14 08:30:35 +00:00
Michael Raskin
194c66eeeb Stupid error in script 2008-06-13 14:34:19 +00:00
Michael Raskin
f903d86740 OK, I will believe that fix does no worse.. 2008-06-13 13:53:14 +00:00
Michael Raskin
ce85b55cf0 Updated help text 2008-06-12 17:45:38 +00:00
Michael Raskin
4532e4b90d Added verbosity for nix-reduce-build 2008-06-12 16:26:53 +00:00
Michael Raskin
9819bb20da Added support for file:// archive (.nar.gz) repositories to nix-reduce-build. /tmp/nix-export created by nix-http-export.cgi is OK. 2008-05-11 15:54:30 +00:00
Michael Raskin
b4bc8b7616 --proxy=proxy:3128 2008-05-07 14:18:28 +00:00
Michael Raskin
b1e321d6ce Added http alternative transport for nix-reduce-build 2008-04-29 04:03:54 +00:00
Eelco Dolstra
f8985d195e * Fix for NIX-101 (should use an absolute path for call to nix-hash). 2008-03-28 17:52:33 +00:00
Eelco Dolstra
f106868110 * Cleanup. 2008-03-20 10:16:36 +00:00
Eelco Dolstra
d7caac3e81 * Don't install nix-reduce-build by default yet please, I first want
to understand better what it does...
2008-02-28 21:27:47 +00:00
Michael Raskin
11d512e7a8 Added nix-reduce-build. You point it to some path you want to build and it fetches whatever it can from specified computers via nix-copy-closure. NOTE: You do want to set up RSA keys or ssh-agent or something... You really do want it. It will run separate ssh instances insane number of times. 2008-02-27 21:26:47 +00:00
Eelco Dolstra
de012e76e2 * Don't use "store expression", it's obsolete. 2007-12-30 23:35:45 +00:00
Eelco Dolstra
c370755583 * Flag `--no-build-hook' to disable distributed builds.
* queryDeriver in daemon mode: don't barf if the other side returns an
  empty string (which means there is no deriver).
2007-11-16 16:15:26 +00:00
Eelco Dolstra
6784b14241 * nix-build: pass --argstr to nix-instantiate. 2007-11-15 16:52:40 +00:00
Eelco Dolstra
ca210d2a58 * Doh! 2007-11-15 15:33:30 +00:00
Eelco Dolstra
bfec245efa * Add build-remote.pl to the Nix distribution. 2007-11-15 14:28:08 +00:00
Eelco Dolstra
5b74a59570 * nix-prefetch-url: don't fail if /tmp/nix-prefetch-url-<pid> exists,
instead use a counter just like we do for temporary build
  directories.
2007-11-05 18:12:29 +00:00
Eelco Dolstra
f8ed2e64a5 2007-10-22 12:05:18 +00:00
Eelco Dolstra
a8629de827 * New command `nix-store --optimise' to reduce Nix store disk space
usage by finding identical files in the store and hard-linking them
  to each other.  It typically reduces the size of the store by
  something like 25-35%.  This is what the optimise-store.pl script
  did, but the new command is faster and more correct (it's safe wrt
  garbage collection and concurrent builds).
2007-10-09 22:14:27 +00:00
Eelco Dolstra
26f981c2e5 * Remove garbage. 2007-09-18 08:26:55 +00:00
Eelco Dolstra
055608227f * nix-env: allow ~/.nix-defexpr to be a directory. If it is, then the
Nix expressions in that directory are combined into an attribute set
  {file1 = import file1; file2 = import file2; ...}, i.e. each Nix
  expression is an attribute with the file name as the attribute
  name.  Also recurses into directories.

* nix-env: removed the "--import" (-I) option which set the
  ~/.nix-defexpr symlink.

* nix-channel: don't use "nix-env --import", instead symlink
  ~/.nix-defexpr/channels.  So finally nix-channel --update doesn't
  override any default Nix expressions but combines with them.

  This means that you can have (say) a local Nixpkgs SVN tree and use
  it as a default for nix-env:

  $ ln -s .../path-to-nixpkgs-tree ~/.nix-defexpr/nixpkgs_svn

  and be subscribed to channels (including Nixpkgs) at the same time.
  (If there is any ambiguity, the -A flag can be used to
  disambiguate, e.g. "nix-env -i -A nixpkgs_svn.pan".)
2007-09-17 16:08:24 +00:00
Eelco Dolstra
9441550acb * nix-push / generate-patches: bzip the manifest. 2007-09-04 15:38:09 +00:00
Eelco Dolstra
7d5836b34d * nix-channel: supports users who don't have write permission to the
manifests directory.  In that case, we don't do a nix-pull, so the
  user gets pure source deployment.

  The directory /nix/var/nix/gcroots/per-user/$USER should be
  writable.  (It's created automatically if
  /nix/var/nix/gcroots/per-user is writable, e.g. if it has 1777
  permission.)
2007-08-22 14:52:22 +00:00
Eelco Dolstra
dbc0170ed9 * Show errors in nix-prefetch-url. 2007-08-15 09:24:06 +00:00
Eelco Dolstra
ffa1c61cd5 * Fix the tests. 2007-08-14 13:15:59 +00:00
Eelco Dolstra
9e975458b4 * Get rid of the substitutes database table (NIX-47). Instead, if we
need any info on substitutable paths, we just call the substituters
  (such as download-using-manifests.pl) directly.  This means that
  it's no longer necessary for nix-pull to register substitutes or for
  nix-channel to clear them, which makes those operations much faster
  (NIX-95).  Also, we don't have to worry about keeping nix-pull
  manifests (in /nix/var/nix/manifests) and the database in sync with
  each other.

  The downside is that there is some overhead in calling an external
  program to get the substitutes info.  For instance, "nix-env -qas"
  takes a bit longer.

  Abolishing the substitutes table also makes the logic in
  local-store.cc simpler, as we don't need to store info for invalid
  paths.  On the downside, you cannot do things like "nix-store -qR"
  on a substitutable but invalid path (but nobody did that anyway).

* Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
Eelco Dolstra
4695f4edd6 * nix-pull: support bzipped manifests: when doing a nix-pull on $url,
try $url.bz2 first.
2007-08-10 01:42:00 +00:00
Eelco Dolstra
911bc01454 * Enable nix-prefetch-url caching in nix-channel. 2007-08-10 00:28:44 +00:00
Eelco Dolstra
5e52df18fe * Don't rely on /dev/stdin. 2007-08-10 00:22:21 +00:00
Eelco Dolstra
c5e934dcc9 * nix-pull: using nix-prefetch-url (so that we get caching for free),
and store the manifests in the Nix store.  (So now
  /nix/var/nix/manifests/ just contains symlinks to the store and is
  searched for GC roots.)
2007-08-09 23:52:53 +00:00
Eelco Dolstra
f881f7a017 * nix-prefetch-url: support caching. If the environment variable
NIX_DOWNLOAD_CACHE is set, then nix-prefetch-url will store the hash
  and timestamp of downloaded files in the directory
  $NIX_DOWNLOAD_CACHE.  This allows it to figure out if the file is
  still in the Nix store.
2007-08-09 23:16:44 +00:00
Eelco Dolstra
9bff7ad728 * Check against creation of GC roots in the store. Those roots don't
work, because findRoots() stops when it encounters a symlink to the
  store.  And of course the store is supposed to be read-only.
2007-06-11 11:36:22 +00:00
Eelco Dolstra
f3ebd03bb1 * Remove debug statement. 2007-06-04 12:03:33 +00:00
Eelco Dolstra
34d4c9388b 2007-05-02 16:06:04 +00:00
Eelco Dolstra
93aefd9fc0 * Give unpacked channels more sensible names than 0, 1, ... They now
get the basename of the channel URL (e.g., nixpkgs-unstable).  The
  top-level Nix expression of the channel is now an attribute set, the
  attributes of which are the individual channels (e.g.,
  {nixpkgs_unstable = ...; strategoxt_unstable = ...}).  This makes
  attribute paths ("nix-env -qaA" and "nix-env -iA") more sensible,
  e.g., "nix-env -iA nixpkgs_unstable.subversion".
2007-05-01 23:16:38 +00:00
Eelco Dolstra
4caca58ff7 * Make the maximum patch size configurable. 2007-03-30 09:01:05 +00:00
Eelco Dolstra
efd31139df * Forgot a @bindir@. 2007-03-27 09:53:58 +00:00
Eelco Dolstra
d303b389a9 * `nix-copy-closure --from': copy from a remote machine instead of to
a remote machine.
2007-03-26 21:05:17 +00:00
Eelco Dolstra
7edd2e2cd2 * Refactoring. 2007-03-26 20:49:22 +00:00
Eelco Dolstra
a8ea4cbcc8 * Scan /proc/sys/kernel/modprobe for roots to prevent the kernel
modules for the running kernel from being garbage-collected.  Idem
  for /proc/sys/kernel/fbsplash.
2007-03-20 11:13:15 +00:00
Eelco Dolstra
eb2dd4815c * Remove old generations in all directories under
/nix/var/nix/profiles, not just in that directory itself.  (NixOS
  puts profiles in /nix/var/nix/profiles/per-user.)
2007-03-13 11:30:57 +00:00
Eelco Dolstra
df0283ae86 * Get rid of those stupid --login tricks, it's the responsibility of
the remote system to make sure that Nix is in the $PATH.
2007-03-01 13:55:47 +00:00
Eelco Dolstra
30394a4f3f * sh -> bash. 2007-03-01 13:49:20 +00:00
Eelco Dolstra
363e307fd3 * Error message to stdout. 2007-02-26 23:32:10 +00:00
Eelco Dolstra
fa2be32034 * nix-copy-closure: force a login shell on the remote machine to make
sure that nix-store is in the PATH.
* nix-copy-closure: option --gzip to compress data.
2007-02-22 16:42:01 +00:00
Eelco Dolstra
4c5e6d1a2f * nix-copy-closure: option --sign.
* nix-copy-closure: set SSH options through NIX_SSHOPTS..
2007-02-22 15:48:20 +00:00
Eelco Dolstra
024a8ed382 * New command `nix-copy-closure' to copy a closure to a Nix store on
another machine through ssh.  E.g.,

    $ nix-copy-closure xyzzy $(which svn)

  copies the closure of Subversion to machine `xyzzy'.  This is like
  `nix-pack-closure $(which svn) | ssh xyzzy', but it's much more
  efficient since it only copies those paths that are missing on the
  target machine.
2007-02-21 23:14:53 +00:00
Eelco Dolstra
8181a1c3bb * Close the file - just in case. 2007-02-20 22:49:08 +00:00
Eelco Dolstra
bae75ca5a1 * New kind of manifest object: "localPath", which denotes that a store
path can be created by copying it from another location in the file
  system.  This is useful in the NixOS installation.
2007-01-23 16:50:19 +00:00
Eelco Dolstra
36d9258c0d * Successors have been gone for ages. 2007-01-23 16:05:59 +00:00
Eelco Dolstra
7bc30e1ca8 * nix-prefetch-url: change the default hash to SHA-256 (in base-32). 2007-01-22 09:53:36 +00:00
Eelco Dolstra
afe23b5f38 * nix-pack-closure: store the top-level store paths in the closure.
* nix-unpack-closure: extract the top-level paths from the closure and
  print them on stdout.  This allows them to be installed, e.g.,
  "nix-env -i $(nix-unpack-closure)".  (NIX-64)
2007-01-13 19:50:42 +00:00
Eelco Dolstra
215505bb46 * Removed chroot support. 2007-01-13 17:54:01 +00:00
Eelco Dolstra
1b7840b949 2007-01-11 19:28:28 +00:00
Eelco Dolstra
69c8b5b8a7 * Install generate-patches into libexec. 2007-01-11 16:19:45 +00:00
Eelco Dolstra
1f3722bd4a * Reject patches that are larger than a certain fraction of the full archive
(currently 60%).  Large patches aren't very economical.
2007-01-08 15:32:15 +00:00
Eelco Dolstra
50bdec410a * Huge speedup in patch propagation (20 minutes or so to 3 seconds). 2007-01-08 15:17:18 +00:00
Eelco Dolstra
cafaceb707 * Handle weird cases when the server redirects us while setting a cookie. 2006-12-15 21:27:26 +00:00
Eelco Dolstra
9dbfe242e3 * Kill a build if it has gone for more than a certain number of
seconds without producing output on stdout or stderr (NIX-65).  This
  timeout can be specified using the `--max-silent-time' option or the
  `build-max-silent-time' configuration setting.  The default is
  infinity (0).

* Fix a tricky race condition: if we kill the build user before the
  child has done its setuid() to the build user uid, then it won't be
  killed, and we'll potentially lock up in pid.wait().  So also send a
  conventional kill to the child.
2006-12-08 15:44:00 +00:00
Eelco Dolstra
d941186289 * Show more progress. 2006-11-18 19:03:45 +00:00
Eelco Dolstra
b3f916995a * Oops, `nix-build --no-out-link' was broken. 2006-10-31 18:45:17 +00:00
Eelco Dolstra
8d17265ac4 * Don't use EPSV. 2006-10-28 22:07:09 +00:00
Eelco Dolstra
b3d3700e11 * nix-build: check the exit status of `nix-store -r'. 2006-10-19 17:30:09 +00:00
Eelco Dolstra
2a535689fe * Reduce the maximum archive size for patch generation to 100 MB to
prevent trashing on nix.cs.uu.nl.
2006-10-12 20:13:29 +00:00
Eelco Dolstra
d98f750fd8 * tmpnam() -> File::Temp::tempdir(). 2006-10-04 18:58:11 +00:00
Eelco Dolstra
3632019b73 * Quick hack to let nix-install-package set the package name properly
(e.g., "java-front-0.9pre15899" instead of "java-front";
  particularly important when doing upgrades later on).
2006-09-25 14:00:59 +00:00
Eelco Dolstra
d43565c3e8 * In `nix-channel --update', skip manifests that assume a Nix store at
a different location than the user's.  This makes channels usable as
  a source deployment mechanism for people who install Nix under
  non-standard prefixes.  (NIX-57)
2006-09-25 11:11:16 +00:00
Eelco Dolstra
68ae953d8a * Clean up calls to system(). 2006-09-25 10:44:27 +00:00
Eelco Dolstra
76c9710091 * Use builtins.toPath. 2006-09-25 10:29:25 +00:00
Eelco Dolstra
0bd5eb71a0 * `nix-install-package --url': install from a URL (NIX-12).
* `nix-install-package --help' (NIX-9).
* `nix-install-package --non-interactive': don't prompt or pause.
* Tests for nix-install-package.
* Security fixes: filter the values obtained from the nixpkg.
2006-09-21 18:54:08 +00:00
Eelco Dolstra
ee5040421f * Try konsole and gnome-terminal in addition to xterm. 2006-09-21 11:29:14 +00:00
Eelco Dolstra
1b804f88e4 * Absolute path to rm (NIX-51).
* Don't hardcore /nix/bin and /nix/store.
2006-09-20 16:23:14 +00:00
Eelco Dolstra
947e645789 * Hide warnings about a missing "lsof" (NIX-54). 2006-09-20 15:14:19 +00:00
Eelco Dolstra
a060adf165 * Use paths, not strings, when calling the function that generates
NARs.  Fixes the impurity of nix-push (NIX-21).
* Better help.
2006-09-20 15:04:04 +00:00
Eelco Dolstra
7dd342e482 * Doh. 2006-09-19 16:40:22 +00:00
Eelco Dolstra
ee6cf99660 * Doh! Of course we have to take execute permission into account.
* Restore the mtime on modified directories.
2006-09-19 16:14:15 +00:00