Commit graph

68 commits

Author SHA1 Message Date
Eelco Dolstra
049c0eb49c nix-daemon: Add trusted-users and allowed-users options
‘trusted-users’ is a list of users and groups that have elevated
rights, such as the ability to specify binary caches. It defaults to
‘root’. A typical value would be ‘@wheel’ to specify all users in the
wheel group.

‘allowed-users’ is a list of users and groups that are allowed to
connect to the daemon. It defaults to ‘*’. A typical value would be
‘@users’ to specify the ‘users’ group.
2014-07-17 16:57:07 +02:00
Eelco Dolstra
276a40b31f Handle case collisions on case-insensitive systems
When running NixOps under Mac OS X, we need to be able to import store
paths built on Linux into the local Nix store. However, HFS+ is
usually case-insensitive, so if there are directories with file names
that differ only in case, then importing will fail.

The solution is to add a suffix ("~nix~case~hack~<integer>") to
colliding files. For instance, if we have a directory containing
xt_CONNMARK.h and xt_connmark.h, then the latter will be renamed to
"xt_connmark.h~nix~case~hack~1". If a store path is dumped as a NAR,
the suffixes are removed. Thus, importing and exporting via a
case-insensitive Nix store is round-tripping. So when NixOps calls
nix-copy-closure to copy the path to a Linux machine, you get the
original file names back.

Closes #119.
2014-07-16 16:02:05 +02:00
Eelco Dolstra
e196eecbe6 Allow $NIX_BUILD_HOOK to be relative to Nix libexec directory 2014-07-11 13:55:06 +02:00
Eelco Dolstra
8504e7d604 allow-arbitrary-code-during-evaluation -> allow-unsafe-native-code-during-evaluation 2014-06-27 11:20:16 +02:00
Shea Levy
d62f46e500 Only add the importNative primop if the allow-arbitrary-code-during-evaluation option is true (default false) 2014-06-24 10:50:03 -04:00
Eelco Dolstra
9f9080e2c0 nix-store -l: Fetch build logs from the Internet
If a build log is not available locally, then ‘nix-store -l’ will now
try to download it from the servers listed in the ‘log-servers’ option
in nix.conf. For instance, if you have:

  log-servers = http://hydra.nixos.org/log

then it will try to get logs from http://hydra.nixos.org/log/<base
name of the store path>. So you can do things like:

  $ nix-store -l $(which xterm)

and get a log even if xterm wasn't built locally.
2014-05-21 17:19:36 +02:00
Eelco Dolstra
de4cdd0d47 Set build-max-jobs to the number of available cores by default
More zero configuration.
2014-05-02 12:51:43 +02:00
Eelco Dolstra
ada3e3fa15 When running as root, use build users by default
This removes the need to have a nix.conf, and prevents people from
accidentally running Nix builds as root.
2014-05-02 12:46:03 +02:00
Eelco Dolstra
1017bd68ea Set up a private /dev/pts in the chroot 2014-02-27 23:35:23 +01:00
Eelco Dolstra
3fd01b171a Set up a minimal /dev in chroots
Not bind-mounting the /dev from the host also solves the problem with
/dev/shm being a symlink to something not in the chroot.
2014-02-27 23:17:53 +01:00
Eelco Dolstra
dcaea042fc Only start download-via-ssh if it's enabled 2014-02-26 13:41:03 +01:00
Shea Levy
df5de9dfd7 Add use-ssh-substituter setting.
It defaults to false and can be overridden by RemoteStore.

Untested currently, just quickly put this together
2014-02-26 13:32:47 +01:00
Eelco Dolstra
dba33d4018 Minor style fixes 2014-02-14 11:48:42 +01:00
Shea Levy
1614603165 Pass in params by const ref
Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 06:49:37 -05:00
Shea Levy
64e23d0a38 Add download-via-ssh substituter
This substituter connects to a remote host, runs nix-store --serve
there, and then forwards substituter commands on to the remote host and
sends their results to the calling program. The ssh-substituter-hosts
option can be specified as a list of hosts to try.

This is an initial implementation and, while it works, it has some
limitations:

* Only the first host is used
* There is no caching of query results (all queries are sent to the
  remote machine)
* There is no informative output (such as progress bars)
* Some failure modes may cause unhelpful error messages
* There is no concept of trusted-ssh-substituter-hosts

Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-08 00:13:33 -05:00
Eelco Dolstra
6b5f89f2cf Drop the dependency on Automake 2013-11-22 19:30:24 +00:00
Eelco Dolstra
89e6781cc5 Make function calls show up in stack traces again
Note that adding --show-trace prevents functions calls from being
tail-recursive, so an expression that evaluates without --show-trace
may fail with a stack overflow if --show-trace is given.
2013-11-12 12:51:59 +01:00
Eelco Dolstra
936f9d45ba Don't apply the CPU affinity hack to nix-shell (and other Perl programs)
As discovered by Todd Veldhuizen, the shell started by nix-shell has
its affinity set to a single CPU.  This is because nix-shell connects
to the Nix daemon, which causes the affinity hack to be applied.  So
we turn this off for Perl programs.
2013-09-06 16:36:56 +02:00
Eelco Dolstra
efe4289464 Add an option to limit the log output of builders
This is mostly useful for Hydra to deal with builders that get stuck
in an infinite loop writing data to stdout/stderr.
2013-09-02 11:58:18 +02:00
Eelco Dolstra
9b11165aec Disable the copy-from-other-stores substituter
This substituter basically cannot work reliably since we switched to
SQLite, since SQLite databases may need write access to open them even
just for reading (and in WAL mode they always do).
2013-06-20 12:01:33 +02:00
Eelco Dolstra
bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra
28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Ludovic Courtès
3e067ac11c Add `Settings::nixDaemonSocketFile'. 2013-02-19 10:19:18 +01:00
Eelco Dolstra
b7629778ef Allow mounting a path in a different location in the chroot
Fixes #24.
2012-12-29 23:04:02 +01:00
Eelco Dolstra
8b8ee53bc7 Add builtin constants ‘langVersion’ and ‘nixVersion’
The integer constant ‘langVersion’ denotes the current language
version.  It gets increased every time a language feature is
added/changed/removed.  It's currently 1.

The string constant ‘nixVersion’ contains the current Nix version,
e.g. "1.2pre2980_9de6bc5".
2012-11-27 13:29:55 +01:00
Eelco Dolstra
f581ce0b0c Don't start copy-from-other-stores if $NIX_OTHER_STORES is unset
Slight optimisation.
2012-11-09 14:58:10 +01:00
Eelco Dolstra
76e88871b2 Templatise tokenizeString() 2012-09-19 15:43:23 -04:00
Eelco Dolstra
6c4ac29917 Disable auto store optimisation for now
I've seen operations like "nix-store --import" take much longer on one
system.  So default to off until I've investigated this a bit further.
2012-09-13 10:28:20 -04:00
Eelco Dolstra
eb7849e3a2 Prevent an injection attack in passing untrusted options to substituters 2012-07-31 18:50:32 -04:00
Eelco Dolstra
90d9c58d4d Pass all --option flags to the daemon 2012-07-31 18:19:44 -04:00
Eelco Dolstra
89a8207029 Add an option ‘build-fallback’ (equivalent to the --fallback flag) 2012-07-31 17:56:02 -04:00
Eelco Dolstra
97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra
d059bf48e4 Pass configuration settings to the substituters
Previously substituters could read nix.conf themselves, but this
didn't take --option flags into account.
2012-07-30 16:09:54 -04:00
Eelco Dolstra
f2bdc87595 Update the other substituters 2012-07-11 18:52:09 -04:00
Eelco Dolstra
11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00
Eelco Dolstra
993fa94fb4 * Move initialisation of variables like nixConfDir from libmain to
libstore so that the Perl bindings can use it as well.  It's vital
  that the Perl bindings use the configuration file, because otherwise
  nix-copy-closure will fail with a ‘database locked’ message if the
  value of ‘use-sqlite-wal’ is changed from the default.
2011-11-22 17:28:41 +00:00
Ludovic Courtès
5c9e9f732d Add support for the build-timeout' and --timeout' options. 2011-06-30 15:19:13 +00:00
Eelco Dolstra
80e722278c * When using the build hook, distinguish between the stderr of the
hook script proper, and the stdout/stderr of the builder.  Only the
  latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
Peter Simons
8b7f8b56f1 Added support for passing an (impure) NIX_BUILD_CORES variable to build expressions.
This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.

The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
2010-06-23 14:34:08 +00:00
Eelco Dolstra
cacff1be88 * No longer block while waiting for a lock on a store path. Instead
poll for it (i.e. if we can't acquire the lock, then let the main
  select() loop wait for at most a few seconds and then try again).
  This improves parallelism: if two nix-store processes are both
  trying to build a path at the same time, the second one shouldn't
  block; it should first see if it can build other goals.  Also, it
  prevents the deadlocks that have been occuring in Hydra lately,
  where a process waits for a lock held by another process that's
  waiting for a lock held by the first.

  The downside is that polling isn't really elegant, but POSIX doesn't
  provide a way to wait for locks in a select() loop.  The only
  solution would be to spawn a thread for each lock to do a blocking
  fcntl() and then signal the main thread, but that would require
  pthreads.
2009-03-23 01:05:54 +00:00
Eelco Dolstra
60cb7de336 * Allow options from the Nix config file to be overriden from the
command line (e.g. "--option build-use-chroot true").
2009-02-27 11:04:41 +00:00
Eelco Dolstra
652817046b * Revert r13150: now that we use private namespaces for the chroot, we
don't have to put the chroot in /nix/var/nix/chroots anymore.
  They're back in /tmp now.
2008-12-11 17:52:34 +00:00
Eelco Dolstra
a55113411f * Nix daemon: reload the configuration file after forking (NIX-100). 2008-11-20 12:25:11 +00:00
Eelco Dolstra
a519bb0635 * Some somewhat ad hoc mechanism to allow the build farm to monitor
build progress.
2008-11-12 11:08:27 +00:00
Eelco Dolstra
709b55ee02 * Put the chroots under /nix/var/nix/chroots to reduce the risk of
disasters involving `rm -rf' on bind mounts.  Will try the
  definitive fix (per-process mounts, apparently possible via the
  CLONE_NEWNS flag in clone()) some other time.
2008-10-29 15:34:48 +00:00
Eelco Dolstra
c370755583 * Flag `--no-build-hook' to disable distributed builds.
* queryDeriver in daemon mode: don't barf if the other side returns an
  empty string (which means there is no deriver).
2007-11-16 16:15:26 +00:00
Eelco Dolstra
9e975458b4 * Get rid of the substitutes database table (NIX-47). Instead, if we
need any info on substitutable paths, we just call the substituters
  (such as download-using-manifests.pl) directly.  This means that
  it's no longer necessary for nix-pull to register substitutes or for
  nix-channel to clear them, which makes those operations much faster
  (NIX-95).  Also, we don't have to worry about keeping nix-pull
  manifests (in /nix/var/nix/manifests) and the database in sync with
  each other.

  The downside is that there is some overhead in calling an external
  program to get the substitutes info.  For instance, "nix-env -qas"
  takes a bit longer.

  Abolishing the substitutes table also makes the logic in
  local-store.cc simpler, as we don't need to store info for invalid
  paths.  On the downside, you cannot do things like "nix-store -qR"
  on a substitutable but invalid path (but nobody did that anyway).

* Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
Eelco Dolstra
9dbfe242e3 * Kill a build if it has gone for more than a certain number of
seconds without producing output on stdout or stderr (NIX-65).  This
  timeout can be specified using the `--max-silent-time' option or the
  `build-max-silent-time' configuration setting.  The default is
  infinity (0).

* Fix a tricky race condition: if we kill the build user before the
  child has done its setuid() to the build user uid, then it won't be
  killed, and we'll potentially lock up in pid.wait().  So also send a
  conventional kill to the child.
2006-12-08 15:44:00 +00:00
Eelco Dolstra
1e16d20655 * Install the worker in bindir, not libexecdir.
* Allow the worker path to be overriden through the NIX_WORKER
  environment variable.
2006-12-04 13:09:16 +00:00
Eelco Dolstra
75068e7d75 * Use a proper namespace.
* Optimise header file usage a bit.
* Compile the parser as C++.
2006-09-04 21:06:23 +00:00