tvl-depot/doc/manual/advanced-topics/distributed-builds.xml

<chapter xmlns="http://docbook.org/ns/docbook"
      xmlns:xlink="http://www.w3.org/1999/xlink"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      version="5.0"
      xml:id='chap-distributed-builds'>

<title>Distributed Builds</title>

<para>Nix supports distributed builds, where a local Nix installation can
forward Nix builds to other machines over the network.  This allows
multiple builds to be performed in parallel (thus improving
performance) and allows Nix to perform multi-platform builds in a
semi-transparent way.  For instance, if you perform a build for a
<literal>x86_64-darwin</literal> on an <literal>i686-linux</literal>
machine, Nix can automatically forward the build to a
<literal>x86_64-darwin</literal> machine, if available.</para>

<para>You can enable distributed builds by setting the environment
variable <envar>NIX_BUILD_HOOK</envar> to point to a program that Nix
will call whenever it wants to build a derivation.  The build hook
(typically a shell or Perl script) can decline the build, in which Nix
will perform it in the usual way if possible, or it can accept it, in
which case it is responsible for somehow getting the inputs of the
build to another machine, doing the build there, and getting the
results back.  The details of the build hook protocol are described in
the documentation of the <link
linkend="envar-build-hook"><envar>NIX_BUILD_HOOK</envar>
variable</link>.</para>

<example xml:id='ex-remote-systems'><title>Remote machine configuration:
<filename>remote-systems.conf</filename></title>
<programlisting>
nix@mcflurry.labs.cs.uu.nl  x86_64-darwin   /home/nix/.ssh/id_quarterpounder_auto  2
nix@scratchy.labs.cs.uu.nl  i686-linux      /home/nix/.ssh/id_scratchy_auto        8 1 kvm
nix@itchy.labs.cs.uu.nl     i686-linux      /home/nix/.ssh/id_scratchy_auto        8 2
nix@poochie.labs.cs.uu.nl   i686-linux      /home/nix/.ssh/id_scratchy_auto        8 2 kvm perf
</programlisting>
</example>

<para>Nix ships with a build hook that should be suitable for most
purposes.  It uses <command>ssh</command> and
<command>nix-copy-closure</command> to copy the build inputs and
outputs and perform the remote build.  To use it, you should set
<envar>NIX_BUILD_HOOK</envar> to
<filename><replaceable>prefix</replaceable>/libexec/nix/build-remote.pl</filename>.
You should also define a list of available build machines and point
the environment variable <envar>NIX_REMOTE_SYSTEMS</envar> to
it. <envar>NIX_REMOTE_SYSTEMS</envar> must be an absolute path. An
example configuration is shown in <xref linkend='ex-remote-systems'
/>.  Each line in the file specifies a machine, with the following
bits of information:

<orderedlist>

  <listitem><para>The name of the remote machine, with optionally the
  user under which the remote build should be performed.  This is
  actually passed as an argument to <command>ssh</command>, so it can
  be an alias defined in your
  <filename>~/.ssh/config</filename>.</para></listitem>

  <listitem><para>A comma-separated list of Nix platform type
  identifiers, such as <literal>x86_64-darwin</literal>.  It is
  possible for a machine to support multiple platform types, e.g.,
  <literal>i686-linux,x86_64-linux</literal>.</para></listitem>

  <listitem><para>The SSH private key to be used to log in to the
  remote machine.  Since builds should be non-interactive, this key
  should not have a passphrase!</para></listitem>

  <listitem><para>The maximum number of builds that
  <filename>build-remote.pl</filename> will execute in parallel on the
  machine.  Typically this should be equal to the number of CPU cores.
  For instance, the machine <literal>itchy</literal> in the example
  will execute up to 8 builds in parallel.</para></listitem>

  <listitem><para>The “speed factor”, indicating the relative speed of
  the machine.  If there are multiple machines of the right type, Nix
  will prefer the fastest, taking load into account.</para></listitem>

  <listitem><para>A comma-separated list of <emphasis>supported
  features</emphasis>.  If a derivation has the
  <varname>requiredSystemFeatures</varname> attribute, then
  <filename>build-remote.pl</filename> will only perform the
  derivation on a machine that has the specified features.  For
  instance, the attribute

<programlisting>
requiredSystemFeatures = [ "kvm" ];
</programlisting>

  will cause the build to be performed on a machine that has the
  <literal>kvm</literal> feature (i.e., <literal>scratchy</literal> in
  the example above).</para></listitem>

  <listitem><para>A comma-separated list of <emphasis>mandatory
  features</emphasis>.  A machine will only be used to build a
  derivation if all of the machine’s mandatory features appear in the
  derivation’s <varname>requiredSystemFeatures</varname> attribute.
  Thus, in the example, the machine <literal>poochie</literal> will
  only do derivations that have
  <varname>requiredSystemFeatures</varname> set to <literal>["kvm"
  "perf"]</literal> or <literal>["perf"]</literal>.</para></listitem>

</orderedlist>

You should also set up the environment variable
<envar>NIX_CURRENT_LOAD</envar> to point at a directory (e.g.,
<filename>/var/run/nix/current-load</filename>) that
<filename>build-remote.pl</filename> uses to remember how many builds
it is currently executing remotely.  It doesn't look at the actual
load on the remote machine, so if you have multiple instances of Nix
running, they should use the same <envar>NIX_CURRENT_LOAD</envar>
file.  Maybe in the future <filename>build-remote.pl</filename> will
look at the actual remote load.</para>

</chapter>