Commit graph

53 commits

Author SHA1 Message Date
Petr Rockai
f1e5dedb61 perl: Call loadConfFile() in doInit to avoid screwing sqlite journal mode.
If the database is opened through perl bindings (and even though nix.conf has
use-sqlite-wal set to false), the database is automatically converted into WAL
mode. This makes the next nix process to access the database convert it back to
"truncate". If the database is still open at the time in wal mode by the perl
program, this fails and crashes the nix doing the wal -> truncate conversion.
2013-12-20 13:59:52 +01:00
Eelco Dolstra
2d9bb56e55 Fix segfault on Darwin
Ever since SQLite in Nixpkgs was updated to 3.8.0.2, Nix has randomly
segfaulted on Darwin:

  http://hydra.nixos.org/build/6175515
  http://hydra.nixos.org/build/6611038

It turns out that this is because the binary cache substituter somehow
ends up loading two versions of SQLite: the one in Nixpkgs and the
other from /usr/lib/libsqlite3.dylib.  It's not exactly clear why the
latter is loaded, but it appears to be because WWW::Curl indirectly loads
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation,
which in turn seems to load /usr/lib/libsqlite3.dylib.  This leads to
a segfault when Perl exits:

  #0  0x00000001010375f4 in sqlite3_finalize ()
  #1  0x000000010125806e in sqlite_st_destroy ()
  #2  0x000000010124bc30 in XS_DBD__SQLite__st_DESTROY ()
  #3  0x00000001001c8155 in XS_DBI_dispatch ()
  ...
  #14 0x0000000100023224 in perl_destruct ()
  #15 0x0000000100000d6a in main ()
  ...

The workaround is to explicitly load DBD::SQLite before WWW::Curl.
2013-10-24 19:15:52 +02:00
Eelco Dolstra
936f9d45ba Don't apply the CPU affinity hack to nix-shell (and other Perl programs)
As discovered by Todd Veldhuizen, the shell started by nix-shell has
its affinity set to a single CPU.  This is because nix-shell connects
to the Nix daemon, which causes the affinity hack to be applied.  So
we turn this off for Perl programs.
2013-09-06 16:36:56 +02:00
Eelco Dolstra
22144afa8d Don't keep "disabled" substituters running
For instance, it's pointless to keep copy-from-other-stores running if
there are no other stores, or download-using-manifests if there are no
manifests.  This also speeds things up because we don't send queries
to those substituters.
2013-06-20 11:55:15 +02:00
Eelco Dolstra
3a0cc43ac8 build-remote.pl: Properly close the SSH connection between attempts 2013-05-10 02:38:50 +02:00
Lluís Batlle i Rossell
e6c44d166a Fixing the pv position regarding compression
Problem noticed by niksnut.
2013-05-03 11:08:51 +02:00
Lluís Batlle i Rossell
5cc2fc46ec Adding ETA support to the --show-progress in nix-copy-closure
Based on https://github.com/NixOS/nix/pull/6 from shlevy
2013-05-03 11:08:51 +02:00
Eelco Dolstra
bdd4646338 Revert "Prevent config.h from being clobbered"
This reverts commit 28bba8c44f.
2013-03-08 01:24:59 +01:00
Eelco Dolstra
e73d9e9488 Fix annoying Perl 5.16 warnings
I.e.

Subroutine Nix::Store::isValidPath redefined at /nix/store/clfzsf6gi7qh5i9c0vks1ifjam47rijn-perl-5.16.2/lib/perl5/5.16.2/XSLoader.pm line 92.

and so on.
2013-03-08 00:27:32 +01:00
Eelco Dolstra
28bba8c44f Prevent config.h from being clobbered 2013-03-07 23:55:55 +01:00
Eelco Dolstra
8add116acd Nix::Store::derivationFromPath: Return derivation outputs 2013-02-05 16:02:57 +01:00
Eelco Dolstra
9c29a2ed35 Give a better error message if writeFile fails due to permission issues 2012-12-20 12:22:13 +01:00
Eelco Dolstra
a3d6585c5a nix-copy-closure: Add flag ‘--use-substitutes’ 2012-11-23 16:20:16 +01:00
Eelco Dolstra
167e36a5c3 nix-push: Only generate and copy a NAR if it doesn't already exist
This prevents unnecessary and slow rebuilds of NARs that already exist
in the binary cache.
2012-10-17 16:58:05 -04:00
Eelco Dolstra
20582e9ae3 Support xz compression in the download-using-manifests substituter 2012-09-19 17:33:42 -04:00
Eelco Dolstra
b14717ab90 Delete manifests in "nix-channel --remove" or when a binary cache is available 2012-09-13 11:35:46 -04:00
Eelco Dolstra
8b8fe6139e Drop dependency on List::MoreUtils 2012-08-27 11:28:34 -04:00
Eelco Dolstra
babe54bf97 Add missing file 2012-08-27 11:11:30 -04:00
Eelco Dolstra
97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra
d059bf48e4 Pass configuration settings to the substituters
Previously substituters could read nix.conf themselves, but this
didn't take --option flags into account.
2012-07-30 16:09:54 -04:00
Eelco Dolstra
fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00
Eelco Dolstra
ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra
220818f758 queryPathInfo(): return hash in base-32 if desired
Cherry-picked from the no-manifests branch.
2012-07-17 16:55:45 -04:00
Eelco Dolstra
d287b62b64 Set the User-Agent header to "Nix/<version>" 2012-07-11 18:05:30 -04:00
Eelco Dolstra
099125435f download-from-binary-cache: add nix.conf options 2012-07-09 10:57:28 -04:00
Eelco Dolstra
cf49472d60 nix-push: Always generate base-32 hashes 2012-07-02 18:05:57 -04:00
Eelco Dolstra
37f7098464 First attempt at the manifest-less substituter 2012-06-29 18:28:52 -04:00
Eelco Dolstra
4911a10a4e Use XZ compression in binary caches
XZ compresses significantly better than bzip2.  Here are the
compression ratios and execution times (using 4 cores in parallel) on
my /var/run/current-system (3.1 GiB):

  bzip2: total compressed size 849.56 MiB, 30.8% [2m08]
  xz -6: total compressed size 641.84 MiB, 23.4% [6m53]
  xz -7: total compressed size 621.82 MiB, 22.6% [7m19]
  xz -8: total compressed size 599.33 MiB, 21.8% [7m18]
  xz -9: total compressed size 588.18 MiB, 21.4% [7m40]

Note that compression takes much longer.  More importantly, however,
decompression is much faster:

  bzip2: 1m47.274s
  xz -6: 0m55.446s
  xz -7: 0m54.119s
  xz -8: 0m52.388s
  xz -9: 0m51.842s

The only downside to using -9 is that decompression takes a fair
amount (~65 MB) of memory.
2012-06-29 15:24:52 -04:00
Eelco Dolstra
4f7bab7db1 Support building with the Perl XS bindings disabled
Since the Perl bindings require shared libraries, this is required on
platforms such as Cygwin where we do a static build.
2012-05-10 19:03:23 -04:00
Shea Levy
34a85c5405 nix-copy-closure: Move the progressViewer directly adjacent to the ssh call so that network progress is what's measured 2012-04-13 14:29:12 +02:00
Shea Levy
055e803851 Add the '--show-progress' flag to nix-copy-closure 2012-04-13 14:29:12 +02:00
Eelco Dolstra
209927bb27 Unconfuse Rob 2012-04-10 16:40:51 +02:00
Eelco Dolstra
7f38087f35 Add a command "nix-build --run-env" to reproduce the environment of a derivation
This command builds or fetches all dependencies of the given
derivation, then starts a shell with the environment variables from
the derivation.  This shell also sources $stdenv/setup to initialise
the environment further.

The current directory is not changed.  Thus this is a convenient way
to reproduce a build environment in an existing working tree.

Existing environment variables are left untouched (unless the
derivation overrides them).  As a special hack, the original value of
$PATH is appended to the $PATH produced by $stdenv/setup.

Example session:

$ nix-build --run-env '<nixpkgs>' -A xterm
(the dependencies of xterm are built/fetched...)
$ tar xf $src
$ ./configure
$ make
$ emacs
(... hack source ...)
$ make
$ ./xterm
2012-03-19 04:14:21 +01:00
Eelco Dolstra
d645633643 nix-copy-closure: don't print copied path on stdout
We're already printing progress on stderr, so printing them on stdout
afterwards is kind of useless.
2012-03-05 20:52:24 +01:00
Eelco Dolstra
2b4964f319 Restore progress indication during nix-copy-closure 2012-03-05 18:13:44 +01:00
Eelco Dolstra
6c31232e14 * Sync with the trunk. 2012-01-03 12:59:31 +00:00
Eelco Dolstra
502d94048a * Ignore missing manifest symlinks. 2012-01-03 12:59:00 +00:00
Eelco Dolstra
39d45a6b09 * Add a test for nix-channel.
* Refactor the nix-channel unpacker a bit.
2012-01-03 01:51:38 +00:00
Eelco Dolstra
92d6a5ed73 * Add some more functions to the Perl bindings. 2011-12-02 12:09:24 +00:00
Eelco Dolstra
f35c4351e5 * Don't require a specific Perl version. 2011-11-29 22:15:07 +00:00
Eelco Dolstra
1749a7b0ae * download-using-manifests: use the Perl bindings. 2011-11-29 13:01:24 +00:00
Eelco Dolstra
216440b3ff * For consistency with "nix-store -q --hash", produce hashes in
base-32.  (This affects Hydra manifests.)
2011-11-29 12:32:55 +00:00
Eelco Dolstra
5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra
ab20af3e6f * build-remote.pl: drop a hard-coded reference to /nix/etc/nix. 2011-11-23 12:21:35 +00:00
Eelco Dolstra
993fa94fb4 * Move initialisation of variables like nixConfDir from libmain to
libstore so that the Perl bindings can use it as well.  It's vital
  that the Perl bindings use the configuration file, because otherwise
  nix-copy-closure will fail with a ‘database locked’ message if the
  value of ‘use-sqlite-wal’ is changed from the default.
2011-11-22 17:28:41 +00:00
Eelco Dolstra
f8e609c341 * nix-pull: update the Nix manifest cache if necessary. Also, don't
read the manifest just to check the version and print the number of
  paths.  This makes nix-pull very fast for the cached cache (speeding
  up nixos-rebuild without the ‘--no-pull’ or ‘--fast’ options).
2011-11-16 16:41:48 +00:00
Eelco Dolstra
d7d7910ba4 * Don't decompress the manifests in /nix/var/nix/manifest. This saves
disk space, and, since they're typically only decompressed once (to
  fill the manifest cache), doesn't make things slower.
2011-11-16 16:25:38 +00:00
Eelco Dolstra
63ee5e4d2a * Remove obsolete line. 2011-11-16 11:56:19 +00:00
Eelco Dolstra
a5952405d2 * Re-use prepared statements across insertions into the manifest cache
DB.  This speeds up creating the cache from 16.1s to 7.9s on my
  system.
2011-11-16 11:37:40 +00:00
Eelco Dolstra
6761757428 * Use the Store API bindings in nix-copy-closure. 2011-10-11 15:41:13 +00:00