Commit graph

18730 commits

Author SHA1 Message Date
Eelco Dolstra
98193bb440 Remove RPM builds that don't evaluate 2012-07-18 16:50:56 -04:00
Eelco Dolstra
b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra
58337e0e61 Set release date 2012-07-18 11:51:27 -04:00
Eelco Dolstra
aa115e22df download-from-binary-cache: Print correct URL 2012-07-18 11:01:17 -04:00
Eelco Dolstra
fe241ece29 Merge branch 'master' into no-manifests 2012-07-18 10:47:59 -04:00
Eelco Dolstra
ccc52adfb2 Add function queryPathFromHashPart()
To implement binary caches efficiently, Hydra needs to be able to map
the hash part of a store path (e.g. "gbg...zr7") to the full store
path (e.g. "/nix/store/gbg...kzr7-subversion-1.7.5").  (The binary
cache mechanism uses hash parts as a key for looking up store paths to
ensure privacy.)  However, doing a search in the Nix store for
/nix/store/<hash>* is expensive since it requires reading the entire
directory.  queryPathFromHashPart() prevents this by doing a cheap
database lookup.
2012-07-17 18:55:39 -04:00
Eelco Dolstra
220818f758 queryPathInfo(): return hash in base-32 if desired
Cherry-picked from the no-manifests branch.
2012-07-17 16:55:45 -04:00
Eelco Dolstra
a6f348599c Print some debug output 2012-07-17 16:19:40 -04:00
Eelco Dolstra
3a9fdf2747 Return an exit code of 100 for cached failed builds
Exit code 100 should be returned for all permanent failures.  This
includes cached failures.

Fixes #34.
2012-07-17 15:55:30 -04:00
Eelco Dolstra
1217204c81 Remove dead code 2012-07-17 14:07:52 -04:00
Eelco Dolstra
51d71ad3d7 Manual: Don't claim we support Cygwin 2012-07-17 11:49:47 -04:00
Eelco Dolstra
6c01fb4d68 Update Nix 1.1 release notes 2012-07-17 10:06:20 -04:00
Eelco Dolstra
53b24f3518 Allow disabling log compression 2012-07-17 09:40:12 -04:00
Eelco Dolstra
a7a43adb79 builtins.storePath: resolve symlinks
Needed for Charon/Hydra interaction.
2012-07-12 18:25:01 -04:00
Eelco Dolstra
04559a0d45 Merge branch 'master' of github.com:NixOS/nix into no-manifests 2012-07-11 18:53:27 -04:00
Eelco Dolstra
e4d6bcb6cd Update release notes 2012-07-11 18:52:23 -04:00
Eelco Dolstra
f2bdc87595 Update the other substituters 2012-07-11 18:52:09 -04:00
Eelco Dolstra
15c15da482 Add some missing --version switches 2012-07-11 18:07:41 -04:00
Eelco Dolstra
d287b62b64 Set the User-Agent header to "Nix/<version>" 2012-07-11 18:05:30 -04:00
Eelco Dolstra
b74d92755d download-from-binary-cache: Use HEAD requests if possible
In "nix-env -qas", we don't need the substitute info, we just need to
know if it exists.  This can be done using a HTTP HEAD request, which
saves bandwidth.

Note however that curl currently has a bug that prevents it from
reusing HTTP connections if HEAD requests return a 404:

https://sourceforge.net/tracker/?func=detail&aid=3542731&group_id=976&atid=100976

Without the patch attached to the issue, using HEAD is actually quite
a bit slower than GET.
2012-07-11 17:53:20 -04:00
Eelco Dolstra
09a6321aeb Replace hasSubstitutes() with querySubstitutablePaths()
querySubstitutablePaths() takes a set of paths, so this greatly
reduces daemon <-> client latency.
2012-07-11 17:52:18 -04:00
Eelco Dolstra
58ef4d9a95 Add a function queryValidPaths()
queryValidPaths() combines multiple calls to isValidPath() in one.
This matters when using the Nix daemon because it reduces latency.
For instance, on "nix-env -qas \*" it reduces execution time from 5.7s
to 4.7s (which is indistinguishable from the non-daemon case).
2012-07-11 11:08:47 -04:00
Eelco Dolstra
667d5f1936 Rename queryValidPaths() to queryAllValidPaths() 2012-07-11 10:49:04 -04:00
Eelco Dolstra
eb3036da87 Implement querySubstitutablePathInfos() in the daemon
Also removed querySubstitutablePathInfo().
2012-07-11 10:43:24 -04:00
Eelco Dolstra
6586414bc7 nix-env: Determine which paths have substitutes in parallel 2012-07-11 10:14:06 -04:00
Eelco Dolstra
5ee8944155 Cleanup 2012-07-11 10:13:16 -04:00
Eelco Dolstra
eae802459d Pass --insecure to curl so that https works 2012-07-09 15:49:20 -04:00
Eelco Dolstra
2dd3117c27 Inline fetchurl.sh 2012-07-09 15:48:55 -04:00
Eelco Dolstra
51f9f9924b Add a test for the fetchurl function 2012-07-09 15:41:43 -04:00
Shea Levy
035aa11403 Remove obsolete comment 2012-07-09 15:29:49 -04:00
Shea Levy
a2865f6b3d corepkgs/fetchurl: Build locally and outside of the chroot 2012-07-09 15:29:49 -04:00
Shea Levy
53f52c2111 corepkgs/fetchurl: the 'system' argument can be optional 2012-07-09 15:29:49 -04:00
Shea Levy
543bf742c9 corepkgs: distribute fetchurl files 2012-07-09 15:29:49 -04:00
Shea Levy
f863673a90 corepkgs/fetchurl: Call the shell directly instead of using the shebang 2012-07-09 15:29:49 -04:00
Shea Levy
a994eb92a4 corepkgs/fetchurl.sh: Use config.nix's curl 2012-07-09 15:29:49 -04:00
Shea Levy
9d94a28bed The fetchurl builder is now fetchurl.sh 2012-07-09 15:29:49 -04:00
Shea Levy
fd2630e1f7 Remove old fetchurl makefile 2012-07-09 15:29:49 -04:00
Shea Levy
6450f5699f Move fetchurl files out of their subdirectory 2012-07-09 15:29:49 -04:00
Shea Levy
40c01ec467 corepkgs/config.nix.in: We'll need curl 2012-07-09 15:29:49 -04:00
Shea Levy
c4df747267 Resurrect old corepkgs fetchurl 2012-07-09 15:29:49 -04:00
Eelco Dolstra
27f0c34390 Really fix RPM builds 2012-07-09 13:16:09 -04:00
Eelco Dolstra
ae72be1b8b Add WWW::Curl as a dependency 2012-07-09 13:11:37 -04:00
Eelco Dolstra
a560124cdf Fix RPM builds
http://hydra.nixos.org/build/2784908
2012-07-09 11:58:12 -04:00
Eelco Dolstra
087dee6e1b Get rid of nix.conf.example
No need to duplicate the nix.conf manpage.
2012-07-09 11:56:55 -04:00
Eelco Dolstra
5755a5c354 Install a nix.conf manpage 2012-07-09 11:33:38 -04:00
Eelco Dolstra
099125435f download-from-binary-cache: add nix.conf options 2012-07-09 10:57:28 -04:00
Eelco Dolstra
98a423b75a prim_import(): prefetch substitute info in parallel using queryMissing() 2012-07-09 09:59:34 -04:00
Eelco Dolstra
425cc612ad build.cc: Don't use hasSubstitute()
Instead make a single call to querySubstitutablePathInfo() per
derivation output.  This is faster and prevents having to implement
the "have" function in the binary cache substituter.
2012-07-08 18:39:24 -04:00
Eelco Dolstra
400e556b34 Cleanup 2012-07-08 18:39:07 -04:00
Eelco Dolstra
11800e6198 download-from-binary-cache: parallelise fetching of NAR info files
Getting substitute information using the binary cache substituter has
non-trivial latency overhead.  A package or NixOS system configuration
can have hundreds of dependencies, and in the worst case (when the
local info cache is empty) we have to do a separate HTTP request for
each of these.  If the ping time to the server is t, getting N info
files will take tN seconds; e.g., with a ping time of 0.1s to
nixos.org, sequentially downloading 1000 info files (a typical NixOS
config) will take at least 100 seconds.

To fix this problem, the binary cache substituter can now perform
requests in parallel.  This required changing the substituter
interface to support a function querySubstitutablePathInfos() that
queries multiple paths at the same time, and rewriting queryMissing()
to take advantage of parallelism.  (Due to local caching,
parallelising queryMissing() is sufficient for most use cases, since
it's almost always called before building a derivation and thus fills
the local info cache.)

For example, parallelism speeds up querying all 1056 paths in a
particular NixOS system configuration from 116s to 2.6s.  It works so
well because the eccentricity of the top-level derivation in the
dependency graph is only 9.  So we only need 10 round-trips (when
using an unlimited number of parallel connections) to get everything.

Currently we do a maximum of 150 parallel connections to the server.
Thus it's important that the binary cache server (e.g. nixos.org) has
a high connection limit.  Alternatively we could use HTTP pipelining,
but WWW::Curl doesn't support it and libcurl has a hard-coded limit of
5 requests per pipeline.
2012-07-06 19:08:20 -04:00