Commit graph

1529 commits

Author SHA1 Message Date
Eelco Dolstra
2cc345b95f
Give activities a verbosity level again
And print them (separately from the progress bar) given sufficient -v
flags.
2017-08-28 19:13:24 +02:00
Eelco Dolstra
cfc8132391
Don't send progress messages to older clients 2017-08-28 18:54:23 +02:00
Eelco Dolstra
fe34b91289
Tunnel progress messages from the daemon to the client
This makes the progress bar work for non-root users.
2017-08-28 18:49:42 +02:00
Eelco Dolstra
94a0548dc4
Simplify 2017-08-25 21:26:37 +02:00
Eelco Dolstra
0ac35b67b8
Allow derivations to update the build phase
So the progress bar can show

  [1/0/1 built, 0.0 MiB DL] building hello-2.10 (configuring): checking whether pread is declared without a macro... yes
2017-08-25 18:04:05 +02:00
Eelco Dolstra
c137c0a5eb
Allow activities to be nested
In particular, this allows more relevant activities ("substituting X")
to supersede inferior ones ("downloading X").
2017-08-25 17:49:40 +02:00
Eelco Dolstra
0e9ddcc306
Restore activity metadata
This allows the progress bar to display "building perl-5.22.3" instead
of "building /nix/store/<hash>-perl-5.22.3.drv".
2017-08-25 15:58:35 +02:00
Eelco Dolstra
1f56235438
Clean up JSON construction 2017-08-21 12:18:46 +02:00
Eelco Dolstra
4af2611bd1
Allow builders to create activities
Actually, currently they can only create download activities. Thus,
downloads by builtins.fetchurl show up in the progress bar.
2017-08-21 12:18:46 +02:00
Eelco Dolstra
b4ed97e3a3
nix optimise-store: Show how much space has been freed 2017-08-16 20:56:03 +02:00
Eelco Dolstra
23b8b7e096
nix optimise-store: Add
This replaces "nix-store --optimise". Main difference is that it has a
progress indicator.
2017-08-16 20:56:03 +02:00
Eelco Dolstra
40bffe0a43
Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Eelco Dolstra
dff12b38f9
Progress indicator: More improvements 2017-08-16 20:56:03 +02:00
Eelco Dolstra
bf1f123b09
Progress indicator: Show number of active items 2017-08-16 20:56:03 +02:00
Eelco Dolstra
0e0dcf2c7e
Progress indicator: Unify "copying" and "substituting"
They're the same thing after all.

Example:

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 49/98 copied, 16.3/92.8 MiB DL, 55.8/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/0pl9li1jigcj2dany47hpmn0r3r48wc4nz48v5mqhh426lgz3bz6.nar.xz'
2017-08-16 20:56:03 +02:00
Eelco Dolstra
c36467ad2e
Improve substitution progress indicator
E.g.

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-16 20:56:02 +02:00
Eelco Dolstra
b29b6feaba
nix copy: Improve progress indicator
It now shows the amount of data copied:

  [8/1038 copied, 160.4/1590.9 MiB copied] copying path '...'
2017-08-16 20:56:02 +02:00
Eelco Dolstra
c5e4404580
nix copy: Revive progress bar 2017-08-16 20:56:02 +02:00
Eelco Dolstra
e4bd42f98f
Disallow SSH host names starting with a dash 2017-08-16 20:55:58 +02:00
Eelco Dolstra
b6ee5e5bf0
Style fix 2017-08-10 13:51:07 +02:00
Eelco Dolstra
af765a8eab
Use /proc/self/fd to efficiently close all FDs on Linux
Issue #1506.
2017-08-09 16:22:05 +02:00
Eelco Dolstra
a2778988f2
Merge branch 'macOS' of https://github.com/davidak/nix 2017-07-31 10:31:51 +02:00
Jörg Thalheim
2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
davidak
fcb8d6a7a0 replace "OS X" with "macOS" 2017-07-30 12:28:50 +02:00
Eelco Dolstra
fc3568e263
FD_SETSIZE check: BuildError -> Error
BuildError denotes a permanent build failure, which is not the case
here.
2017-07-20 13:33:13 +02:00
Dmitry Kalinkin
d5e1bffd2a
Do not try to fill fd_set with fd>=FD_SETSIZE
This is UB and causes buffer overflow and crash on linux.
2017-07-18 17:51:50 -04:00
Eelco Dolstra
bf6792c0df
Always use base-16 for hashed mirror lookups
In particular, don't use base-64, which we don't support. (We do have
base-32 redirects for hysterical reasons.)

Also, add a test for the hashed mirror feature.
2017-07-17 13:13:18 +02:00
Eelco Dolstra
49304bae81
Make the hashes mirrors used by builtins.fetchurl configurable
In particular, this allows it to be disabled in our tests.
2017-07-17 13:07:08 +02:00
Eelco Dolstra
4ec6eb1fdf
Fix accidental printError 2017-07-17 11:38:15 +02:00
Eelco Dolstra
766ad5db3b
nix path-info: Show download sizes for binary cache stores
E.g.

  $ nix path-info --json --store https://cache.nixos.org nixpkgs.thunderbird -S
  ...
      "downloadHash": "sha256:1jlixpzi225wwa0f4xdrwrqgi47ip1qpj9p06fyxxg07sfmyi4q0",
      "downloadSize": 43047620,
      "closureDownloadSize": 84745960
    }
  ]
2017-07-14 18:29:10 +02:00
Eelco Dolstra
3908d3929c
nix path-info: Don't barf on invalid paths
Now you get

  [
    {
      "path": "/nix/store/fzvliz4j5xzvnd0w5zgw2l0ksqh578yk-bla",
      "valid": false
    }
  ]
2017-07-14 18:29:10 +02:00
Eelco Dolstra
8e8caf7f3e fetchTarball: Prevent concurrent downloads of the same file
Fixes #849.
2017-07-11 23:21:24 +02:00
Eelco Dolstra
d3713716b6 Merge pull request #1445 from matthewbauer/macos-skip-hardlink
Don’t hardlink disallowed paths in OS X.
2017-07-07 11:05:21 +02:00
Matthew Bauer
72e80c59b5 Don’t hardlink disallowed paths in OS X.
Fixes #1443
2017-07-06 19:30:19 -07:00
Eelco Dolstra
a3dc1e65ab
Add X32 to the seccomp filter
Fixes #1432.
2017-07-04 19:00:51 +02:00
Eelco Dolstra
42c5774e78
Sort substituters by priority
Fixes #1438.
2017-07-04 16:34:53 +02:00
Eelco Dolstra
b7203e853e
getDefaultSubstituters(): Simplify initialisation
As shlevy pointed out, static variables in C++11 have thread-safe
initialisation.
2017-07-04 16:26:48 +02:00
Eelco Dolstra
6cf23c3e8f
Add allow-new-privileges option
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.

Example:

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
    cannot raise the capability into the Ambient set
    : Operation not permitted

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
    PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms

Fixes #1429.
2017-07-04 15:48:25 +02:00
Eelco Dolstra
c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra
fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
David McFarland
596b0e0a04 Call SetDllDirectory("") after sqlite3 init on cygwin
Cygwin sqlite3 is patched to call SetDllDirectory("/usr/bin") on init, which
affects the current process and is inherited by child processes.  It causes
DLLs to be loaded from /usr/bin/ before $PATH, which breaks all sorts of
things.  A typical failures would be header/lib version mismatches (e.g.
openssl when running checkPhase on openssh).  We'll just set it back to the
default value.

Note that this is a problem with the cygwin version of sqlite3 (currently
3.18.0).  nixpkgs doesn't have the problematic patch.
2017-06-20 10:59:13 -03:00
Eelco Dolstra
a1355917ec
Disable use of virtual hosting in aws-sdk-cpp
Recently aws-sdk-cpp quietly switched to using S3 virtual host URIs
(https://github.com/aws/aws-sdk-cpp/commit/69d9c53882), i.e. it sends
requests to http://<bucket>.<region>.s3.amazonaws.com rather than
http://<region>.s3.amazonaws.com/<bucket>. However this interacts
badly with curl connection reuse. For example, if we do the following:

1) Check whether a bucket exists using GetBucketLocation.
2) If it doesn't, create it using CreateBucket.
3) Do operations on the bucket.

then 3) will fail for a minute or so with a NoSuchBucket exception,
presumably because the server being hit is a fallback for cases when
buckets don't exist.

Disabling the use of virtual hosts ensures that 3) succeeds
immediately. (I don't know what S3's consistency guarantees are for
bucket creation, but in practice buckets appear to be available
immediately.)
2017-06-19 18:51:33 +02:00
Eelco Dolstra
82a0d614cf
Support creating S3 caches in other regions than us-east-1 2017-06-19 18:51:33 +02:00
Eelco Dolstra
b33621d425
Handle S3Errors::RESOURCE_NOT_FOUND from aws-sdk-cpp
This is returned by recent versions. Also handle NO_SUCH_KEY even
though the library doesn't actually return that at the moment.
2017-06-19 18:51:32 +02:00
Eelco Dolstra
1c969611ba
Suppress "will retry in N ms" for non-retriable errors
Newer versions of aws-sdk-cpp call CalculateDelayBeforeNextRetry()
even for non-retriable errors (like NoSuchKey) whih causes log spam in
hydra-queue-runner.
2017-06-19 18:51:32 +02:00
Eelco Dolstra
00aa7c6705
Show aws-sdk-cpp log messages 2017-06-19 18:51:32 +02:00
Eelco Dolstra
1888f7889b
macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
2017-06-19 14:26:05 +02:00
Eelco Dolstra
b5bdfdef73
macOS: Remove flags
In particular, UF_IMMUTABLE (uchg) needs to be cleared to allow the
path to be garbage-collected or optimised.

See https://github.com/NixOS/nixpkgs/issues/25819.
+       the file from being garbage-collected.
2017-06-19 14:19:21 +02:00
Eelco Dolstra
38b7d55af1
Remove redundant debug line 2017-06-14 13:45:38 +02:00
Eelco Dolstra
88b291ffc4
canonicalisePathMetaData(): Ignore security.selinux attribute
Untested, hopefully fixes #1406.
2017-06-14 11:41:03 +02:00