Commit graph

100 commits

Author SHA1 Message Date
Vincent Ambo
0000b956bb feat(server): Log Nix output live during the builds
Instead of dumping all Nix output as one at the end of the build
process, stream it live as the lines come in.

This is a lot more useful for debugging stuff like where manifest
retrievals get stuck.
2019-09-21 15:25:28 +01:00
Vincent Ambo
c391a7b7f8 fix(build-image): Use absolute paths in tarballs 2019-09-21 15:25:28 +01:00
Vincent Ambo
da6fd1d79e fix(build): Ensure nixery-build-image is on Nixery's PATH
This is useful when running Nixery locally.
2019-09-21 12:44:40 +01:00
Vincent Ambo
f0b69638e1 chore(build): Bump nixpkgs version used in Travis
This version matches the updated popularity URL.
2019-09-21 12:44:40 +01:00
Vincent Ambo
64f74abc4d feat: Add configuration option for popularity data URL 2019-09-21 12:44:40 +01:00
Vincent Ambo
e4d03fdb17 chore(docs): Remove mdbook override
The change has been upstreamed in Nixpkgs.
2019-09-21 12:44:40 +01:00
Vincent Ambo
5a002fe067 refactor(builder): Calculate image cache key only once 2019-09-10 11:32:37 +01:00
Vincent Ambo
4a58b0ab4d feat(server): Cache built manifests to the GCS bucket
Caches manifests under `manifests/$cacheKey` in the GCS bucket and
introduces two-tiered retrieval of manifests from the caches (local
first, bucket second).

There is some cleanup to be done in this code, but the initial version
works.
2019-09-10 11:32:37 +01:00
Vincent Ambo
051eb77b3d refactor(server): Use package source specific cache keys
Use the PackageSource.CacheKey function introduced in the previous
commit to determine the key at which a manifest should be cached in
the local cache.

Due to this change, manifests for moving target sources are no longer
cached and the recency threshold logic has been removed.
2019-09-10 11:32:37 +01:00
Vincent Ambo
980f5e2187 refactor(server): Move package source management logic to server
Introduces three new types representing each of the possible package
sources and moves the logic for specifying the package source to the
server.

Concrete changes:

* Determining whether a specified git reference is a commit vs. a
  branch/tag is now done in the server, and is done more precisely by
  using a regular expression.

* Package sources now have a new `CacheKey` function which can be used
  to retrieve a key under which a build manifest can be cached *if*
  the package source is not a moving target (i.e. a full git commit
  hash of either nixpkgs or a private repository).

  This function is not yet used.

* Users *must* now specify a package source, Nixery no longer defaults
  to anything and will fail to launch if no source is configured.
2019-09-10 11:32:37 +01:00
Vincent Ambo
496a4ab847 docs: Add information about NIX_TIMEOUT variable 2019-09-02 23:44:57 +01:00
Vincent Ambo
32b9b5099e feat(server): Add configuration option for Nix build timeouts
Adds a NIX_TIMEOUT environment variable which can be set to a number
of seconds that is the maximum allowed time each Nix builder can run.

By default this is set to 60 seconds, which should be plenty for most
use-cases as Nixery is not expected to be performing builds of
uncached binaries in most production cases.

Currently the errors Nix throws on a build timeout are not separated
from other types of errors, meaning that users will see a generic 500
server error in case of a timeout.

This fixes #47
2019-09-02 23:44:57 +01:00
Vincent Ambo
ce8635833b refactor(build-image): Remove implicit import of entire package set
Explicitly refer to where things come from, and also don't import
dockerTools as it is no longer used for anything.
2019-09-02 00:08:11 +01:00
Vincent Ambo
92270fcbe4 refactor(build-image): Simplify customisation layer builder
Moves the relevant parts of the customisation layer construction from
dockerTools.mkCustomisationLayer into the Nixery code base.

The version in dockerTools builds additional files (including via
hashing of potentially large files) which are not required when
serving an image over the registry protocol.
2019-09-02 00:08:11 +01:00
Vincent Ambo
306e12787a chore(build): Add iana-etc to Nixery's own image
This package is used by a variety of programs that users may want to
embed into Nixery in addition, for example cachix, but those packages
don't refer to it explicitly.
2019-08-21 10:35:32 +01:00
Vincent Ambo
bb5427a47a chore(docs): Update embedded nix-1p version
The new version of the document has syntactic fixes that render pipes
in code blocks in tables correctly across dialects.

Fixes #44
2019-08-21 10:35:32 +01:00
Vincent Ambo
daa6196c2a fix(build): Force nix-env to use NIX_PATH
Thanks to clever!
2019-08-19 01:56:17 +01:00
Vincent Ambo
ccf6a95f94 chore(build): Pin nixpkgs to a specific commit
This is the same commit for which Nixery has popularity data, but that
isn't particularly relevant.
2019-08-19 01:56:17 +01:00
Vincent Ambo
e7d7f73f7d feat(build): Add 'extraPackages' parameter
This makes it possible to inject additional programs (e.g. Cachix)
into a Nixery container.
2019-08-19 01:56:17 +01:00
Vincent Ambo
ffae282eac fix(docs): Correct link to layering strategy 2019-08-19 01:56:17 +01:00
Vincent Ambo
745b7ce0b8 fix(build): Ensure root user is known inside of container
This is required by git in cases where Nixery is configured with a
custom git repository.

I've also added a shell back into the image to make debugging a
running Nixery easier. It turns out some of the dependencies already
pull in bash anyways, so this is just surfacing it to $PATH.
2019-08-17 10:42:13 +01:00
Vincent Ambo
9a95c4124f fix(server): Sort requested packages in image name & spec
Before this change, Nixery would pass on the image name unmodified to
Nix which would lead it to cache-bust the manifest and configuration
layers for images that are content-identical but have different
package ordering.

This fixes #38.
2019-08-17 10:31:41 +01:00
Vincent Ambo
0ee239874b docs(README): Update links to layering strategy 2019-08-17 10:10:41 +01:00
Florian Klink
3b65fc8c72 feat(server): add iana-etc and cacert to the shell convenience package
These probably should be part of every container image by default, but
adding it to the "shell" convenience name probably is our best bet for
now.
2019-08-16 22:47:22 +01:00
Vincent Ambo
501e6ded5f fix(build): Ensure GCS signing is used in CI 2019-08-15 16:04:34 +01:00
Vincent Ambo
3f232e0170 docs: Add asciinema demo to README & book 2019-08-15 16:04:34 +01:00
Vincent Ambo
0ec369d76c docs(book): Update information on new layering strategy 2019-08-15 16:04:34 +01:00
Vincent Ambo
ca1ffb397d feat(build): Add an integration test that runs on Travis
This test, after performing the usual Nixery build, loads the built
image into Docker, runs it, pulls an image from Nixery and runs that
image.

To make this work, there is some configuration on the Travis side.
Most importantly, the following environment variables have special
values:

* `GOOGLE_KEY`: This is set to a base64-encoded service account key to
  be used in the test.
* `GCS_SIGNING_PEM`: This is set to a base64-encoded signing key (in
  PEM) that is used for signing URLs.

Both of these are available to all branches in the Nixery repository.
2019-08-15 00:26:57 +01:00
Vincent Ambo
85b9c30749 chore(server): Add 'go vet' to build process 2019-08-14 20:18:41 +01:00
Vincent Ambo
36d50d1f19 fix(server): Print correct project name during startup
They grow up so fast :')
2019-08-14 20:18:41 +01:00
Vincent Ambo
cf227c153f feat(builder): Implement build cache for manifests & layers
Implements a cache that keeps track of:

a) Manifests that have already been built (for up to 6 hours)
b) Layers that have already been seen (and uploaded to GCS)

This significantly speeds up response times for images that are full
or partial matches with previous images served by an instance.
2019-08-14 20:18:41 +01:00
Vincent Ambo
58380e3313 refactor(server): Extract build logic into separate module
This module is going to get more complex as the implementation of #32
progresses.
2019-08-14 20:18:41 +01:00
Vincent Ambo
d9168e3e4d refactor(build-image): Extract package set loading into helper
Some upcoming changes might require the Nix build to be split into
multiple separate nix-build invocations of different expressions, thus
splitting this out is useful.

It also fixes an issue where `build-image/default.nix` might be called
in an environment where no Nix channels are configured.
2019-08-14 00:02:04 +01:00
Vincent Ambo
3939722063 style: Apply nixfmt to trivial Nix files
ALl the ones except for build-image.nix are considered trivial. On the
latter, nixfmt makes some useful changes but by-and-large it is not
ready for that code yet.
2019-08-14 00:02:04 +01:00
Vincent Ambo
6285cd8dbf feat(build-image): Use new image layering algorithm for images
Removes usage of the old layering algorithm and replaces it with the
new one.

Apart from the new layer layout this means that each layer is now
built in a separate derivation, which hopefully leads to better
cacheability.
2019-08-14 00:02:04 +01:00
Vincent Ambo
7214d0aa4f feat(build-image): Introduce a terrifying hack to build group-layers
The issue is described in detail in a comment in
`build-image/default.nix`, please read it.
2019-08-14 00:02:04 +01:00
Vincent Ambo
f60f702274 feat: Add shell.nix for running a local Nixery 2019-08-14 00:02:04 +01:00
Vincent Ambo
6035bf36eb feat(popcount): Clean up popularity counting script
Adds the script used to generate the popularity information for all of
nixpkgs.

The README lists the (currently somewhat rough) usage instructions.
2019-08-14 00:02:04 +01:00
Vincent Ambo
6d718bf271 refactor(server): Use wrapper script to avoid path dependency
Instead of requiring the server component to be made aware of the
location of the Nix builder via environment variables, this commit
introduces a wrapper script for the builder that can simply exist on
the builders $PATH.

This is one step towards a slightly nicer out-of-the-box experience
when using `nix-build -A nixery-bin`.
2019-08-14 00:02:04 +01:00
Vincent Ambo
819b460278 chore(docs): Move mdBook derivation to docs/default.nix 2019-08-14 00:02:04 +01:00
Vincent Ambo
1fa93fe6f6 refactor: Move registry server to a subfolder 2019-08-14 00:02:04 +01:00
Vincent Ambo
d699f7f91c chore(build): Update Go dependencies & add gonum 2019-08-14 00:02:04 +01:00
Vincent Ambo
56a426952c feat(group-layers): Finish layering algorithm implementation
This commit adds the actual logic for extracting layer groupings and
merging them until the layer budget is satisfied.

The implementation conforms to the design doc as of the time of this
commit.
2019-08-14 00:02:04 +01:00
Vincent Ambo
590ce994bb feat(group-layers): Add initial popcount scripts
This script generates an entry in a text file for each time a
derivation is referred to by another in nixpkgs.

For initial testing, this output can be turned into group-layers
compatible JSON with this *trivial* invocation:

cat output | awk '{ print "{\"" $2 "\":" $1 "}"}' | jq -s '. | add | with_entries(.key |= sub("/nix/store/[a-z0-9]+-";""))' > test-data.json
2019-08-14 00:02:04 +01:00
Vincent Ambo
92078527db feat(group-layers): Add preliminary size & popularity considerations
As described in the design document, this adds considerations for
closure size and popularity. All closures meeting a certain threshold
for either value will have an extra edge from the image root to
themselves inserted in the graph, which will cause them to be
considered for inclusion in a separate layer.

This is preliminary because popularity is considered as a boolean
toggle (the input I generated only contains the top ~200 most popular
packages), but it should be using either absolute popularity values or
percentiles (needs some experimentation).
2019-08-14 00:02:04 +01:00
Vincent Ambo
ce31598f42 feat(group-layers): Implement first half of new layering strategy
The strategy is described in-depth in the comment at the top of the
implementation file, as well as in the design document:

https://storage.googleapis.com/nixdoc/nixery-layers.html
2019-08-14 00:02:04 +01:00
Vincent Ambo
3e385dc379 docs: Update embedded nix-1p
The new commit has an operator table, which is nice to have!
2019-08-08 21:15:50 +01:00
Vincent Ambo
c727b3ca9e chore(nix): Increase maximum number of layers to 96
This uses a significantly larger percentage of the total available
layers (125) than before, which means that cache hits for layers
become more likely between images.
2019-08-08 21:15:50 +01:00
Vincent Ambo
993fda3377 fix(go): Fix breakage in unsigned URLs
This affected the public instance which is still running without URL
signing. Should add some monitoring!
2019-08-08 18:58:23 +01:00
Vincent Ambo
12a853fab7 docs: Minor fixes to README after new website release 2019-08-05 01:26:19 +01:00