Commit graph

134 commits

Author SHA1 Message Date
Vincent Ambo
f4bf3518f6 refactor(server): Replace log calls with logrus
This introduces a structured logging library that can be used (next
step) to attach additional metadata to log entries.
2019-10-06 23:05:23 +01:00
Vincent Ambo
d9b329ef59 refactor(server): Always include 'cacert' & 'iana-etc'
These two packages almost always end up being required by programs,
but people don't necessarily consider them.

They will now always be added and their popularity is artificially
inflated to ensure they end up at the top of the layer list.
2019-10-03 22:50:02 +01:00
Vincent Ambo
9bb6d0ae25 fix(server): Ensure build cache objects are written to GCS
Cache writes might not be flushed without this call.
2019-10-03 22:50:02 +01:00
Vincent Ambo
48a5ecda97 feat(server): Order layers in image manifest based on merge rating
Image layers in manifests are now sorted in a stable (descending)
order based on their merge rating, meaning that layers more likely to
be shared between images come first.

The reason for this change is Docker's handling of image layers on
overlayfs2: Images are condensed into a single representation on disk
after downloading.

Due to this Docker will constantly redownload all layers that are
applied in a different order in different images (layer order matters
in imperatively created images), based on something it calls the
'ChainID'.

Sorting the layers this way raises the likelihood of a long chain of
matching layers at the beginning of an image.

This relates to #39.
2019-10-03 22:50:02 +01:00
Vincent Ambo
0d820423e9 chore(build-image): Remove nixery-build-layers
This functionality has been rolled into the server component and is no
longer required.
2019-10-03 22:29:50 +01:00
Vincent Ambo
6b06fe27be feat(server): Implement creation of layer tarballs in the server
This will create, upload and hash the layer tarballs in one disk read.
2019-10-03 22:29:50 +01:00
Vincent Ambo
1124b8c236 fix(server): Do not invoke layer build if no layers are missing
This previously invoked a Nix derivation that spent a few seconds on
making an empty object in JSON ...
2019-10-03 13:21:04 +01:00
Vincent Ambo
feba42e409 feat(server): Fetch popularity data on launch
The last missing puzzle piece for #50!
2019-10-03 13:21:04 +01:00
Vincent Ambo
43a642435b feat(server): Reimplement local manifest cache backed by files
Implements a local manifest cache that uses the temporary directory to
cache manifest builds.

This is necessary due to the size of manifests: Keeping them entirely
in-memory would quickly balloon the memory usage of Nixery, unless
some mechanism for cache eviction is implemented.
2019-10-03 13:21:04 +01:00
Vincent Ambo
313e5d08f1 refactor(builder): Streamline layer creation & reintroduce caching
The functions used for layer creation are now easier to follow and
have clear points at which the layer cache is checked and populated.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
53906024ff refactor: Remove remaining MD5-hash mentions and computations 2019-10-03 13:21:04 +01:00
Vincent Ambo
f6b40ed6c7 refactor(server): Cache manifest entries for layer builds
MD5 hash checking is no longer performed by Nixery (it does not seem
to be necessary), hence the layer cache now only keeps the SHA256 hash
and size in the form of the manifest entry.

This makes it possible to restructure the builder code to perform
cache-fetching and cache-populating for layers in the same place.
2019-10-03 13:21:04 +01:00
Vincent Ambo
355fe3f5ec feat(server): Reintroduce manifest caching to GCS
The new builder now caches and reads cached manifests to/from GCS. The
in-memory cache is disabled, as manifests are no longer written to
local file and the caching of file paths does not work (unless we
reintroduce reading/writing from temp files as part of the local
cache).
2019-10-03 13:21:04 +01:00
Vincent Ambo
1308a6e1fd refactor(server): Clean up cache implementation
A couple of minor fixes and improvements to the cache implementation.
2019-10-03 13:21:04 +01:00
Vincent Ambo
0698d7f2aa chore(server): Remove "layer seen" cache
This cache is no longer required as it is implicit because the layer
cache (mapping store path hashes to layer hashes) implies that a layer
has been seen.
2019-10-03 13:21:04 +01:00
Vincent Ambo
64fca61ea1 fix(server): Upload symlink layer created by first Nix build
This layer is needed in addition to those that are built in the second
Nix build.
2019-10-03 13:21:04 +01:00
Vincent Ambo
f4f2909573 fix(server): Specify correct authentication scope for GCS
When retrieving tokens for service service accounts, some methods of
retrieval require a scope to be specified.
2019-10-03 13:21:04 +01:00
Vincent Ambo
aa02ae1421 feat(server): Implement new build process core
Implements the new build process to the point where it can actually
construct and serve image manifests.

It is worth noting that this build process works even if the Nix
sandbox is enabled!

It is also worth nothing that none of the caching functionality that
the new build process enables (such as per-layer build caching) is
actually in use yet, hence running Nixery at this commit is prone to
doing more work than previously.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
17adda0355 fix(server): Minor fixes to updated new builder code 2019-10-03 13:21:04 +01:00
Vincent Ambo
ef2623d168 fix(nix): Minor fixes to derivations for new build process 2019-10-03 13:21:04 +01:00
Vincent Ambo
3f40c0a2d2 feat(server): Implement package for creating image manifests
The new manifest package creates image manifests and their
configuration. This previously happened in Nix, but is now part of the
server's workload.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
87e196757b feat(server): Reimplement creation & uploading of layers
The new build process can now call out to Nix to create layers and
upload them to the bucket if necessary.

The layer cache is populated, but not yet used.
2019-10-03 13:21:04 +01:00
Vincent Ambo
61269175c0 refactor(server): Introduce a state type to carry runtime state
The state type contains things such as the bucket handle and Nixery's
configuration which need to be passed around in the builder.

This is only added for convenience.
2019-10-03 13:21:04 +01:00
Vincent Ambo
6e2b84f475 feat(server): Add cache for layer builds in GCS & local cache
This cache is going to be used for looking up whether a layer build
has taken place already (based on a hash of the layer contents).

See the caching section in the updated documentation for details.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
6262dec8aa feat(nix): Add derivation to create layer tars from a store path set
This introduces a new Nix derivation that, given an attribute set of
layer hashes mapped to store paths, will create a layer tarball for
each of the store paths.

This is going to be used by the builder to create layers that are not
present in the cache.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
2c8ef634f6 docs(caching): Add information about Nixery's caching strategies 2019-10-03 13:21:04 +01:00
Vincent Ambo
e60805c9b2 feat(server): Introduce function to hash contents of a layer
This creates a cache key which can be used to check if a layer has
already been built.
2019-10-03 13:21:04 +01:00
Vincent Ambo
9c3c622403 refactor(server): Expose layer grouping logic via a function
Refactors the layer grouping package (which previously compiled to a
separate binary) to expose the layer grouping logic via a function
instead.

This is the next step towards creating layers inside of the server
component instead of in Nix.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
8c79d085ae chore(server): Import layer grouping logic into server component 2019-10-03 13:21:04 +01:00
Vincent Ambo
0898d8a961 chore(build-image): Simplify wrapper build & remove layer grouping
Simplifies the wrapper script used to invoke Nix builds from Nixery to
just contain the essentials, since the layer grouping logic is moving
into the server itself.
2019-10-03 13:21:04 +01:00
Vincent Ambo
712b38cbbc refactor(build-image): Do not assemble image layers in Nix
This is the first step towards a more granular build process where
some of the build responsibility moves into the server component.

Rather than assembling all layers inside of Nix, it will only create
the symlink forest and return information about the runtime paths
required by the image.

The server is then responsible for grouping these paths into layers,
and assembling the layers themselves.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
ad9b3eb262 refactor(build): Add group-layers to top-level Nix derivations
This makes CI build the group-layers tool (and cache it to Cachix!)
2019-09-22 18:04:40 +01:00
Vincent Ambo
7b987530d1 refactor(build-image): Minor tweak to layer construction script 2019-09-22 18:04:40 +01:00
Vincent Ambo
21a17b33f4 fix(build): Ensure launch script compatibility with other runtimes
Fixes two launch script compatibility issues with other container
runtimes (such as gvisor):

* don't fail if /tmp already exists
* don't fail if the environment becomes unset
2019-09-21 15:25:28 +01:00
Vincent Ambo
0000b956bb feat(server): Log Nix output live during the builds
Instead of dumping all Nix output as one at the end of the build
process, stream it live as the lines come in.

This is a lot more useful for debugging stuff like where manifest
retrievals get stuck.
2019-09-21 15:25:28 +01:00
Vincent Ambo
c391a7b7f8 fix(build-image): Use absolute paths in tarballs 2019-09-21 15:25:28 +01:00
Vincent Ambo
da6fd1d79e fix(build): Ensure nixery-build-image is on Nixery's PATH
This is useful when running Nixery locally.
2019-09-21 12:44:40 +01:00
Vincent Ambo
f0b69638e1 chore(build): Bump nixpkgs version used in Travis
This version matches the updated popularity URL.
2019-09-21 12:44:40 +01:00
Vincent Ambo
64f74abc4d feat: Add configuration option for popularity data URL 2019-09-21 12:44:40 +01:00
Vincent Ambo
e4d03fdb17 chore(docs): Remove mdbook override
The change has been upstreamed in Nixpkgs.
2019-09-21 12:44:40 +01:00
Vincent Ambo
5a002fe067 refactor(builder): Calculate image cache key only once 2019-09-10 11:32:37 +01:00
Vincent Ambo
4a58b0ab4d feat(server): Cache built manifests to the GCS bucket
Caches manifests under `manifests/$cacheKey` in the GCS bucket and
introduces two-tiered retrieval of manifests from the caches (local
first, bucket second).

There is some cleanup to be done in this code, but the initial version
works.
2019-09-10 11:32:37 +01:00
Vincent Ambo
051eb77b3d refactor(server): Use package source specific cache keys
Use the PackageSource.CacheKey function introduced in the previous
commit to determine the key at which a manifest should be cached in
the local cache.

Due to this change, manifests for moving target sources are no longer
cached and the recency threshold logic has been removed.
2019-09-10 11:32:37 +01:00
Vincent Ambo
980f5e2187 refactor(server): Move package source management logic to server
Introduces three new types representing each of the possible package
sources and moves the logic for specifying the package source to the
server.

Concrete changes:

* Determining whether a specified git reference is a commit vs. a
  branch/tag is now done in the server, and is done more precisely by
  using a regular expression.

* Package sources now have a new `CacheKey` function which can be used
  to retrieve a key under which a build manifest can be cached *if*
  the package source is not a moving target (i.e. a full git commit
  hash of either nixpkgs or a private repository).

  This function is not yet used.

* Users *must* now specify a package source, Nixery no longer defaults
  to anything and will fail to launch if no source is configured.
2019-09-10 11:32:37 +01:00
Vincent Ambo
496a4ab847 docs: Add information about NIX_TIMEOUT variable 2019-09-02 23:44:57 +01:00
Vincent Ambo
32b9b5099e feat(server): Add configuration option for Nix build timeouts
Adds a NIX_TIMEOUT environment variable which can be set to a number
of seconds that is the maximum allowed time each Nix builder can run.

By default this is set to 60 seconds, which should be plenty for most
use-cases as Nixery is not expected to be performing builds of
uncached binaries in most production cases.

Currently the errors Nix throws on a build timeout are not separated
from other types of errors, meaning that users will see a generic 500
server error in case of a timeout.

This fixes #47
2019-09-02 23:44:57 +01:00
Vincent Ambo
ce8635833b refactor(build-image): Remove implicit import of entire package set
Explicitly refer to where things come from, and also don't import
dockerTools as it is no longer used for anything.
2019-09-02 00:08:11 +01:00
Vincent Ambo
92270fcbe4 refactor(build-image): Simplify customisation layer builder
Moves the relevant parts of the customisation layer construction from
dockerTools.mkCustomisationLayer into the Nixery code base.

The version in dockerTools builds additional files (including via
hashing of potentially large files) which are not required when
serving an image over the registry protocol.
2019-09-02 00:08:11 +01:00
Vincent Ambo
306e12787a chore(build): Add iana-etc to Nixery's own image
This package is used by a variety of programs that users may want to
embed into Nixery in addition, for example cachix, but those packages
don't refer to it explicitly.
2019-08-21 10:35:32 +01:00
Vincent Ambo
bb5427a47a chore(docs): Update embedded nix-1p version
The new version of the document has syntactic fixes that render pipes
in code blocks in tables correctly across dialects.

Fixes #44
2019-08-21 10:35:32 +01:00