Commit graph

165 commits

Author SHA1 Message Date
Vincent Ambo
904c3dade0 refactor(server): Handle non-error errors safely
This case should not be possible unless something manually constructs
a logrus entry with a non-error value in the log.ErrorKey field, but
it's better to be safe than sorry.
2019-10-29 19:16:24 +01:00
Vincent Ambo
3a7c964a22 chore(build): Configure build to use new GCS configuration options 2019-10-28 22:31:44 +01:00
Vincent Ambo
ab190256ab fix(server): Use correct scope for GCS tokens 2019-10-28 22:31:44 +01:00
Vincent Ambo
3611baf040 docs(under-the-hood): Update builder & storage backend information
Both of these no longer matched the reality of what was actually going
on in Nixery.
2019-10-28 22:31:44 +01:00
Vincent Ambo
b736f5580d docs: Add storage configuration options to README 2019-10-28 22:31:44 +01:00
Vincent Ambo
d8fba23365 fix(server): Thread request context to all relevant places
Previously background contexts where created where necessary (e.g. in
GCS interactions). Should I begin to use request timeouts or other
context-dependent things in the future, it's useful to have the actual
HTTP request context around.

This threads the request context through the application to all places
that need it.
2019-10-28 22:31:44 +01:00
Vincent Ambo
30e618b65b chore(server): Move cache miss log statement to debug level
This is very annoying otherwise.
2019-10-28 22:31:44 +01:00
Vincent Ambo
4332d38f4f fix(server): Correctly construct filesystem paths for layer serving 2019-10-28 22:31:44 +01:00
Vincent Ambo
b60a8d007b fix(server): Ensure paths exist when renaming in filesystem storage
The point at which files are moved happens to also (initially) be the
point where the `layers` directory is created. For this reason
renaming must ensure that all path components exist, which this commit
takes care of.
2019-10-28 22:31:44 +01:00
Vincent Ambo
c08aa52558 fix(server): Ensure error messages are correctly printed in logs
I assumed (incorrectly) that logrus would already take care of
surfacing error messages in human-readable form.
2019-10-28 22:31:44 +01:00
Vincent Ambo
790bce219c feat(server): Add filesystem storage backend config options
The filesystem storage backend can be enabled by setting
`NIXERY_STORAGE_BACKEND` to `filesystem` and `STORAGE_PATH` to a disk
location from which Nixery can serve files.
2019-10-28 22:31:44 +01:00
Vincent Ambo
167a0b3263 refactor(server): Pass HTTP request to storage.ServeLayer
The request object is required for some serving methods (e.g. the
filesystem one).
2019-10-28 22:31:44 +01:00
Vincent Ambo
e5bb2fc887 feat(server): Implement initial filesystem storage backend
This allows users to store and serve layers from a local filesystem
path.
2019-10-28 22:31:44 +01:00
Vincent Ambo
e8fd6b6734 refactor(server): Change setup to create new storage backends 2019-10-28 22:31:44 +01:00
Vincent Ambo
20e0ca53cb feat(server): Implement GCS storage backend with new interface
Logical implementation is mostly identical to the previous one, but
adhering to the new storage.Backend interface.
2019-10-28 22:31:44 +01:00
Vincent Ambo
f7d16c5d45 refactor(server): Introduce pluggable interface for storage backends
This abstracts over the functionality of Google Cloud Storage and
other potential underlying storage backends to make it possible to
replace these in Nixery.

The GCS backend is not yet reimplemented.
2019-10-28 22:31:44 +01:00
Vincent Ambo
ffe58d6cb5 refactor(build): Do not expose nixery-server attribute
In most cases this is not useful for users without the wrapper script,
so users should always build nixery-bin anyways.
2019-10-28 22:31:44 +01:00
Vincent Ambo
7b7d21205f docs: Update GCS signing key documentation
This key is now taken straight from the configured service account
key.
2019-10-27 13:58:04 +01:00
Vincent Ambo
3a5db4f9f1 refactor(server): Load GCS signing key from service account key
The JSON file generated for service account keys already contains the
required information for signing URLs in GCS, thus the environment
variables for toggling signing behaviour have been removed.

Signing is now enabled automatically in the presence of service
account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
2019-10-27 13:58:04 +01:00
Vincent Ambo
cca835ae37 fix(build): Only take the first matching hash for source hashing
Some Nix download mechanisms will add a second hash in the store path,
which had been added to the source hash output (breaking argument
interpolation).
2019-10-11 14:23:41 +01:00
Vincent Ambo
1853c74998 refactor(server): Only compress symlink forest layer once
Instead of compressing & decompressing again to get the underlying tar
hash, use a similar mechanism as for store path layers for the symlink
layer and only compress it once while uploading.
2019-10-11 12:37:00 +01:00
Vincent Ambo
e22ff5d176 fix(server): Use uncompressed tarball hashes in image config
Docker expects hashes of compressed tarballs in the manifest (as these
are used to fetch from the content-addressable layer store), but for
some reason it expects hashes in the configuration layer to be of
uncompressed tarballs.

To achieve this an additional SHA256 hash is calculcated while
creating the layer tarballs, but before passing them to the gzip
writer.

In the current constellation the symlink layer is first compressed and
then decompressed again to calculate its hash. This can be refactored
in a future change.
2019-10-11 12:37:00 +01:00
Vincent Ambo
0693e371d6 feat(server): Apply GZIP compression to all image layers
This fixes #62
2019-10-11 12:37:00 +01:00
Vincent Ambo
bf2718cebb chore(build): Use separate GCS bucket for CI runs
This has become an issue recently with changes such as GZIP
compression, where CI runs no longer work because they conflict with
the production bucket for the public instance.
2019-10-11 12:37:00 +01:00
Vincent Ambo
d7ffbbdea4 refactor(server): Use logrus convenience functions for logs
Makes use of the `.WithError` and `.WithField` convenience functions
in logrus to simplify log statement construction.

This has the added benefit of making it easier to correctly log
errors.
2019-10-06 23:05:23 +01:00
Vincent Ambo
c1020754a2 fix(build-image): Import 'match' from builtins 2019-10-06 23:05:23 +01:00
Vincent Ambo
6f148f789f refactor(server): Convert existing log entries to structured format
This rewrites all existing log statements into the structured logrus
format. For consistency, all errors are always logged separately from
the primary message in a field called `error`.

Only the "info", "error" and "warn" severities are used.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f77c93b6ae feat(server): Add log level to severity mapping
The output format now writes a `severity` field that follows that
format that should be recognised by Stackdriver Logging.
2019-10-06 23:05:23 +01:00
Vincent Ambo
6912658c72 feat(server): Use hash of Nixery source as version
Uses a hash of Nixery's sources as the version displayed when Nixery
launches or logs an error. This makes it possible to distinguish
between errors logged from different versions.

The source hashes should be reproducible between different checkouts
of the same source tree.
2019-10-06 23:05:23 +01:00
Vincent Ambo
95abb1bcde feat(server): Initial Stackdriver-compatible log formatter
This formatter has basic support for the Stackdriver Error Reporting
format, but several things are still lacking:

* the service version (preferably git commit?) needs to be included in
  the server somehow
* log streams should be split between stdout/stderr as that is how
  AppEngine (and several other GCP services?) seemingly differentiate
  between info/error logs
2019-10-06 23:05:23 +01:00
Vincent Ambo
0642f7044d fix(server): Amend package path for Go tooling compatibility
With these changes it is possible to keep Nixery in $GOPATH and build
the server in there, while still having things work correctly via Nix.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f4bf3518f6 refactor(server): Replace log calls with logrus
This introduces a structured logging library that can be used (next
step) to attach additional metadata to log entries.
2019-10-06 23:05:23 +01:00
Vincent Ambo
d9b329ef59 refactor(server): Always include 'cacert' & 'iana-etc'
These two packages almost always end up being required by programs,
but people don't necessarily consider them.

They will now always be added and their popularity is artificially
inflated to ensure they end up at the top of the layer list.
2019-10-03 22:50:02 +01:00
Vincent Ambo
9bb6d0ae25 fix(server): Ensure build cache objects are written to GCS
Cache writes might not be flushed without this call.
2019-10-03 22:50:02 +01:00
Vincent Ambo
48a5ecda97 feat(server): Order layers in image manifest based on merge rating
Image layers in manifests are now sorted in a stable (descending)
order based on their merge rating, meaning that layers more likely to
be shared between images come first.

The reason for this change is Docker's handling of image layers on
overlayfs2: Images are condensed into a single representation on disk
after downloading.

Due to this Docker will constantly redownload all layers that are
applied in a different order in different images (layer order matters
in imperatively created images), based on something it calls the
'ChainID'.

Sorting the layers this way raises the likelihood of a long chain of
matching layers at the beginning of an image.

This relates to #39.
2019-10-03 22:50:02 +01:00
Vincent Ambo
0d820423e9 chore(build-image): Remove nixery-build-layers
This functionality has been rolled into the server component and is no
longer required.
2019-10-03 22:29:50 +01:00
Vincent Ambo
6b06fe27be feat(server): Implement creation of layer tarballs in the server
This will create, upload and hash the layer tarballs in one disk read.
2019-10-03 22:29:50 +01:00
Vincent Ambo
1124b8c236 fix(server): Do not invoke layer build if no layers are missing
This previously invoked a Nix derivation that spent a few seconds on
making an empty object in JSON ...
2019-10-03 13:21:04 +01:00
Vincent Ambo
feba42e409 feat(server): Fetch popularity data on launch
The last missing puzzle piece for #50!
2019-10-03 13:21:04 +01:00
Vincent Ambo
43a642435b feat(server): Reimplement local manifest cache backed by files
Implements a local manifest cache that uses the temporary directory to
cache manifest builds.

This is necessary due to the size of manifests: Keeping them entirely
in-memory would quickly balloon the memory usage of Nixery, unless
some mechanism for cache eviction is implemented.
2019-10-03 13:21:04 +01:00
Vincent Ambo
313e5d08f1 refactor(builder): Streamline layer creation & reintroduce caching
The functions used for layer creation are now easier to follow and
have clear points at which the layer cache is checked and populated.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
53906024ff refactor: Remove remaining MD5-hash mentions and computations 2019-10-03 13:21:04 +01:00
Vincent Ambo
f6b40ed6c7 refactor(server): Cache manifest entries for layer builds
MD5 hash checking is no longer performed by Nixery (it does not seem
to be necessary), hence the layer cache now only keeps the SHA256 hash
and size in the form of the manifest entry.

This makes it possible to restructure the builder code to perform
cache-fetching and cache-populating for layers in the same place.
2019-10-03 13:21:04 +01:00
Vincent Ambo
355fe3f5ec feat(server): Reintroduce manifest caching to GCS
The new builder now caches and reads cached manifests to/from GCS. The
in-memory cache is disabled, as manifests are no longer written to
local file and the caching of file paths does not work (unless we
reintroduce reading/writing from temp files as part of the local
cache).
2019-10-03 13:21:04 +01:00
Vincent Ambo
1308a6e1fd refactor(server): Clean up cache implementation
A couple of minor fixes and improvements to the cache implementation.
2019-10-03 13:21:04 +01:00
Vincent Ambo
0698d7f2aa chore(server): Remove "layer seen" cache
This cache is no longer required as it is implicit because the layer
cache (mapping store path hashes to layer hashes) implies that a layer
has been seen.
2019-10-03 13:21:04 +01:00
Vincent Ambo
64fca61ea1 fix(server): Upload symlink layer created by first Nix build
This layer is needed in addition to those that are built in the second
Nix build.
2019-10-03 13:21:04 +01:00
Vincent Ambo
f4f2909573 fix(server): Specify correct authentication scope for GCS
When retrieving tokens for service service accounts, some methods of
retrieval require a scope to be specified.
2019-10-03 13:21:04 +01:00
Vincent Ambo
aa02ae1421 feat(server): Implement new build process core
Implements the new build process to the point where it can actually
construct and serve image manifests.

It is worth noting that this build process works even if the Nix
sandbox is enabled!

It is also worth nothing that none of the caching functionality that
the new build process enables (such as per-layer build caching) is
actually in use yet, hence running Nixery at this commit is prone to
doing more work than previously.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
17adda0355 fix(server): Minor fixes to updated new builder code 2019-10-03 13:21:04 +01:00