This abstracts over the functionality of Google Cloud Storage and
other potential underlying storage backends to make it possible to
replace these in Nixery.
The GCS backend is not yet reimplemented.
Makes use of the `.WithError` and `.WithField` convenience functions
in logrus to simplify log statement construction.
This has the added benefit of making it easier to correctly log
errors.
This rewrites all existing log statements into the structured logrus
format. For consistency, all errors are always logged separately from
the primary message in a field called `error`.
Only the "info", "error" and "warn" severities are used.
Implements a local manifest cache that uses the temporary directory to
cache manifest builds.
This is necessary due to the size of manifests: Keeping them entirely
in-memory would quickly balloon the memory usage of Nixery, unless
some mechanism for cache eviction is implemented.
MD5 hash checking is no longer performed by Nixery (it does not seem
to be necessary), hence the layer cache now only keeps the SHA256 hash
and size in the form of the manifest entry.
This makes it possible to restructure the builder code to perform
cache-fetching and cache-populating for layers in the same place.
The new builder now caches and reads cached manifests to/from GCS. The
in-memory cache is disabled, as manifests are no longer written to
local file and the caching of file paths does not work (unless we
reintroduce reading/writing from temp files as part of the local
cache).
This cache is no longer required as it is implicit because the layer
cache (mapping store path hashes to layer hashes) implies that a layer
has been seen.
This cache is going to be used for looking up whether a layer build
has taken place already (based on a hash of the layer contents).
See the caching section in the updated documentation for details.
Relates to #50.
Caches manifests under `manifests/$cacheKey` in the GCS bucket and
introduces two-tiered retrieval of manifests from the caches (local
first, bucket second).
There is some cleanup to be done in this code, but the initial version
works.
Use the PackageSource.CacheKey function introduced in the previous
commit to determine the key at which a manifest should be cached in
the local cache.
Due to this change, manifests for moving target sources are no longer
cached and the recency threshold logic has been removed.
Implements a cache that keeps track of:
a) Manifests that have already been built (for up to 6 hours)
b) Layers that have already been seen (and uploaded to GCS)
This significantly speeds up response times for images that are full
or partial matches with previous images served by an instance.
