To ensure that registry clients which attempt to pull manifests by
their content hash can interact with Nixery, this change implements
persisting image manifests in the CAS in the same way as image layers.
In combination with the previous refactorings this means that Nixery's
serving flow is now compatible with containerd.
I have verified this locally, but CI currently only runs against
Docker and not containerd, which is something I plan to address in a
subsequent PR.
This fixes#102
Modifies the layer serving endpoint to be a generic blob-serving
endpoint that can handle both manifest and layer object "types".
Note that this commit does not yet populate the CAS with any
manifests.
Permission changes in the Travis CI Nix builders have caused this to
start failing, as the build user now has insufficient permissions to
use caches.
There may be a way to change the permissions instead, but in the
meantime we will just cause things to rebuild.
Channel serving has moved to a new subdomain, and the redirect
semantics have changed. Instead of serving temporary redirects,
permanent redirects are now issued.
I've reported this upstream as a bug, but this workaround will fix it
in the meantime.
This moves the pin from just being in the Travis configuration to also
being set in a nixpkgs-pin.nix file, which makes it trivial to build
at the right commit when performing local builds.
Previously, this was failing as follows:
```
these derivations will be built:
/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv
building '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv'...
building
warning: GOPATH set to GOROOT (/nix/store/4859cp1v7zqcqh43jkqsayl4wrz3g6hp-go-1.13.4/share/go) has no effect
failed to initialize build cache at /homeless-shelter/.cache/go-build: mkdir /homeless-shelter: permission denied
builder for '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed with exit code 1
error: build of '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed
```
This gets rid of the package called "server" and instead moves
everything into the project root, such that Go actually builds us a
binary called `nixery`.
This is the first step towards factoring out CLI-based functionality
for Nixery.
The previous logic failed because single meta-packages such as
"nixery.dev/shell" would not end up removing the meta-package itself
from the list of packages passed to Nix, causing a build failure.
This was a regression introduced in 827468a.
Nixery itself is built with the buildLayeredImage system, which takes
some time to create large numbers of layers.
This adjusts the default number of image layers from 96 to 20.
Additionally Nixery's image is often loaded with `docker load -i`,
which ignores layer cache hits anyways.
Additionaly the CI build is configured to use only 1, which speeds up
CI runs.
Imports the package set twice in the builder expression: Once
configured for the target system, once configured for the native
system.
This makes it possible to fetch the actual image contents for the
required architecture, but use local tools to assemble the symlink
layer and metadata.
Specifying this meta-package toggles support for ARM64 images, for
example:
# Pull a default x86_64 image
docker pull nixery.dev/hello
# Pull an ARM64 image
docker pull nixery.dev/arm64/hello
Adds an implementation of popcount that, instead of realising
derivations locally, just queries the cache's narinfo files.
The downside of this is that calculating popularity for arbitrary Nix
package sets is not possible with this implementation. The upside is
that calculating the popularity for an entire Nix channel can now be
done in ~10 seconds[0].
This fixes#65.
[0]: Assuming a /fast/ internet connection.
This case should not be possible unless something manually constructs
a logrus entry with a non-error value in the log.ErrorKey field, but
it's better to be safe than sorry.
Previously background contexts where created where necessary (e.g. in
GCS interactions). Should I begin to use request timeouts or other
context-dependent things in the future, it's useful to have the actual
HTTP request context around.
This threads the request context through the application to all places
that need it.
The point at which files are moved happens to also (initially) be the
point where the `layers` directory is created. For this reason
renaming must ensure that all path components exist, which this commit
takes care of.
The filesystem storage backend can be enabled by setting
`NIXERY_STORAGE_BACKEND` to `filesystem` and `STORAGE_PATH` to a disk
location from which Nixery can serve files.
This abstracts over the functionality of Google Cloud Storage and
other potential underlying storage backends to make it possible to
replace these in Nixery.
The GCS backend is not yet reimplemented.
The JSON file generated for service account keys already contains the
required information for signing URLs in GCS, thus the environment
variables for toggling signing behaviour have been removed.
Signing is now enabled automatically in the presence of service
account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
Some Nix download mechanisms will add a second hash in the store path,
which had been added to the source hash output (breaking argument
interpolation).
Instead of compressing & decompressing again to get the underlying tar
hash, use a similar mechanism as for store path layers for the symlink
layer and only compress it once while uploading.
Docker expects hashes of compressed tarballs in the manifest (as these
are used to fetch from the content-addressable layer store), but for
some reason it expects hashes in the configuration layer to be of
uncompressed tarballs.
To achieve this an additional SHA256 hash is calculcated while
creating the layer tarballs, but before passing them to the gzip
writer.
In the current constellation the symlink layer is first compressed and
then decompressed again to calculate its hash. This can be refactored
in a future change.
