Commit graph

181 commits

Author SHA1 Message Date
Raphael Borun Das Gupta
bdda24a772 chore(nix): update channel 19.03 -> 20.03
Use a NixOS / NixPkgs release that's actually being supported
and regularly updated.
2020-05-01 11:58:31 +01:00
Florian Klink
215df37187 fix(popcount): Fix nix-build -A nixery-popcount
Previously, this was failing as follows:

```
these derivations will be built:
  /nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv
building '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv'...
building
warning: GOPATH set to GOROOT (/nix/store/4859cp1v7zqcqh43jkqsayl4wrz3g6hp-go-1.13.4/share/go) has no effect
failed to initialize build cache at /homeless-shelter/.cache/go-build: mkdir /homeless-shelter: permission denied
builder for '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed with exit code 1
error: build of '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed
```
2020-02-26 08:31:46 +00:00
Vincent Ambo
1031d890ec fix(builder): Fix minor logging switcharoo 2020-01-19 07:44:36 +00:00
Vincent Ambo
2b82f1b71a refactor: Reshuffle file structure for better code layout
This gets rid of the package called "server" and instead moves
everything into the project root, such that Go actually builds us a
binary called `nixery`.

This is the first step towards factoring out CLI-based functionality
for Nixery.
2019-11-27 14:12:38 +00:00
Vincent Ambo
df88da126a fix(builder): Ensure "solo-metapackages" do not break builds
The previous logic failed because single meta-packages such as
"nixery.dev/shell" would not end up removing the meta-package itself
from the list of packages passed to Nix, causing a build failure.

This was a regression introduced in 827468a.
2019-11-27 13:28:13 +00:00
Vincent Ambo
a924093d09 test(builder): Add test coverage for name->image conversion
Adds tests to cover that packages & metapackages are parsed into image
names correctly.
2019-11-27 13:28:13 +00:00
Vincent Ambo
104c930040 chore(build): Use significantly fewer layers for Nixery itself
Nixery itself is built with the buildLayeredImage system, which takes
some time to create large numbers of layers.

This adjusts the default number of image layers from 96 to 20.

Additionally Nixery's image is often loaded with `docker load -i`,
which ignores layer cache hits anyways.

Additionaly the CI build is configured to use only 1, which speeds up
CI runs.
2019-11-09 15:35:58 +00:00
Vincent Ambo
9a8abeff97 feat(build): Integration test on both CPU architectures 2019-11-09 15:35:58 +00:00
Vincent Ambo
1d6898a7cc feat(build): Include arm64 in build matrix 2019-11-09 15:35:58 +00:00
Vincent Ambo
145b7f4289 fix(build-image): Allow "cross-builds" of images for different arch
Imports the package set twice in the builder expression: Once
configured for the target system, once configured for the native
system.

This makes it possible to fetch the actual image contents for the
required architecture, but use local tools to assemble the symlink
layer and metadata.
2019-11-09 15:35:58 +00:00
Vincent Ambo
d7ccf35149 feat(builder): Support 'arm64' meta-package
Specifying this meta-package toggles support for ARM64 images, for
example:

    # Pull a default x86_64 image
    docker pull nixery.dev/hello

    # Pull an ARM64 image
    docker pull nixery.dev/arm64/hello
2019-11-09 15:35:58 +00:00
Vincent Ambo
3c2de4c037 refactor(builder): Parameterise CPU architecture to use for images
Adds the CPU architecture to the image configuration. This will make
it possible to let users toggle architecture via meta-packages.

Relates to #13
2019-11-09 15:35:58 +00:00
Vincent Ambo
7afbc912ce chore(build): Add nixery-popcount to top-level package set 2019-11-03 01:33:36 +00:00
Vincent Ambo
05b5b1718a feat(popcount): Cache seen narinfos on disk 2019-11-03 01:33:36 +00:00
Vincent Ambo
6a2fb092a7 chore: Add missing copyright headers to source files 2019-11-03 01:33:36 +00:00
Vincent Ambo
b03f7a1b4d feat(popcount): Add new narinfo-based popcount implementation
Adds an implementation of popcount that, instead of realising
derivations locally, just queries the cache's narinfo files.

The downside of this is that calculating popularity for arbitrary Nix
package sets is not possible with this implementation. The upside is
that calculating the popularity for an entire Nix channel can now be
done in ~10 seconds[0].

This fixes #65.

[0]: Assuming a /fast/ internet connection.
2019-11-03 01:33:36 +00:00
Vincent Ambo
2d4a3ea307 chore(server): Remove outdated TODO
Real-life experience has shown that the weighting of the metric
produced here is appropriate.
2019-10-29 19:16:24 +01:00
Vincent Ambo
904c3dade0 refactor(server): Handle non-error errors safely
This case should not be possible unless something manually constructs
a logrus entry with a non-error value in the log.ErrorKey field, but
it's better to be safe than sorry.
2019-10-29 19:16:24 +01:00
Vincent Ambo
3a7c964a22 chore(build): Configure build to use new GCS configuration options 2019-10-28 22:31:44 +01:00
Vincent Ambo
ab190256ab fix(server): Use correct scope for GCS tokens 2019-10-28 22:31:44 +01:00
Vincent Ambo
3611baf040 docs(under-the-hood): Update builder & storage backend information
Both of these no longer matched the reality of what was actually going
on in Nixery.
2019-10-28 22:31:44 +01:00
Vincent Ambo
b736f5580d docs: Add storage configuration options to README 2019-10-28 22:31:44 +01:00
Vincent Ambo
d8fba23365 fix(server): Thread request context to all relevant places
Previously background contexts where created where necessary (e.g. in
GCS interactions). Should I begin to use request timeouts or other
context-dependent things in the future, it's useful to have the actual
HTTP request context around.

This threads the request context through the application to all places
that need it.
2019-10-28 22:31:44 +01:00
Vincent Ambo
30e618b65b chore(server): Move cache miss log statement to debug level
This is very annoying otherwise.
2019-10-28 22:31:44 +01:00
Vincent Ambo
4332d38f4f fix(server): Correctly construct filesystem paths for layer serving 2019-10-28 22:31:44 +01:00
Vincent Ambo
b60a8d007b fix(server): Ensure paths exist when renaming in filesystem storage
The point at which files are moved happens to also (initially) be the
point where the `layers` directory is created. For this reason
renaming must ensure that all path components exist, which this commit
takes care of.
2019-10-28 22:31:44 +01:00
Vincent Ambo
c08aa52558 fix(server): Ensure error messages are correctly printed in logs
I assumed (incorrectly) that logrus would already take care of
surfacing error messages in human-readable form.
2019-10-28 22:31:44 +01:00
Vincent Ambo
790bce219c feat(server): Add filesystem storage backend config options
The filesystem storage backend can be enabled by setting
`NIXERY_STORAGE_BACKEND` to `filesystem` and `STORAGE_PATH` to a disk
location from which Nixery can serve files.
2019-10-28 22:31:44 +01:00
Vincent Ambo
167a0b3263 refactor(server): Pass HTTP request to storage.ServeLayer
The request object is required for some serving methods (e.g. the
filesystem one).
2019-10-28 22:31:44 +01:00
Vincent Ambo
e5bb2fc887 feat(server): Implement initial filesystem storage backend
This allows users to store and serve layers from a local filesystem
path.
2019-10-28 22:31:44 +01:00
Vincent Ambo
e8fd6b6734 refactor(server): Change setup to create new storage backends 2019-10-28 22:31:44 +01:00
Vincent Ambo
20e0ca53cb feat(server): Implement GCS storage backend with new interface
Logical implementation is mostly identical to the previous one, but
adhering to the new storage.Backend interface.
2019-10-28 22:31:44 +01:00
Vincent Ambo
f7d16c5d45 refactor(server): Introduce pluggable interface for storage backends
This abstracts over the functionality of Google Cloud Storage and
other potential underlying storage backends to make it possible to
replace these in Nixery.

The GCS backend is not yet reimplemented.
2019-10-28 22:31:44 +01:00
Vincent Ambo
ffe58d6cb5 refactor(build): Do not expose nixery-server attribute
In most cases this is not useful for users without the wrapper script,
so users should always build nixery-bin anyways.
2019-10-28 22:31:44 +01:00
Vincent Ambo
7b7d21205f docs: Update GCS signing key documentation
This key is now taken straight from the configured service account
key.
2019-10-27 13:58:04 +01:00
Vincent Ambo
3a5db4f9f1 refactor(server): Load GCS signing key from service account key
The JSON file generated for service account keys already contains the
required information for signing URLs in GCS, thus the environment
variables for toggling signing behaviour have been removed.

Signing is now enabled automatically in the presence of service
account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
2019-10-27 13:58:04 +01:00
Vincent Ambo
cca835ae37 fix(build): Only take the first matching hash for source hashing
Some Nix download mechanisms will add a second hash in the store path,
which had been added to the source hash output (breaking argument
interpolation).
2019-10-11 14:23:41 +01:00
Vincent Ambo
1853c74998 refactor(server): Only compress symlink forest layer once
Instead of compressing & decompressing again to get the underlying tar
hash, use a similar mechanism as for store path layers for the symlink
layer and only compress it once while uploading.
2019-10-11 12:37:00 +01:00
Vincent Ambo
e22ff5d176 fix(server): Use uncompressed tarball hashes in image config
Docker expects hashes of compressed tarballs in the manifest (as these
are used to fetch from the content-addressable layer store), but for
some reason it expects hashes in the configuration layer to be of
uncompressed tarballs.

To achieve this an additional SHA256 hash is calculcated while
creating the layer tarballs, but before passing them to the gzip
writer.

In the current constellation the symlink layer is first compressed and
then decompressed again to calculate its hash. This can be refactored
in a future change.
2019-10-11 12:37:00 +01:00
Vincent Ambo
0693e371d6 feat(server): Apply GZIP compression to all image layers
This fixes #62
2019-10-11 12:37:00 +01:00
Vincent Ambo
bf2718cebb chore(build): Use separate GCS bucket for CI runs
This has become an issue recently with changes such as GZIP
compression, where CI runs no longer work because they conflict with
the production bucket for the public instance.
2019-10-11 12:37:00 +01:00
Vincent Ambo
d7ffbbdea4 refactor(server): Use logrus convenience functions for logs
Makes use of the `.WithError` and `.WithField` convenience functions
in logrus to simplify log statement construction.

This has the added benefit of making it easier to correctly log
errors.
2019-10-06 23:05:23 +01:00
Vincent Ambo
c1020754a2 fix(build-image): Import 'match' from builtins 2019-10-06 23:05:23 +01:00
Vincent Ambo
6f148f789f refactor(server): Convert existing log entries to structured format
This rewrites all existing log statements into the structured logrus
format. For consistency, all errors are always logged separately from
the primary message in a field called `error`.

Only the "info", "error" and "warn" severities are used.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f77c93b6ae feat(server): Add log level to severity mapping
The output format now writes a `severity` field that follows that
format that should be recognised by Stackdriver Logging.
2019-10-06 23:05:23 +01:00
Vincent Ambo
6912658c72 feat(server): Use hash of Nixery source as version
Uses a hash of Nixery's sources as the version displayed when Nixery
launches or logs an error. This makes it possible to distinguish
between errors logged from different versions.

The source hashes should be reproducible between different checkouts
of the same source tree.
2019-10-06 23:05:23 +01:00
Vincent Ambo
95abb1bcde feat(server): Initial Stackdriver-compatible log formatter
This formatter has basic support for the Stackdriver Error Reporting
format, but several things are still lacking:

* the service version (preferably git commit?) needs to be included in
  the server somehow
* log streams should be split between stdout/stderr as that is how
  AppEngine (and several other GCP services?) seemingly differentiate
  between info/error logs
2019-10-06 23:05:23 +01:00
Vincent Ambo
0642f7044d fix(server): Amend package path for Go tooling compatibility
With these changes it is possible to keep Nixery in $GOPATH and build
the server in there, while still having things work correctly via Nix.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f4bf3518f6 refactor(server): Replace log calls with logrus
This introduces a structured logging library that can be used (next
step) to attach additional metadata to log entries.
2019-10-06 23:05:23 +01:00
Vincent Ambo
d9b329ef59 refactor(server): Always include 'cacert' & 'iana-etc'
These two packages almost always end up being required by programs,
but people don't necessarily consider them.

They will now always be added and their popularity is artificially
inflated to ensure they end up at the top of the layer list.
2019-10-03 22:50:02 +01:00