Commit graph

605 commits

Author SHA1 Message Date
Vincent Ambo
6f148f789f refactor(server): Convert existing log entries to structured format
This rewrites all existing log statements into the structured logrus
format. For consistency, all errors are always logged separately from
the primary message in a field called `error`.

Only the "info", "error" and "warn" severities are used.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f77c93b6ae feat(server): Add log level to severity mapping
The output format now writes a `severity` field that follows that
format that should be recognised by Stackdriver Logging.
2019-10-06 23:05:23 +01:00
Vincent Ambo
6912658c72 feat(server): Use hash of Nixery source as version
Uses a hash of Nixery's sources as the version displayed when Nixery
launches or logs an error. This makes it possible to distinguish
between errors logged from different versions.

The source hashes should be reproducible between different checkouts
of the same source tree.
2019-10-06 23:05:23 +01:00
Vincent Ambo
95abb1bcde feat(server): Initial Stackdriver-compatible log formatter
This formatter has basic support for the Stackdriver Error Reporting
format, but several things are still lacking:

* the service version (preferably git commit?) needs to be included in
  the server somehow
* log streams should be split between stdout/stderr as that is how
  AppEngine (and several other GCP services?) seemingly differentiate
  between info/error logs
2019-10-06 23:05:23 +01:00
Vincent Ambo
0642f7044d fix(server): Amend package path for Go tooling compatibility
With these changes it is possible to keep Nixery in $GOPATH and build
the server in there, while still having things work correctly via Nix.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f4bf3518f6 refactor(server): Replace log calls with logrus
This introduces a structured logging library that can be used (next
step) to attach additional metadata to log entries.
2019-10-06 23:05:23 +01:00
Vincent Ambo
d9b329ef59 refactor(server): Always include 'cacert' & 'iana-etc'
These two packages almost always end up being required by programs,
but people don't necessarily consider them.

They will now always be added and their popularity is artificially
inflated to ensure they end up at the top of the layer list.
2019-10-03 22:50:02 +01:00
Vincent Ambo
9bb6d0ae25 fix(server): Ensure build cache objects are written to GCS
Cache writes might not be flushed without this call.
2019-10-03 22:50:02 +01:00
Vincent Ambo
48a5ecda97 feat(server): Order layers in image manifest based on merge rating
Image layers in manifests are now sorted in a stable (descending)
order based on their merge rating, meaning that layers more likely to
be shared between images come first.

The reason for this change is Docker's handling of image layers on
overlayfs2: Images are condensed into a single representation on disk
after downloading.

Due to this Docker will constantly redownload all layers that are
applied in a different order in different images (layer order matters
in imperatively created images), based on something it calls the
'ChainID'.

Sorting the layers this way raises the likelihood of a long chain of
matching layers at the beginning of an image.

This relates to #39.
2019-10-03 22:50:02 +01:00
Vincent Ambo
0d820423e9 chore(build-image): Remove nixery-build-layers
This functionality has been rolled into the server component and is no
longer required.
2019-10-03 22:29:50 +01:00
Vincent Ambo
6b06fe27be feat(server): Implement creation of layer tarballs in the server
This will create, upload and hash the layer tarballs in one disk read.
2019-10-03 22:29:50 +01:00
Vincent Ambo
1124b8c236 fix(server): Do not invoke layer build if no layers are missing
This previously invoked a Nix derivation that spent a few seconds on
making an empty object in JSON ...
2019-10-03 13:21:04 +01:00
Vincent Ambo
feba42e409 feat(server): Fetch popularity data on launch
The last missing puzzle piece for #50!
2019-10-03 13:21:04 +01:00
Vincent Ambo
43a642435b feat(server): Reimplement local manifest cache backed by files
Implements a local manifest cache that uses the temporary directory to
cache manifest builds.

This is necessary due to the size of manifests: Keeping them entirely
in-memory would quickly balloon the memory usage of Nixery, unless
some mechanism for cache eviction is implemented.
2019-10-03 13:21:04 +01:00
Vincent Ambo
313e5d08f1 refactor(builder): Streamline layer creation & reintroduce caching
The functions used for layer creation are now easier to follow and
have clear points at which the layer cache is checked and populated.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
53906024ff refactor: Remove remaining MD5-hash mentions and computations 2019-10-03 13:21:04 +01:00
Vincent Ambo
f6b40ed6c7 refactor(server): Cache manifest entries for layer builds
MD5 hash checking is no longer performed by Nixery (it does not seem
to be necessary), hence the layer cache now only keeps the SHA256 hash
and size in the form of the manifest entry.

This makes it possible to restructure the builder code to perform
cache-fetching and cache-populating for layers in the same place.
2019-10-03 13:21:04 +01:00
Vincent Ambo
355fe3f5ec feat(server): Reintroduce manifest caching to GCS
The new builder now caches and reads cached manifests to/from GCS. The
in-memory cache is disabled, as manifests are no longer written to
local file and the caching of file paths does not work (unless we
reintroduce reading/writing from temp files as part of the local
cache).
2019-10-03 13:21:04 +01:00
Vincent Ambo
1308a6e1fd refactor(server): Clean up cache implementation
A couple of minor fixes and improvements to the cache implementation.
2019-10-03 13:21:04 +01:00
Vincent Ambo
0698d7f2aa chore(server): Remove "layer seen" cache
This cache is no longer required as it is implicit because the layer
cache (mapping store path hashes to layer hashes) implies that a layer
has been seen.
2019-10-03 13:21:04 +01:00
Vincent Ambo
64fca61ea1 fix(server): Upload symlink layer created by first Nix build
This layer is needed in addition to those that are built in the second
Nix build.
2019-10-03 13:21:04 +01:00
Vincent Ambo
f4f2909573 fix(server): Specify correct authentication scope for GCS
When retrieving tokens for service service accounts, some methods of
retrieval require a scope to be specified.
2019-10-03 13:21:04 +01:00
Vincent Ambo
aa02ae1421 feat(server): Implement new build process core
Implements the new build process to the point where it can actually
construct and serve image manifests.

It is worth noting that this build process works even if the Nix
sandbox is enabled!

It is also worth nothing that none of the caching functionality that
the new build process enables (such as per-layer build caching) is
actually in use yet, hence running Nixery at this commit is prone to
doing more work than previously.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
17adda0355 fix(server): Minor fixes to updated new builder code 2019-10-03 13:21:04 +01:00
Vincent Ambo
ef2623d168 fix(nix): Minor fixes to derivations for new build process 2019-10-03 13:21:04 +01:00
Vincent Ambo
3f40c0a2d2 feat(server): Implement package for creating image manifests
The new manifest package creates image manifests and their
configuration. This previously happened in Nix, but is now part of the
server's workload.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
87e196757b feat(server): Reimplement creation & uploading of layers
The new build process can now call out to Nix to create layers and
upload them to the bucket if necessary.

The layer cache is populated, but not yet used.
2019-10-03 13:21:04 +01:00
Vincent Ambo
61269175c0 refactor(server): Introduce a state type to carry runtime state
The state type contains things such as the bucket handle and Nixery's
configuration which need to be passed around in the builder.

This is only added for convenience.
2019-10-03 13:21:04 +01:00
Vincent Ambo
6e2b84f475 feat(server): Add cache for layer builds in GCS & local cache
This cache is going to be used for looking up whether a layer build
has taken place already (based on a hash of the layer contents).

See the caching section in the updated documentation for details.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
6262dec8aa feat(nix): Add derivation to create layer tars from a store path set
This introduces a new Nix derivation that, given an attribute set of
layer hashes mapped to store paths, will create a layer tarball for
each of the store paths.

This is going to be used by the builder to create layers that are not
present in the cache.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
2c8ef634f6 docs(caching): Add information about Nixery's caching strategies 2019-10-03 13:21:04 +01:00
Vincent Ambo
e60805c9b2 feat(server): Introduce function to hash contents of a layer
This creates a cache key which can be used to check if a layer has
already been built.
2019-10-03 13:21:04 +01:00
Vincent Ambo
9c3c622403 refactor(server): Expose layer grouping logic via a function
Refactors the layer grouping package (which previously compiled to a
separate binary) to expose the layer grouping logic via a function
instead.

This is the next step towards creating layers inside of the server
component instead of in Nix.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
8c79d085ae chore(server): Import layer grouping logic into server component 2019-10-03 13:21:04 +01:00
Vincent Ambo
0898d8a961 chore(build-image): Simplify wrapper build & remove layer grouping
Simplifies the wrapper script used to invoke Nix builds from Nixery to
just contain the essentials, since the layer grouping logic is moving
into the server itself.
2019-10-03 13:21:04 +01:00
Vincent Ambo
712b38cbbc refactor(build-image): Do not assemble image layers in Nix
This is the first step towards a more granular build process where
some of the build responsibility moves into the server component.

Rather than assembling all layers inside of Nix, it will only create
the symlink forest and return information about the runtime paths
required by the image.

The server is then responsible for grouping these paths into layers,
and assembling the layers themselves.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
ad9b3eb262 refactor(build): Add group-layers to top-level Nix derivations
This makes CI build the group-layers tool (and cache it to Cachix!)
2019-09-22 18:04:40 +01:00
Vincent Ambo
7b987530d1 refactor(build-image): Minor tweak to layer construction script 2019-09-22 18:04:40 +01:00
Vincent Ambo
21a17b33f4 fix(build): Ensure launch script compatibility with other runtimes
Fixes two launch script compatibility issues with other container
runtimes (such as gvisor):

* don't fail if /tmp already exists
* don't fail if the environment becomes unset
2019-09-21 15:25:28 +01:00
Vincent Ambo
0000b956bb feat(server): Log Nix output live during the builds
Instead of dumping all Nix output as one at the end of the build
process, stream it live as the lines come in.

This is a lot more useful for debugging stuff like where manifest
retrievals get stuck.
2019-09-21 15:25:28 +01:00
Vincent Ambo
c391a7b7f8 fix(build-image): Use absolute paths in tarballs 2019-09-21 15:25:28 +01:00
Vincent Ambo
da6fd1d79e fix(build): Ensure nixery-build-image is on Nixery's PATH
This is useful when running Nixery locally.
2019-09-21 12:44:40 +01:00
Vincent Ambo
f0b69638e1 chore(build): Bump nixpkgs version used in Travis
This version matches the updated popularity URL.
2019-09-21 12:44:40 +01:00
Vincent Ambo
64f74abc4d feat: Add configuration option for popularity data URL 2019-09-21 12:44:40 +01:00
Vincent Ambo
e4d03fdb17 chore(docs): Remove mdbook override
The change has been upstreamed in Nixpkgs.
2019-09-21 12:44:40 +01:00
Vincent Ambo
5a002fe067 refactor(builder): Calculate image cache key only once 2019-09-10 11:32:37 +01:00
Vincent Ambo
4a58b0ab4d feat(server): Cache built manifests to the GCS bucket
Caches manifests under `manifests/$cacheKey` in the GCS bucket and
introduces two-tiered retrieval of manifests from the caches (local
first, bucket second).

There is some cleanup to be done in this code, but the initial version
works.
2019-09-10 11:32:37 +01:00
Vincent Ambo
051eb77b3d refactor(server): Use package source specific cache keys
Use the PackageSource.CacheKey function introduced in the previous
commit to determine the key at which a manifest should be cached in
the local cache.

Due to this change, manifests for moving target sources are no longer
cached and the recency threshold logic has been removed.
2019-09-10 11:32:37 +01:00
Vincent Ambo
980f5e2187 refactor(server): Move package source management logic to server
Introduces three new types representing each of the possible package
sources and moves the logic for specifying the package source to the
server.

Concrete changes:

* Determining whether a specified git reference is a commit vs. a
  branch/tag is now done in the server, and is done more precisely by
  using a regular expression.

* Package sources now have a new `CacheKey` function which can be used
  to retrieve a key under which a build manifest can be cached *if*
  the package source is not a moving target (i.e. a full git commit
  hash of either nixpkgs or a private repository).

  This function is not yet used.

* Users *must* now specify a package source, Nixery no longer defaults
  to anything and will fail to launch if no source is configured.
2019-09-10 11:32:37 +01:00
Vincent Ambo
bcd7710be5 feat(tools): Introduce pass-compatible wrapper using Cloud KMS
Adds a shell script that supports a subset of the 'pass' interface for
compatibility with kontemplate, and wraps kontemplate in a script that
places this version on the PATH.

This makes it possible to use Cloud KMS encrypted secrets with kontemplate.
2019-09-03 16:12:30 +01:00
Vincent Ambo
496a4ab847 docs: Add information about NIX_TIMEOUT variable 2019-09-02 23:44:57 +01:00
Vincent Ambo
32b9b5099e feat(server): Add configuration option for Nix build timeouts
Adds a NIX_TIMEOUT environment variable which can be set to a number
of seconds that is the maximum allowed time each Nix builder can run.

By default this is set to 60 seconds, which should be plenty for most
use-cases as Nixery is not expected to be performing builds of
uncached binaries in most production cases.

Currently the errors Nix throws on a build timeout are not separated
from other types of errors, meaning that users will see a generic 500
server error in case of a timeout.

This fixes #47
2019-09-02 23:44:57 +01:00
Vincent Ambo
f2e0f3ee27 chore(third_party): Remove git-appraise
Not actually in use here ...
2019-09-02 20:01:25 +01:00
Vincent Ambo
640b497950 feat(tools): Add stern, a k8s log watcher 2019-09-02 18:18:28 +01:00
Vincent Ambo
ce8635833b refactor(build-image): Remove implicit import of entire package set
Explicitly refer to where things come from, and also don't import
dockerTools as it is no longer used for anything.
2019-09-02 00:08:11 +01:00
Vincent Ambo
92270fcbe4 refactor(build-image): Simplify customisation layer builder
Moves the relevant parts of the customisation layer construction from
dockerTools.mkCustomisationLayer into the Nixery code base.

The version in dockerTools builds additional files (including via
hashing of potentially large files) which are not required when
serving an image over the registry protocol.
2019-09-02 00:08:11 +01:00
Vincent Ambo
6450347bf1 chore(tools): Remove ormolu from tools
The Ormolu derivation is too large to be instantiated on-demand. I've
resorted to installing this tool into my profile instead.
2019-08-25 20:17:03 +01:00
Vincent Ambo
4f88851971 fix(blog_cli): Quote JSON output for post dates 2019-08-23 11:24:03 +01:00
Vincent Ambo
57059fec2b feat(blog_cli): Add flag to set post date and serialise as date only 2019-08-23 10:57:15 +01:00
Vincent Ambo
306e12787a chore(build): Add iana-etc to Nixery's own image
This package is used by a variety of programs that users may want to
embed into Nixery in addition, for example cachix, but those packages
don't refer to it explicitly.
2019-08-21 10:35:32 +01:00
Vincent Ambo
bb5427a47a chore(docs): Update embedded nix-1p version
The new version of the document has syntactic fixes that render pipes
in code blocks in tables correctly across dialects.

Fixes #44
2019-08-21 10:35:32 +01:00
Vincent Ambo
b856648127 feat(tools): Introduce ormolu for formatting Haskell code 2019-08-20 00:26:54 +01:00
Vincent Ambo
1d5b53abf8 chore(tools/blog_cli): Shorten metadata key names
Every character counts because I want to keep metadata below 255
bytes.
2019-08-19 19:11:32 +01:00
Vincent Ambo
d5710db900 refactor(tools/blog_cli): More sensible chunk encoding as records
Instead of many rrdatas in one record, make many records!
2019-08-19 17:51:29 +01:00
Vincent Ambo
2a16740445 feat(tools): Add a horrifying tool to write blog posts into DNS 2019-08-19 16:59:55 +01:00
Vincent Ambo
daa6196c2a fix(build): Force nix-env to use NIX_PATH
Thanks to clever!
2019-08-19 01:56:17 +01:00
Vincent Ambo
ccf6a95f94 chore(build): Pin nixpkgs to a specific commit
This is the same commit for which Nixery has popularity data, but that
isn't particularly relevant.
2019-08-19 01:56:17 +01:00
Vincent Ambo
e7d7f73f7d feat(build): Add 'extraPackages' parameter
This makes it possible to inject additional programs (e.g. Cachix)
into a Nixery container.
2019-08-19 01:56:17 +01:00
Vincent Ambo
ffae282eac fix(docs): Correct link to layering strategy 2019-08-19 01:56:17 +01:00
Vincent Ambo
745b7ce0b8 fix(build): Ensure root user is known inside of container
This is required by git in cases where Nixery is configured with a
custom git repository.

I've also added a shell back into the image to make debugging a
running Nixery easier. It turns out some of the dependencies already
pull in bash anyways, so this is just surfacing it to $PATH.
2019-08-17 10:42:13 +01:00
Vincent Ambo
9a95c4124f fix(server): Sort requested packages in image name & spec
Before this change, Nixery would pass on the image name unmodified to
Nix which would lead it to cache-bust the manifest and configuration
layers for images that are content-identical but have different
package ordering.

This fixes #38.
2019-08-17 10:31:41 +01:00
Vincent Ambo
0ee239874b docs(README): Update links to layering strategy 2019-08-17 10:10:41 +01:00
Florian Klink
3b65fc8c72 feat(server): add iana-etc and cacert to the shell convenience package
These probably should be part of every container image by default, but
adding it to the "shell" convenience name probably is our best bet for
now.
2019-08-16 22:47:22 +01:00
Vincent Ambo
27036e18f5 feat(tools): Add kontemplate 2019-08-16 16:50:50 +01:00
Vincent Ambo
b652ea2f03 feat(tools): Add Terraform 0.12 to local tools
This is pre-configured with the GCP provider.
2019-08-15 16:28:42 +01:00
Vincent Ambo
38493b83ea chore(tools): Remove bazel & ghc 2019-08-15 16:15:59 +01:00
Vincent Ambo
501e6ded5f fix(build): Ensure GCS signing is used in CI 2019-08-15 16:04:34 +01:00
Vincent Ambo
3f232e0170 docs: Add asciinema demo to README & book 2019-08-15 16:04:34 +01:00
Vincent Ambo
0ec369d76c docs(book): Update information on new layering strategy 2019-08-15 16:04:34 +01:00
Vincent Ambo
ca1ffb397d feat(build): Add an integration test that runs on Travis
This test, after performing the usual Nixery build, loads the built
image into Docker, runs it, pulls an image from Nixery and runs that
image.

To make this work, there is some configuration on the Travis side.
Most importantly, the following environment variables have special
values:

* `GOOGLE_KEY`: This is set to a base64-encoded service account key to
  be used in the test.
* `GCS_SIGNING_PEM`: This is set to a base64-encoded signing key (in
  PEM) that is used for signing URLs.

Both of these are available to all branches in the Nixery repository.
2019-08-15 00:26:57 +01:00
Vincent Ambo
85b9c30749 chore(server): Add 'go vet' to build process 2019-08-14 20:18:41 +01:00
Vincent Ambo
36d50d1f19 fix(server): Print correct project name during startup
They grow up so fast :')
2019-08-14 20:18:41 +01:00
Vincent Ambo
cf227c153f feat(builder): Implement build cache for manifests & layers
Implements a cache that keeps track of:

a) Manifests that have already been built (for up to 6 hours)
b) Layers that have already been seen (and uploaded to GCS)

This significantly speeds up response times for images that are full
or partial matches with previous images served by an instance.
2019-08-14 20:18:41 +01:00
Vincent Ambo
58380e3313 refactor(server): Extract build logic into separate module
This module is going to get more complex as the implementation of #32
progresses.
2019-08-14 20:18:41 +01:00
Vincent Ambo
d9168e3e4d refactor(build-image): Extract package set loading into helper
Some upcoming changes might require the Nix build to be split into
multiple separate nix-build invocations of different expressions, thus
splitting this out is useful.

It also fixes an issue where `build-image/default.nix` might be called
in an environment where no Nix channels are configured.
2019-08-14 00:02:04 +01:00
Vincent Ambo
3939722063 style: Apply nixfmt to trivial Nix files
ALl the ones except for build-image.nix are considered trivial. On the
latter, nixfmt makes some useful changes but by-and-large it is not
ready for that code yet.
2019-08-14 00:02:04 +01:00
Vincent Ambo
6285cd8dbf feat(build-image): Use new image layering algorithm for images
Removes usage of the old layering algorithm and replaces it with the
new one.

Apart from the new layer layout this means that each layer is now
built in a separate derivation, which hopefully leads to better
cacheability.
2019-08-14 00:02:04 +01:00
Vincent Ambo
7214d0aa4f feat(build-image): Introduce a terrifying hack to build group-layers
The issue is described in detail in a comment in
`build-image/default.nix`, please read it.
2019-08-14 00:02:04 +01:00
Vincent Ambo
f60f702274 feat: Add shell.nix for running a local Nixery 2019-08-14 00:02:04 +01:00
Vincent Ambo
6035bf36eb feat(popcount): Clean up popularity counting script
Adds the script used to generate the popularity information for all of
nixpkgs.

The README lists the (currently somewhat rough) usage instructions.
2019-08-14 00:02:04 +01:00
Vincent Ambo
6d718bf271 refactor(server): Use wrapper script to avoid path dependency
Instead of requiring the server component to be made aware of the
location of the Nix builder via environment variables, this commit
introduces a wrapper script for the builder that can simply exist on
the builders $PATH.

This is one step towards a slightly nicer out-of-the-box experience
when using `nix-build -A nixery-bin`.
2019-08-14 00:02:04 +01:00
Vincent Ambo
819b460278 chore(docs): Move mdBook derivation to docs/default.nix 2019-08-14 00:02:04 +01:00
Vincent Ambo
1fa93fe6f6 refactor: Move registry server to a subfolder 2019-08-14 00:02:04 +01:00
Vincent Ambo
d699f7f91c chore(build): Update Go dependencies & add gonum 2019-08-14 00:02:04 +01:00
Vincent Ambo
56a426952c feat(group-layers): Finish layering algorithm implementation
This commit adds the actual logic for extracting layer groupings and
merging them until the layer budget is satisfied.

The implementation conforms to the design doc as of the time of this
commit.
2019-08-14 00:02:04 +01:00
Vincent Ambo
590ce994bb feat(group-layers): Add initial popcount scripts
This script generates an entry in a text file for each time a
derivation is referred to by another in nixpkgs.

For initial testing, this output can be turned into group-layers
compatible JSON with this *trivial* invocation:

cat output | awk '{ print "{\"" $2 "\":" $1 "}"}' | jq -s '. | add | with_entries(.key |= sub("/nix/store/[a-z0-9]+-";""))' > test-data.json
2019-08-14 00:02:04 +01:00
Vincent Ambo
92078527db feat(group-layers): Add preliminary size & popularity considerations
As described in the design document, this adds considerations for
closure size and popularity. All closures meeting a certain threshold
for either value will have an extra edge from the image root to
themselves inserted in the graph, which will cause them to be
considered for inclusion in a separate layer.

This is preliminary because popularity is considered as a boolean
toggle (the input I generated only contains the top ~200 most popular
packages), but it should be using either absolute popularity values or
percentiles (needs some experimentation).
2019-08-14 00:02:04 +01:00
Vincent Ambo
ce31598f42 feat(group-layers): Implement first half of new layering strategy
The strategy is described in-depth in the comment at the top of the
implementation file, as well as in the design document:

https://storage.googleapis.com/nixdoc/nixery-layers.html
2019-08-14 00:02:04 +01:00
Vincent Ambo
3e385dc379 docs: Update embedded nix-1p
The new commit has an operator table, which is nice to have!
2019-08-08 21:15:50 +01:00
Vincent Ambo
c727b3ca9e chore(nix): Increase maximum number of layers to 96
This uses a significantly larger percentage of the total available
layers (125) than before, which means that cache hits for layers
become more likely between images.
2019-08-08 21:15:50 +01:00
Vincent Ambo
993fda3377 fix(go): Fix breakage in unsigned URLs
This affected the public instance which is still running without URL
signing. Should add some monitoring!
2019-08-08 18:58:23 +01:00
Vincent Ambo
12a853fab7 docs: Minor fixes to README after new website release 2019-08-05 01:26:19 +01:00
Vincent Ambo
d87662b7b5 docs: Add a section on Nix itself 2019-08-05 00:32:53 +01:00
Vincent Ambo
6293d69fd9 docs: Add a section on running your own Nixery 2019-08-05 00:32:53 +01:00
Vincent Ambo
a3f6278913 docs: Add an "under-the-hood" page explaining the build process
This page describes the various steps that Nixery goes through when
"procuring" an image.

The intention is to give users some more visibility into what is going
on and to make it clear that this is not just an image storage
service.
2019-08-05 00:32:53 +01:00
Vincent Ambo
2bef0ba240 feat(build): Build Nixery book and embed it into Nixery image
Executes the previously added mdBook on the previously added book
source to yield a directory that can be served by Nixery on its index
page.

This is one of those 'I <3 Nix' things due to how easy it is to do.
2019-08-05 00:32:53 +01:00
Vincent Ambo
85e8d760fc feat(build): Add mdBook 0.3.1 to build environment
Upstream nixpkgs currently only has an older versin of mdBook. Until
that changes, we keep a different version in here.
2019-08-05 00:32:53 +01:00
Vincent Ambo
7c41a7a872 docs: Replace static page with mdBook site
Uses mdBook[1] to generate a documentation overview page instead of
the previous HTML site.

This makes it possible to add more elaborate documentation without
having to deal with finicky markup.

[1]: https://github.com/rust-lang-nursery/mdBook
2019-08-05 00:32:53 +01:00
Vincent Ambo
099c99b7ad feat(build): Configure Cachix for build caching in CI
The CI setup is configured with an appropriate key to enable pushes to
the nixery.cachix.org binary cache.
2019-08-04 01:41:35 +01:00
Vincent Ambo
a0d7d693d3 feat(build): Support additional pre-launch commands in image
This makes it possible for users to hook basically arbitrary things
into the Nixery container image.
2019-08-04 01:41:35 +01:00
Vincent Ambo
20103640fa fix(nix): Support retrieving differently cased top-level attributes
As described in issue #14, the registry API does not allow image names
with uppercase-characters in them.

However, the Nix package set has several top-level keys with uppercase
characters in them which could previously not be retrieved using
Nixery.

This change implements a method for retrieving those keys, but it is
explicitly only working for the top-level package set as nested
sets (such as `haskellPackages`) often contain packages that differ in
case only.
2019-08-03 23:11:40 +01:00
Vincent Ambo
aa260af1ff docs: Add GCS signing envvars to README 2019-08-03 01:31:51 +01:00
Vincent Ambo
07ef06dcfa feat(go): Support signed GCS URLs with static keys
Google Cloud Storage supports granting access to protected objects via
time-restricted URLs that are cryptographically signed.

This makes it possible to store private data in buckets and to
distribute it to eligible clients without having to make those clients
aware of GCS authentication methods.

Nixery now uses this feature to sign URLs for GCS buckets when
returning layer URLs to clients on image pulls. This means that a
private Nixery instance can run a bucket with restricted access just
fine.

Under the hood Nixery uses a key provided via environment
variables to sign the URL with a 5 minute expiration time.

This can be set up by adding the following two environment variables:

* GCS_SIGNING_KEY: Path to the PEM file containing the signing key.
* GCS_SIGNING_ACCOUNT: Account ("e-mail" address) to use for signing.

If the variables are not set, the previous behaviour is not modified.
2019-08-03 01:31:51 +01:00
Vincent Ambo
3347c38ba7 fix(go): Registry API acknowledgement URI has a trailing slash
Previously the acknowledgement calls from Docker were receiving a
404 (which apparently doesn't bother it?!). This corrects the URL,
which meant that acknowledgement had to move inside of the
registryHandler.
2019-08-03 01:25:36 +01:00
Vincent Ambo
ecee1ec1b8 chore: Prevent accidental key leaks via gitignore 2019-08-03 01:25:36 +01:00
Vincent Ambo
62ade18b7b fix(static): Fix logo nitpick (smoothened λ edges) 2019-08-03 01:25:36 +01:00
Vincent Ambo
c84543a9b5 style(static): Fix favicon background colour 2019-08-03 01:25:36 +01:00
Vincent Ambo
a4c0d3e8d3 chore(go): Remove 'builder' metapackage
This metapackage isn't actually particularly useful (stdenv is rarely
what users want).
2019-08-03 01:25:36 +01:00
Vincent Ambo
da5df525c8 docs: Update all nixery.appspot.com references to nixery.dev
Shiny, new domain is much better and eliminates the TLS redirect issue
because there is a HSTS preload for the entire .dev TLD (which, by the
way, is awesome!)
2019-08-02 17:14:06 +01:00
Vincent Ambo
1f885e43b6 style(static): Update Nixery logo to a healthier version
This might not yet be the final version, but it's going in the right
direction.

Additionally the favicon has been reduced to just the coloured Nix
logo, because details are pretty much invisible at that size anyways.
2019-08-02 17:14:06 +01:00
Vincent Ambo
02dfff393a fix(build): coreutils are still required by launch script
Mea culpa!
2019-08-02 01:33:39 +01:00
Vincent Ambo
92f1758014 docs(static): Note that the demo instance is just a demo
People should not start depending on the demo instance. There have
been discussions around making a NixOS-official instance, but the
project needs to mature a little bit first.
2019-08-02 01:17:36 +01:00
Vincent Ambo
bf34bb327c fix(nix): Calculate MD5 sum of config layer correctly
The MD5 sum is used for verifying contents in the layer cache before
accidentally re-uploading, but the syntax of the hash invocation was
incorrect leading to a cache-bust on the manifest layer on every
single build (even for identical images).
2019-08-02 01:08:14 +01:00
Vincent Ambo
3d0596596a feat(go): Return error responses in registry format
The registry specifies a format for how errors should be returned and
this commit implements it:

https://docs.docker.com/registry/spec/api/#errors
2019-08-02 01:08:14 +01:00
Vincent Ambo
119af77b43 feat(go): Return errors with correct status codes to clients
Uses the structured errors feature introduced in the Nix code to
return more sensible errors to clients. For now this is quite limited,
but already a lot better than before:

* packages that could not be found result in 404s
* all other errors result in 500s

This way the registry clients will not attempt to interpret the
returned garbage data/empty response as something useful.
2019-08-02 01:08:14 +01:00
Vincent Ambo
2f1bc55597 fix(go): Return response code 500 if Nix builds fail 2019-08-02 01:08:14 +01:00
Vincent Ambo
3070d88051 feat(nix): Return structured errors if packages are not found
Changes the return format of Nixery's build procedure to return a JSON
structure that can indicate which errors have occured.

The server can use this information to send appropriate status codes
back to clients.
2019-08-02 01:08:14 +01:00
Vincent Ambo
ec8e9eed5d docs(README): Revamp with updated information on package sources
Adds documentation for configuration options and supported features.
2019-07-31 15:28:36 +01:00
Vincent Ambo
3bc04530a7 feat(go): Add environment configuration for package set sources
Adds environment variables with which users can configure the package
set source to use. Not setting a source lets Nix default to a recent
NixOS channel (currently nixos-19.03).
2019-07-31 15:28:36 +01:00
Vincent Ambo
2db92243e7 feat(nix): Support package set imports from different sources
This extends the package set import mechanism in
build-registry-image.nix with several different options:

1. Importing a nixpkgs channel from Github (the default, pinned to
   nixos-19.03)

2. Importing a custom Nix git repository. This uses builtins.fetchGit
   and can thus rely on git/SSH configuration in the environment (such
   as keys)

3. Importing a local filesystem path

As long as the repository pointed at is either a checkout of nixpkgs,
or nixpkgs overlaid with custom packages this will work.

A special syntax has been defined for how these three options are
passed in, but users should not need to concern themselves with it as
it will be taken care of by the server component.

This relates to #3.
2019-07-31 15:28:36 +01:00
Vincent Ambo
a83701a14b feat(build): Add dependencies for custom repo clones
Adds git & SSH as part of the Nixery image, which are required to use
Nix's builtins.fetchGit.

The dependency on interactive tools is dropped, as it was only
required during development when debugging the image building process
itself.
2019-07-31 15:28:36 +01:00
Vincent Ambo
2e4b1f85ee fix(nix): Add empty image config to allow k8s usage
Introduce an empty runtime configuration object in each built layer.

This is required because Kubernetes expects the configuration to be
present (even if it's just empty values).

Providing an empty configuration will make Docker's API return a full
configuration struct with default (i.e. empty) values rather than
`null`, which works for Kubernetes.

This fixes issue #1. See the issue for additional details.
2019-07-31 00:02:27 +01:00
Vincent Ambo
4802727408 docs(static): Update index page with post-launch information
Points people at the repository and removes some outdated information.
2019-07-30 13:42:43 +01:00
Vincent Ambo
9753df9255 docs(README): Add logo & build status 2019-07-30 13:42:43 +01:00
Vincent Ambo
1b37d8ecf6 feat(static): Add logo & favicon resources 2019-07-30 13:42:43 +01:00
Vincent Ambo
02eba0336e refactor(main): Introduce more flexible request routing
Instead of just dispatching on URL regexes, use handlers to split the
routes into registry-related handlers and otherwise(tm).

For now the otherwise(tm) consists of a file server serving the static
directory, rather than just a plain match on the index route.
2019-07-30 13:42:43 +01:00
Vincent Ambo
fc2e508ab8 feat(build): Add Travis configuration to build everything
The default Travis build command for Nix is `nix-build`, which will
build all derivations specified in the default.nix.
2019-07-30 13:42:43 +01:00
Vincent Ambo
90948a48e1 docs(CONTRIBUTING): Mention commit message format 2019-07-29 21:10:04 +01:00
Vincent Ambo
33d876fda8 docs(README): Update roadmap information
Adds information about Kubernetes integration & custom repository
support as well as links to the relevant tracking issues.
2019-07-29 21:10:04 +01:00
Vincent Ambo
93a3985298 docs(README): Remove known issues from README
These issues have been moved to the issue tracker.
2019-07-29 21:10:04 +01:00
Vincent Ambo
6207782434 fix(build): Specify default command for Nixery's own image
When running on AppEngine, the image is expected to be configured with
a default entry point / command.

This sets the command to the wrapper script, so that the image can
actually run properly when deployed.
2019-07-26 10:22:31 +00:00
Vincent Ambo
6dd0ac3189 feat(nix): Import nixpkgs from a configured Nix channel
Instead of using whatever the current system default is, import a Nix
channel when building an image.

This will use Nix' internal caching behaviour for tarballs fetched
without a SHA-hash.

For now the downloaded channel is pinned to nixos-19.03.
2019-07-24 17:53:08 +00:00
Vincent Ambo
948f308025 feat(build): Configure Nixery image builder to set up env correctly
When running Nix inside of a container image, there are several
environment-specific details that need to be configured appropriately.

Most importantly, since one of the recent Nix 2.x releases, sandboxing
during builds is enabled by default. This, however, requires kernel
privileges which commonly aren't available to containers.

Nixery's demo instance (for instance, hehe) is deployed on AppEngine
where this type of container configuration is difficult, hence this
change.

Specifically the following were changed:

* additional tools (such as tar/gzip) were introduced into the image
  because the builtins-toolset in Nix does not reference these tools
  via their store paths, which leads to them not being included
  automatically
* Nix sandboxing was disabled in the container image
* the users/groups required by Nix were added to the container setup.
  Note that these are being configured manually instead of via the
  tools from the 'shadow'-package, because the latter requires some
  user information (such as root) to be present already, which is not
  the case inside of the container
2019-07-24 17:46:39 +00:00
Vincent Ambo
18b4ae9f28 chore: Remove AppEngine configuration file 2019-07-23 22:37:40 +00:00
Vincent Ambo
28bb3924ff chore: Add gitignore to ignore Nix build results 2019-07-23 23:33:22 +01:00
Vincent Ambo
23260e59d9 chore: Add license scaffolding & contribution guidelines 2019-07-23 23:32:56 +01:00
Vincent Ambo
5f471392cf feat(build): Add wrapper script & container image setup
Introduces a wrapper script which automatically sets the paths to the
required runtime data dependencies.

Additionally configures a container image derivation which will output
a derivation with Nixery, Nix and other dependencies.
2019-07-23 23:23:21 +01:00
Vincent Ambo
8314568199 style(main): Reflow comments to 80 characters maximum 2019-07-23 23:23:21 +01:00
Vincent Ambo
db1086a5bb feat(main): Add additional envvars to configure Nixery
Previously the code had hardcoded paths to runtime data (the Nix
builder & web files), which have now been moved into configuration
options.

Additionally configuration for the application is now centralised in a
single config struct, an instance of which is passed around the
application.

This makes it possible to implement a wrapper in Nix that will
configure the runtime data locations automatically.
2019-07-23 23:23:21 +01:00
Vincent Ambo
5d9b32977d feat(build): Introduce build configuration using Nix
Rather than migrating to Bazel, it seems more appropriate to use Nix
for this project.

The project is split into several different components (for data
dependencies and binaries). A derivation for building an image for
Nixery itself will be added.
2019-07-23 21:48:27 +01:00
Vincent Ambo
3042444757 chore: Import Nixery from experimental
Moves the existing Nixery code base to a git repository and switches
to public equivalents of libraries used.
2019-07-23 20:53:38 +01:00
Vincent Ambo
27c3e77d2f feat(tools): Add custom GHC with dependencies to env 2019-07-04 15:57:12 +01:00
Vincent Ambo
2eb1dc26e4 feat(tools): Add symlinks for bazel & stylish-haskell 2019-07-04 11:17:34 +01:00
Vincent Ambo
b2f40b6ed8 fix(tools): Ensure dispatch script passes arguments correctly 2019-07-04 11:15:15 +01:00
Vincent Ambo
892493a478 feat(tools): Add dispatcher script to transparently access tools
Initial version of tool provider via Nix. This requires two separate
steps for adding a new tool:

1. New symlink in tools/bin to point at the dispatch script.
2. Mapping of tool to Nix package set attribute in dispatch script.
2019-07-02 16:40:51 +01:00