Commit graph

550 commits

Author SHA1 Message Date
Vincent Ambo
2bef0ba240 feat(build): Build Nixery book and embed it into Nixery image
Executes the previously added mdBook on the previously added book
source to yield a directory that can be served by Nixery on its index
page.

This is one of those 'I <3 Nix' things due to how easy it is to do.
2019-08-05 00:32:53 +01:00
Vincent Ambo
85e8d760fc feat(build): Add mdBook 0.3.1 to build environment
Upstream nixpkgs currently only has an older versin of mdBook. Until
that changes, we keep a different version in here.
2019-08-05 00:32:53 +01:00
Vincent Ambo
7c41a7a872 docs: Replace static page with mdBook site
Uses mdBook[1] to generate a documentation overview page instead of
the previous HTML site.

This makes it possible to add more elaborate documentation without
having to deal with finicky markup.

[1]: https://github.com/rust-lang-nursery/mdBook
2019-08-05 00:32:53 +01:00
Vincent Ambo
099c99b7ad feat(build): Configure Cachix for build caching in CI
The CI setup is configured with an appropriate key to enable pushes to
the nixery.cachix.org binary cache.
2019-08-04 01:41:35 +01:00
Vincent Ambo
a0d7d693d3 feat(build): Support additional pre-launch commands in image
This makes it possible for users to hook basically arbitrary things
into the Nixery container image.
2019-08-04 01:41:35 +01:00
Vincent Ambo
20103640fa fix(nix): Support retrieving differently cased top-level attributes
As described in issue #14, the registry API does not allow image names
with uppercase-characters in them.

However, the Nix package set has several top-level keys with uppercase
characters in them which could previously not be retrieved using
Nixery.

This change implements a method for retrieving those keys, but it is
explicitly only working for the top-level package set as nested
sets (such as `haskellPackages`) often contain packages that differ in
case only.
2019-08-03 23:11:40 +01:00
Vincent Ambo
aa260af1ff docs: Add GCS signing envvars to README 2019-08-03 01:31:51 +01:00
Vincent Ambo
07ef06dcfa feat(go): Support signed GCS URLs with static keys
Google Cloud Storage supports granting access to protected objects via
time-restricted URLs that are cryptographically signed.

This makes it possible to store private data in buckets and to
distribute it to eligible clients without having to make those clients
aware of GCS authentication methods.

Nixery now uses this feature to sign URLs for GCS buckets when
returning layer URLs to clients on image pulls. This means that a
private Nixery instance can run a bucket with restricted access just
fine.

Under the hood Nixery uses a key provided via environment
variables to sign the URL with a 5 minute expiration time.

This can be set up by adding the following two environment variables:

* GCS_SIGNING_KEY: Path to the PEM file containing the signing key.
* GCS_SIGNING_ACCOUNT: Account ("e-mail" address) to use for signing.

If the variables are not set, the previous behaviour is not modified.
2019-08-03 01:31:51 +01:00
Vincent Ambo
3347c38ba7 fix(go): Registry API acknowledgement URI has a trailing slash
Previously the acknowledgement calls from Docker were receiving a
404 (which apparently doesn't bother it?!). This corrects the URL,
which meant that acknowledgement had to move inside of the
registryHandler.
2019-08-03 01:25:36 +01:00
Vincent Ambo
ecee1ec1b8 chore: Prevent accidental key leaks via gitignore 2019-08-03 01:25:36 +01:00
Vincent Ambo
62ade18b7b fix(static): Fix logo nitpick (smoothened λ edges) 2019-08-03 01:25:36 +01:00
Vincent Ambo
c84543a9b5 style(static): Fix favicon background colour 2019-08-03 01:25:36 +01:00
Vincent Ambo
a4c0d3e8d3 chore(go): Remove 'builder' metapackage
This metapackage isn't actually particularly useful (stdenv is rarely
what users want).
2019-08-03 01:25:36 +01:00
Vincent Ambo
da5df525c8 docs: Update all nixery.appspot.com references to nixery.dev
Shiny, new domain is much better and eliminates the TLS redirect issue
because there is a HSTS preload for the entire .dev TLD (which, by the
way, is awesome!)
2019-08-02 17:14:06 +01:00
Vincent Ambo
1f885e43b6 style(static): Update Nixery logo to a healthier version
This might not yet be the final version, but it's going in the right
direction.

Additionally the favicon has been reduced to just the coloured Nix
logo, because details are pretty much invisible at that size anyways.
2019-08-02 17:14:06 +01:00
Vincent Ambo
02dfff393a fix(build): coreutils are still required by launch script
Mea culpa!
2019-08-02 01:33:39 +01:00
Vincent Ambo
92f1758014 docs(static): Note that the demo instance is just a demo
People should not start depending on the demo instance. There have
been discussions around making a NixOS-official instance, but the
project needs to mature a little bit first.
2019-08-02 01:17:36 +01:00
Vincent Ambo
bf34bb327c fix(nix): Calculate MD5 sum of config layer correctly
The MD5 sum is used for verifying contents in the layer cache before
accidentally re-uploading, but the syntax of the hash invocation was
incorrect leading to a cache-bust on the manifest layer on every
single build (even for identical images).
2019-08-02 01:08:14 +01:00
Vincent Ambo
3d0596596a feat(go): Return error responses in registry format
The registry specifies a format for how errors should be returned and
this commit implements it:

https://docs.docker.com/registry/spec/api/#errors
2019-08-02 01:08:14 +01:00
Vincent Ambo
119af77b43 feat(go): Return errors with correct status codes to clients
Uses the structured errors feature introduced in the Nix code to
return more sensible errors to clients. For now this is quite limited,
but already a lot better than before:

* packages that could not be found result in 404s
* all other errors result in 500s

This way the registry clients will not attempt to interpret the
returned garbage data/empty response as something useful.
2019-08-02 01:08:14 +01:00
Vincent Ambo
2f1bc55597 fix(go): Return response code 500 if Nix builds fail 2019-08-02 01:08:14 +01:00
Vincent Ambo
3070d88051 feat(nix): Return structured errors if packages are not found
Changes the return format of Nixery's build procedure to return a JSON
structure that can indicate which errors have occured.

The server can use this information to send appropriate status codes
back to clients.
2019-08-02 01:08:14 +01:00
Vincent Ambo
ec8e9eed5d docs(README): Revamp with updated information on package sources
Adds documentation for configuration options and supported features.
2019-07-31 15:28:36 +01:00
Vincent Ambo
3bc04530a7 feat(go): Add environment configuration for package set sources
Adds environment variables with which users can configure the package
set source to use. Not setting a source lets Nix default to a recent
NixOS channel (currently nixos-19.03).
2019-07-31 15:28:36 +01:00
Vincent Ambo
2db92243e7 feat(nix): Support package set imports from different sources
This extends the package set import mechanism in
build-registry-image.nix with several different options:

1. Importing a nixpkgs channel from Github (the default, pinned to
   nixos-19.03)

2. Importing a custom Nix git repository. This uses builtins.fetchGit
   and can thus rely on git/SSH configuration in the environment (such
   as keys)

3. Importing a local filesystem path

As long as the repository pointed at is either a checkout of nixpkgs,
or nixpkgs overlaid with custom packages this will work.

A special syntax has been defined for how these three options are
passed in, but users should not need to concern themselves with it as
it will be taken care of by the server component.

This relates to #3.
2019-07-31 15:28:36 +01:00
Vincent Ambo
a83701a14b feat(build): Add dependencies for custom repo clones
Adds git & SSH as part of the Nixery image, which are required to use
Nix's builtins.fetchGit.

The dependency on interactive tools is dropped, as it was only
required during development when debugging the image building process
itself.
2019-07-31 15:28:36 +01:00
Vincent Ambo
2e4b1f85ee fix(nix): Add empty image config to allow k8s usage
Introduce an empty runtime configuration object in each built layer.

This is required because Kubernetes expects the configuration to be
present (even if it's just empty values).

Providing an empty configuration will make Docker's API return a full
configuration struct with default (i.e. empty) values rather than
`null`, which works for Kubernetes.

This fixes issue #1. See the issue for additional details.
2019-07-31 00:02:27 +01:00
Vincent Ambo
4802727408 docs(static): Update index page with post-launch information
Points people at the repository and removes some outdated information.
2019-07-30 13:42:43 +01:00
Vincent Ambo
9753df9255 docs(README): Add logo & build status 2019-07-30 13:42:43 +01:00
Vincent Ambo
1b37d8ecf6 feat(static): Add logo & favicon resources 2019-07-30 13:42:43 +01:00
Vincent Ambo
02eba0336e refactor(main): Introduce more flexible request routing
Instead of just dispatching on URL regexes, use handlers to split the
routes into registry-related handlers and otherwise(tm).

For now the otherwise(tm) consists of a file server serving the static
directory, rather than just a plain match on the index route.
2019-07-30 13:42:43 +01:00
Vincent Ambo
fc2e508ab8 feat(build): Add Travis configuration to build everything
The default Travis build command for Nix is `nix-build`, which will
build all derivations specified in the default.nix.
2019-07-30 13:42:43 +01:00
Vincent Ambo
90948a48e1 docs(CONTRIBUTING): Mention commit message format 2019-07-29 21:10:04 +01:00
Vincent Ambo
33d876fda8 docs(README): Update roadmap information
Adds information about Kubernetes integration & custom repository
support as well as links to the relevant tracking issues.
2019-07-29 21:10:04 +01:00
Vincent Ambo
93a3985298 docs(README): Remove known issues from README
These issues have been moved to the issue tracker.
2019-07-29 21:10:04 +01:00
Vincent Ambo
6207782434 fix(build): Specify default command for Nixery's own image
When running on AppEngine, the image is expected to be configured with
a default entry point / command.

This sets the command to the wrapper script, so that the image can
actually run properly when deployed.
2019-07-26 10:22:31 +00:00
Vincent Ambo
6dd0ac3189 feat(nix): Import nixpkgs from a configured Nix channel
Instead of using whatever the current system default is, import a Nix
channel when building an image.

This will use Nix' internal caching behaviour for tarballs fetched
without a SHA-hash.

For now the downloaded channel is pinned to nixos-19.03.
2019-07-24 17:53:08 +00:00
Vincent Ambo
948f308025 feat(build): Configure Nixery image builder to set up env correctly
When running Nix inside of a container image, there are several
environment-specific details that need to be configured appropriately.

Most importantly, since one of the recent Nix 2.x releases, sandboxing
during builds is enabled by default. This, however, requires kernel
privileges which commonly aren't available to containers.

Nixery's demo instance (for instance, hehe) is deployed on AppEngine
where this type of container configuration is difficult, hence this
change.

Specifically the following were changed:

* additional tools (such as tar/gzip) were introduced into the image
  because the builtins-toolset in Nix does not reference these tools
  via their store paths, which leads to them not being included
  automatically
* Nix sandboxing was disabled in the container image
* the users/groups required by Nix were added to the container setup.
  Note that these are being configured manually instead of via the
  tools from the 'shadow'-package, because the latter requires some
  user information (such as root) to be present already, which is not
  the case inside of the container
2019-07-24 17:46:39 +00:00
Vincent Ambo
18b4ae9f28 chore: Remove AppEngine configuration file 2019-07-23 22:37:40 +00:00
Vincent Ambo
28bb3924ff chore: Add gitignore to ignore Nix build results 2019-07-23 23:33:22 +01:00
Vincent Ambo
23260e59d9 chore: Add license scaffolding & contribution guidelines 2019-07-23 23:32:56 +01:00
Vincent Ambo
5f471392cf feat(build): Add wrapper script & container image setup
Introduces a wrapper script which automatically sets the paths to the
required runtime data dependencies.

Additionally configures a container image derivation which will output
a derivation with Nixery, Nix and other dependencies.
2019-07-23 23:23:21 +01:00
Vincent Ambo
8314568199 style(main): Reflow comments to 80 characters maximum 2019-07-23 23:23:21 +01:00
Vincent Ambo
db1086a5bb feat(main): Add additional envvars to configure Nixery
Previously the code had hardcoded paths to runtime data (the Nix
builder & web files), which have now been moved into configuration
options.

Additionally configuration for the application is now centralised in a
single config struct, an instance of which is passed around the
application.

This makes it possible to implement a wrapper in Nix that will
configure the runtime data locations automatically.
2019-07-23 23:23:21 +01:00
Vincent Ambo
5d9b32977d feat(build): Introduce build configuration using Nix
Rather than migrating to Bazel, it seems more appropriate to use Nix
for this project.

The project is split into several different components (for data
dependencies and binaries). A derivation for building an image for
Nixery itself will be added.
2019-07-23 21:48:27 +01:00
Vincent Ambo
3042444757 chore: Import Nixery from experimental
Moves the existing Nixery code base to a git repository and switches
to public equivalents of libraries used.
2019-07-23 20:53:38 +01:00
Vincent Ambo
27c3e77d2f feat(tools): Add custom GHC with dependencies to env 2019-07-04 15:57:12 +01:00
Vincent Ambo
2eb1dc26e4 feat(tools): Add symlinks for bazel & stylish-haskell 2019-07-04 11:17:34 +01:00
Vincent Ambo
b2f40b6ed8 fix(tools): Ensure dispatch script passes arguments correctly 2019-07-04 11:15:15 +01:00
Vincent Ambo
892493a478 feat(tools): Add dispatcher script to transparently access tools
Initial version of tool provider via Nix. This requires two separate
steps for adding a new tool:

1. New symlink in tools/bin to point at the dispatch script.
2. Mapping of tool to Nix package set attribute in dispatch script.
2019-07-02 16:40:51 +01:00