Commit graph

212 commits

Author SHA1 Message Date
Vincent Ambo
af337010e9 feat(prepare-image): Ensure /usr/bin/env is always present
This is required by common patterns in shell scripts.

There are some caveats around this. Adding logic to filter whether
coreutils is included in an image would slow down the Nix evaluation,
so the link is currently created even in cases where it doesn't point
to anything.

Fixes #109
2021-08-25 16:49:05 +03:00
Vincent Ambo
02455bd0fd chore(build): Allow passing in a specific commit hash when building
Required for builds where the full repository isn't available (e.g.
from a tarball).
2021-08-06 14:27:21 +03:00
Vincent Ambo
84fb380f57 docs: Update build badge in README
Moves the build badge to point at Github Actions, instead of the old (failing) Travis build
2021-07-15 16:23:42 +02:00
Jérôme Petazzoni
94e04a76b6 feat(storage): Store blob content-type in extended attributes
After the discussion in #116, this stores the blob content types
in extended attributes when using the filesystem backend.

If the underlying filesystem doesn't support extended attributes,
storing blobs won't work; also, if extended attributes get removed,
blobs won't be served anymore. We can relax this behavior if
needed (i.e. log errors but still accept to store or serve blobs).
However, since the Docker Engine (and possibly other container
engines) won't accept to pull images from a registry that doesn't
use correct content types for manifest files, it could be argued
that it's better to give a hard fail. (Otherwise, the container
engine gives cryptic error messages like "missing signature key".)

I can change that behavior (and log errors but still store/serve
blobs to the filesystem) if you think it's better.
2021-06-26 01:27:43 +02:00
Florian Klink
3efbbfcd4e feat(ci): don't mount /var/cache/nixery from tmpfs into docker container
With https://github.com/google/nixery/pull/127, nixery will use extended
attributes to store metadata (when using local storage).

Right now, our integration test mounts a tmpfs to /var/cache/nixery.
However, *user* xattrs aren't supported with tmpfs [1], so setting
xattrs would fail.

To workaround this, use a folder in the current working directory and
hope it's backed by something supporting user xattrs (which is the case
for GitHub Actions).

[1]: https://man7.org/linux/man-pages/man5/tmpfs.5.html#NOTES
2021-06-20 18:33:53 +02:00
Vincent Ambo
768f3986a9 feat(build): Run go vet as a step in the GitHub Actions workflow 2021-04-30 13:27:59 +02:00
Vincent Ambo
13d97c9e51 refactor(build): Pin dependencies using Go modules
Drops the go2nix configuration in favour of pkgs.buildGoModule.

Note that the go.sum file is bloated by issues with cyclic
dependencies in some Google projects, but this large number of
dependencies is not actually built.
2021-04-30 13:27:59 +02:00
Vincent Ambo
5c2db7b8ce chore(build): Use current git commit hash as build version 2021-04-30 13:27:59 +02:00
Vincent Ambo
7520f2cb96 chore: Update default NixOS channel to nixos-20.09 2021-04-30 12:28:17 +02:00
Vincent Ambo
8a1add9ef8 chore(ci): Remove unnecessary commands from new CI setup
* remove a step that was not supposed to be committed ("Do we have
  Docker?")
* remove setup of old temporary storage directory (now done in
  integration script test instead)
* skip creation of out-link for initial Nixery build (to avoid
  cache-busting on the second build)
2021-04-30 11:02:38 +02:00
Florian Klink
7e8295189b docs: document unset GOOGLE_APPLICATION_CREDENTIALS
In case the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is not
set, a redirect to storage.googleapis.com is issued, which means the
underlying bucket objects need to be publicly accessible.

This wasn't really obvious until now, so further clarify it.
2021-04-29 23:55:34 +02:00
Florian Klink
970f492235 feat(ci): add integration tests to GitHub Actions, remove .travis.yaml
This copies the integration tests from `.travis.yaml` into a script,
documents the assumptions it makes, and wires it into GitHub Actions.

Contrary to the travis version, we don't use Nixery's GCS backend, as
handing out access to the bucket used, especially for PRs, needs to be
done carefully.

Adding back GCS to the integration test can be done at a later point,
either by using a mock server, or by only exposing the credentials for
master builds (and have the test script decide on whether
GOOGLE_APPLICATION_CREDENTIALS is set or not).

The previous travis version had some complicated post-mortem log
gathering - instead of doing this, we can just `docker run` nixery, but
fork it into the background with the shell - causing it to still be able
to log its output as it's running.

An additional `--rm` is appended, so the container gets cleaned up on
termination - this allows subsequent runs on non-CI infrastructure (like
developer laptops), without having to manually clean up containers.

Fixes #119.
2021-04-29 23:44:42 +02:00
Florian Klink
ee48bd891c feat(ci): remove unneeded permissions: read-all
We don't intend to label, authenticate or whatever with the
GITHUB_TOKEN, so there's not really a reason to give any broader
permissions than the defaults.
2021-04-29 20:23:15 +02:00
Vincent Ambo
d2767bbe8a feat(ci): Configure initial GitHub Actions setup
Travis is being deprecated, and this might be the best option for now.
2021-04-28 22:50:58 +02:00
Jerome Petazzoni
f172107ef1 feat(storage): Add generic support for content-types
When serving a manifest, it is important to set the content-type
correctly (otherwise pulling an image is likely to give a cryptic
error message, "Error response from daemon: missing signature key").

This makes sure that we set the content-type properly for both
manifests and layers.
2021-04-27 15:39:58 +02:00
Jerome Petazzoni
954953d8ba chore(nix): update channel URL
It looks like NixPkgs channels have moved. Fixing this URL allows
using nixos-20.09, for instance.
2021-04-14 14:10:53 +02:00
Vincent Ambo
3e1d63ccb3 docs: Update README with a link to the NixCon talk 2020-12-05 14:34:09 +01:00
Dave Nicponski
8ad5c55ad2 docs(config): Fix comment typo 2020-12-04 14:17:21 +01:00
Vincent Ambo
cc35bf0fc3 feat(storage): Add support for content-types (GCS only)
Extends storage.Persist to accept a Content-Type argument, which in
the GCS backend is persisted with the object to ensure that the object
is served back with this content-type.

This is not yet implemented for the filesystem backend, where the
parameter is simply ignored.

This should help in the case of clients which expect the returned
objects to have content-types set when, for example, fetching layers
by digest.
2020-10-29 17:07:52 +01:00
Vincent Ambo
8a5c446bab docs: Add a note about a Nix-native builder to the roadmap
... if I don't mention this somewhere I'll probably never do it!
2020-10-27 23:07:31 +01:00
Vincent Ambo
9e5ebb2f4f feat(main): Implement caching of manifests in CAS
To ensure that registry clients which attempt to pull manifests by
their content hash can interact with Nixery, this change implements
persisting image manifests in the CAS in the same way as image layers.

In combination with the previous refactorings this means that Nixery's
serving flow is now compatible with containerd.

I have verified this locally, but CI currently only runs against
Docker and not containerd, which is something I plan to address in a
subsequent PR.

This fixes #102
2020-10-27 15:29:11 +01:00
Vincent Ambo
94570aa83f feat(main): Implement serving of manifests by digest
Modifies the layer serving endpoint to be a generic blob-serving
endpoint that can handle both manifest and layer object "types".

Note that this commit does not yet populate the CAS with any
manifests.
2020-10-27 15:29:11 +01:00
Vincent Ambo
cbbf45b5cb refactor(storage): Rename ServeLayer -> Serve
This is going to be used for general content-addressed objects, and is
not layer specific anymore.
2020-10-27 15:29:11 +01:00
Vincent Ambo
5ce745d104 refactor(main): Split HTTP handlers into separate functions
There is a new handler coming up to fix #102 and I want to avoid
falling into the classic Go trap of creating thousand-line functions.
2020-10-27 14:03:14 +01:00
Vincent Ambo
4ce32adfe8 fix(build): Work around arbitrary new maxLayers restriction 2020-10-27 13:16:38 +01:00
Vincent Ambo
ad0541940f fix(build): Completely remove Cachix from build setup
Installing Cachix started failing on ARM64.
2020-10-27 13:16:38 +01:00
Vincent Ambo
c194c5662b fix(build): Don't use Cachix as the binary cache during builds
Permission changes in the Travis CI Nix builders have caused this to
start failing, as the build user now has insufficient permissions to
use caches.

There may be a way to change the permissions instead, but in the
meantime we will just cause things to rebuild.
2020-07-25 14:43:21 +01:00
Vincent Ambo
bc9742f927 chore(build): Update pinned Go dependencies 2020-07-25 14:43:21 +01:00
Vincent Ambo
987a90510a fix(popcount): Accommodate upstream changes on nixos.org
Channel serving has moved to a new subdomain, and the redirect
semantics have changed. Instead of serving temporary redirects,
permanent redirects are now issued.

I've reported this upstream as a bug, but this workaround will fix it
in the meantime.
2020-07-25 14:43:21 +01:00
Vincent Ambo
b4e0b55e56 chore(build): Change pin for default nixpkgs used to build Nixery
This moves the pin from just being in the Travis configuration to also
being set in a nixpkgs-pin.nix file, which makes it trivial to build
at the right commit when performing local builds.
2020-07-25 14:43:21 +01:00
Raphael Borun Das Gupta
bdda24a772 chore(nix): update channel 19.03 -> 20.03
Use a NixOS / NixPkgs release that's actually being supported
and regularly updated.
2020-05-01 11:58:31 +01:00
Florian Klink
215df37187 fix(popcount): Fix nix-build -A nixery-popcount
Previously, this was failing as follows:

```
these derivations will be built:
  /nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv
building '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv'...
building
warning: GOPATH set to GOROOT (/nix/store/4859cp1v7zqcqh43jkqsayl4wrz3g6hp-go-1.13.4/share/go) has no effect
failed to initialize build cache at /homeless-shelter/.cache/go-build: mkdir /homeless-shelter: permission denied
builder for '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed with exit code 1
error: build of '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed
```
2020-02-26 08:31:46 +00:00
Vincent Ambo
1031d890ec fix(builder): Fix minor logging switcharoo 2020-01-19 07:44:36 +00:00
Vincent Ambo
2b82f1b71a refactor: Reshuffle file structure for better code layout
This gets rid of the package called "server" and instead moves
everything into the project root, such that Go actually builds us a
binary called `nixery`.

This is the first step towards factoring out CLI-based functionality
for Nixery.
2019-11-27 14:12:38 +00:00
Vincent Ambo
df88da126a fix(builder): Ensure "solo-metapackages" do not break builds
The previous logic failed because single meta-packages such as
"nixery.dev/shell" would not end up removing the meta-package itself
from the list of packages passed to Nix, causing a build failure.

This was a regression introduced in 827468a.
2019-11-27 13:28:13 +00:00
Vincent Ambo
a924093d09 test(builder): Add test coverage for name->image conversion
Adds tests to cover that packages & metapackages are parsed into image
names correctly.
2019-11-27 13:28:13 +00:00
Vincent Ambo
104c930040 chore(build): Use significantly fewer layers for Nixery itself
Nixery itself is built with the buildLayeredImage system, which takes
some time to create large numbers of layers.

This adjusts the default number of image layers from 96 to 20.

Additionally Nixery's image is often loaded with `docker load -i`,
which ignores layer cache hits anyways.

Additionaly the CI build is configured to use only 1, which speeds up
CI runs.
2019-11-09 15:35:58 +00:00
Vincent Ambo
9a8abeff97 feat(build): Integration test on both CPU architectures 2019-11-09 15:35:58 +00:00
Vincent Ambo
1d6898a7cc feat(build): Include arm64 in build matrix 2019-11-09 15:35:58 +00:00
Vincent Ambo
145b7f4289 fix(build-image): Allow "cross-builds" of images for different arch
Imports the package set twice in the builder expression: Once
configured for the target system, once configured for the native
system.

This makes it possible to fetch the actual image contents for the
required architecture, but use local tools to assemble the symlink
layer and metadata.
2019-11-09 15:35:58 +00:00
Vincent Ambo
d7ccf35149 feat(builder): Support 'arm64' meta-package
Specifying this meta-package toggles support for ARM64 images, for
example:

    # Pull a default x86_64 image
    docker pull nixery.dev/hello

    # Pull an ARM64 image
    docker pull nixery.dev/arm64/hello
2019-11-09 15:35:58 +00:00
Vincent Ambo
3c2de4c037 refactor(builder): Parameterise CPU architecture to use for images
Adds the CPU architecture to the image configuration. This will make
it possible to let users toggle architecture via meta-packages.

Relates to #13
2019-11-09 15:35:58 +00:00
Vincent Ambo
7afbc912ce chore(build): Add nixery-popcount to top-level package set 2019-11-03 01:33:36 +00:00
Vincent Ambo
05b5b1718a feat(popcount): Cache seen narinfos on disk 2019-11-03 01:33:36 +00:00
Vincent Ambo
6a2fb092a7 chore: Add missing copyright headers to source files 2019-11-03 01:33:36 +00:00
Vincent Ambo
b03f7a1b4d feat(popcount): Add new narinfo-based popcount implementation
Adds an implementation of popcount that, instead of realising
derivations locally, just queries the cache's narinfo files.

The downside of this is that calculating popularity for arbitrary Nix
package sets is not possible with this implementation. The upside is
that calculating the popularity for an entire Nix channel can now be
done in ~10 seconds[0].

This fixes #65.

[0]: Assuming a /fast/ internet connection.
2019-11-03 01:33:36 +00:00
Vincent Ambo
2d4a3ea307 chore(server): Remove outdated TODO
Real-life experience has shown that the weighting of the metric
produced here is appropriate.
2019-10-29 19:16:24 +01:00
Vincent Ambo
904c3dade0 refactor(server): Handle non-error errors safely
This case should not be possible unless something manually constructs
a logrus entry with a non-error value in the log.ErrorKey field, but
it's better to be safe than sorry.
2019-10-29 19:16:24 +01:00
Vincent Ambo
3a7c964a22 chore(build): Configure build to use new GCS configuration options 2019-10-28 22:31:44 +01:00
Vincent Ambo
ab190256ab fix(server): Use correct scope for GCS tokens 2019-10-28 22:31:44 +01:00