Commit graph

49 commits

Author SHA1 Message Date
Vincent Ambo
385591d8bf chore(nixery): Bump Go dependencies
Change-Id: Id6ff48d66368732cba0b8af6e1cbab64b0f2afbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5671
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-26 10:05:55 +00:00
Vincent Ambo
85943eeed4 feat(nixery): Automatically mirror subtree to Github
This exports the `:/tools/nixery` subtree to Github automatically
after merges to `canon`.

Due to the way the project was imported this continues the existing
git history in the external repository.

Change-Id: Ie871c14ad5d8f1019f8be86adecbe9b130ffb01a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5667
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-26 08:41:57 +00:00
Vincent Ambo
f31edeec1b refactor(nixery): Modernise structure of binaries
Nixery is going to gain a new binary (used for building images without
a registry server); to prepare for this the server binary has moved to
cmd/server and the Nix build logic has been updated to wrap this
binary and set the required environment variables.

Change-Id: I9b4f49f47872ae76430463e2fcb8f68114070f72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5603
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-23 15:04:56 +00:00
Vincent Ambo
119462d720 fix(nixery): Avoid impure reading of .git directory
Change-Id: I67405f9c9bd9cc8cb34fafff80e30b2fca53a2b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5502
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-04-21 17:48:14 +00:00
Vincent Ambo
6716bf018c chore(nixery): Housekeeping for depot compatibility
Cleans up a whole bunch of things I wanted to get out of the door
right away:

* depot internal references to //third_party/nixery have been replaced
  with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks

Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-04-20 15:31:16 +00:00
Vincent Ambo
70779f4e65 refactor(nixery): Adapt Nix build instructions for readTree
This does not fully change the build structure of Nixery to be
depot-compatible yet, but should allow most targets to be built in
depot CI.

This contains some hacks to work around surface incompatibilities
which we'll clear away later.

Change-Id: I84e7734334abbe299983956f528c0897f49fa8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5485
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-20 14:23:17 +00:00
Vincent Ambo
e459a6cf3b feat(tools/nixery): Absorb Nixery into depot
This absorbs a josh-filtered Nix subtree into depot, at
//tools/nixery.

This subtree was created through `josh-filter ':prefix=tools/nixery'`,
which allows a filter on tools/nixery to yield the same commit hashes
as the original Nixery repository (allowing for history continuity).

Change-Id: Icc1a99bf1248226b91f437b0a90361d36fb0d327
2022-04-20 16:04:17 +02:00
Vincent Ambo
fc62b90514 chore: Bump all Go dependencies
Result of 'go get -u && go mod tidy'
2021-10-29 19:18:27 +02:00
Vincent Ambo
02455bd0fd chore(build): Allow passing in a specific commit hash when building
Required for builds where the full repository isn't available (e.g.
from a tarball).
2021-08-06 14:27:21 +03:00
Jérôme Petazzoni
94e04a76b6 feat(storage): Store blob content-type in extended attributes
After the discussion in #116, this stores the blob content types
in extended attributes when using the filesystem backend.

If the underlying filesystem doesn't support extended attributes,
storing blobs won't work; also, if extended attributes get removed,
blobs won't be served anymore. We can relax this behavior if
needed (i.e. log errors but still accept to store or serve blobs).
However, since the Docker Engine (and possibly other container
engines) won't accept to pull images from a registry that doesn't
use correct content types for manifest files, it could be argued
that it's better to give a hard fail. (Otherwise, the container
engine gives cryptic error messages like "missing signature key".)

I can change that behavior (and log errors but still store/serve
blobs to the filesystem) if you think it's better.
2021-06-26 01:27:43 +02:00
Vincent Ambo
13d97c9e51 refactor(build): Pin dependencies using Go modules
Drops the go2nix configuration in favour of pkgs.buildGoModule.

Note that the go.sum file is bloated by issues with cyclic
dependencies in some Google projects, but this large number of
dependencies is not actually built.
2021-04-30 13:27:59 +02:00
Vincent Ambo
5c2db7b8ce chore(build): Use current git commit hash as build version 2021-04-30 13:27:59 +02:00
Vincent Ambo
b4e0b55e56 chore(build): Change pin for default nixpkgs used to build Nixery
This moves the pin from just being in the Travis configuration to also
being set in a nixpkgs-pin.nix file, which makes it trivial to build
at the right commit when performing local builds.
2020-07-25 14:43:21 +01:00
Vincent Ambo
2b82f1b71a refactor: Reshuffle file structure for better code layout
This gets rid of the package called "server" and instead moves
everything into the project root, such that Go actually builds us a
binary called `nixery`.

This is the first step towards factoring out CLI-based functionality
for Nixery.
2019-11-27 14:12:38 +00:00
Vincent Ambo
104c930040 chore(build): Use significantly fewer layers for Nixery itself
Nixery itself is built with the buildLayeredImage system, which takes
some time to create large numbers of layers.

This adjusts the default number of image layers from 96 to 20.

Additionally Nixery's image is often loaded with `docker load -i`,
which ignores layer cache hits anyways.

Additionaly the CI build is configured to use only 1, which speeds up
CI runs.
2019-11-09 15:35:58 +00:00
Vincent Ambo
7afbc912ce chore(build): Add nixery-popcount to top-level package set 2019-11-03 01:33:36 +00:00
Vincent Ambo
ffe58d6cb5 refactor(build): Do not expose nixery-server attribute
In most cases this is not useful for users without the wrapper script,
so users should always build nixery-bin anyways.
2019-10-28 22:31:44 +01:00
Vincent Ambo
cca835ae37 fix(build): Only take the first matching hash for source hashing
Some Nix download mechanisms will add a second hash in the store path,
which had been added to the source hash output (breaking argument
interpolation).
2019-10-11 14:23:41 +01:00
Vincent Ambo
6912658c72 feat(server): Use hash of Nixery source as version
Uses a hash of Nixery's sources as the version displayed when Nixery
launches or logs an error. This makes it possible to distinguish
between errors logged from different versions.

The source hashes should be reproducible between different checkouts
of the same source tree.
2019-10-06 23:05:23 +01:00
Vincent Ambo
0642f7044d fix(server): Amend package path for Go tooling compatibility
With these changes it is possible to keep Nixery in $GOPATH and build
the server in there, while still having things work correctly via Nix.
2019-10-06 23:05:23 +01:00
Vincent Ambo
0d820423e9 chore(build-image): Remove nixery-build-layers
This functionality has been rolled into the server component and is no
longer required.
2019-10-03 22:29:50 +01:00
Vincent Ambo
ef2623d168 fix(nix): Minor fixes to derivations for new build process 2019-10-03 13:21:04 +01:00
Vincent Ambo
6262dec8aa feat(nix): Add derivation to create layer tars from a store path set
This introduces a new Nix derivation that, given an attribute set of
layer hashes mapped to store paths, will create a layer tarball for
each of the store paths.

This is going to be used by the builder to create layers that are not
present in the cache.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
0898d8a961 chore(build-image): Simplify wrapper build & remove layer grouping
Simplifies the wrapper script used to invoke Nix builds from Nixery to
just contain the essentials, since the layer grouping logic is moving
into the server itself.
2019-10-03 13:21:04 +01:00
Vincent Ambo
ad9b3eb262 refactor(build): Add group-layers to top-level Nix derivations
This makes CI build the group-layers tool (and cache it to Cachix!)
2019-09-22 18:04:40 +01:00
Vincent Ambo
21a17b33f4 fix(build): Ensure launch script compatibility with other runtimes
Fixes two launch script compatibility issues with other container
runtimes (such as gvisor):

* don't fail if /tmp already exists
* don't fail if the environment becomes unset
2019-09-21 15:25:28 +01:00
Vincent Ambo
da6fd1d79e fix(build): Ensure nixery-build-image is on Nixery's PATH
This is useful when running Nixery locally.
2019-09-21 12:44:40 +01:00
Vincent Ambo
980f5e2187 refactor(server): Move package source management logic to server
Introduces three new types representing each of the possible package
sources and moves the logic for specifying the package source to the
server.

Concrete changes:

* Determining whether a specified git reference is a commit vs. a
  branch/tag is now done in the server, and is done more precisely by
  using a regular expression.

* Package sources now have a new `CacheKey` function which can be used
  to retrieve a key under which a build manifest can be cached *if*
  the package source is not a moving target (i.e. a full git commit
  hash of either nixpkgs or a private repository).

  This function is not yet used.

* Users *must* now specify a package source, Nixery no longer defaults
  to anything and will fail to launch if no source is configured.
2019-09-10 11:32:37 +01:00
Vincent Ambo
306e12787a chore(build): Add iana-etc to Nixery's own image
This package is used by a variety of programs that users may want to
embed into Nixery in addition, for example cachix, but those packages
don't refer to it explicitly.
2019-08-21 10:35:32 +01:00
Vincent Ambo
e7d7f73f7d feat(build): Add 'extraPackages' parameter
This makes it possible to inject additional programs (e.g. Cachix)
into a Nixery container.
2019-08-19 01:56:17 +01:00
Vincent Ambo
745b7ce0b8 fix(build): Ensure root user is known inside of container
This is required by git in cases where Nixery is configured with a
custom git repository.

I've also added a shell back into the image to make debugging a
running Nixery easier. It turns out some of the dependencies already
pull in bash anyways, so this is just surfacing it to $PATH.
2019-08-17 10:42:13 +01:00
Vincent Ambo
d9168e3e4d refactor(build-image): Extract package set loading into helper
Some upcoming changes might require the Nix build to be split into
multiple separate nix-build invocations of different expressions, thus
splitting this out is useful.

It also fixes an issue where `build-image/default.nix` might be called
in an environment where no Nix channels are configured.
2019-08-14 00:02:04 +01:00
Vincent Ambo
3939722063 style: Apply nixfmt to trivial Nix files
ALl the ones except for build-image.nix are considered trivial. On the
latter, nixfmt makes some useful changes but by-and-large it is not
ready for that code yet.
2019-08-14 00:02:04 +01:00
Vincent Ambo
7214d0aa4f feat(build-image): Introduce a terrifying hack to build group-layers
The issue is described in detail in a comment in
`build-image/default.nix`, please read it.
2019-08-14 00:02:04 +01:00
Vincent Ambo
6d718bf271 refactor(server): Use wrapper script to avoid path dependency
Instead of requiring the server component to be made aware of the
location of the Nix builder via environment variables, this commit
introduces a wrapper script for the builder that can simply exist on
the builders $PATH.

This is one step towards a slightly nicer out-of-the-box experience
when using `nix-build -A nixery-bin`.
2019-08-14 00:02:04 +01:00
Vincent Ambo
819b460278 chore(docs): Move mdBook derivation to docs/default.nix 2019-08-14 00:02:04 +01:00
Vincent Ambo
1fa93fe6f6 refactor: Move registry server to a subfolder 2019-08-14 00:02:04 +01:00
Vincent Ambo
c727b3ca9e chore(nix): Increase maximum number of layers to 96
This uses a significantly larger percentage of the total available
layers (125) than before, which means that cache hits for layers
become more likely between images.
2019-08-08 21:15:50 +01:00
Vincent Ambo
2bef0ba240 feat(build): Build Nixery book and embed it into Nixery image
Executes the previously added mdBook on the previously added book
source to yield a directory that can be served by Nixery on its index
page.

This is one of those 'I <3 Nix' things due to how easy it is to do.
2019-08-05 00:32:53 +01:00
Vincent Ambo
85e8d760fc feat(build): Add mdBook 0.3.1 to build environment
Upstream nixpkgs currently only has an older versin of mdBook. Until
that changes, we keep a different version in here.
2019-08-05 00:32:53 +01:00
Vincent Ambo
a0d7d693d3 feat(build): Support additional pre-launch commands in image
This makes it possible for users to hook basically arbitrary things
into the Nixery container image.
2019-08-04 01:41:35 +01:00
Vincent Ambo
da5df525c8 docs: Update all nixery.appspot.com references to nixery.dev
Shiny, new domain is much better and eliminates the TLS redirect issue
because there is a HSTS preload for the entire .dev TLD (which, by the
way, is awesome!)
2019-08-02 17:14:06 +01:00
Vincent Ambo
02dfff393a fix(build): coreutils are still required by launch script
Mea culpa!
2019-08-02 01:33:39 +01:00
Vincent Ambo
a83701a14b feat(build): Add dependencies for custom repo clones
Adds git & SSH as part of the Nixery image, which are required to use
Nix's builtins.fetchGit.

The dependency on interactive tools is dropped, as it was only
required during development when debugging the image building process
itself.
2019-07-31 15:28:36 +01:00
Vincent Ambo
6207782434 fix(build): Specify default command for Nixery's own image
When running on AppEngine, the image is expected to be configured with
a default entry point / command.

This sets the command to the wrapper script, so that the image can
actually run properly when deployed.
2019-07-26 10:22:31 +00:00
Vincent Ambo
948f308025 feat(build): Configure Nixery image builder to set up env correctly
When running Nix inside of a container image, there are several
environment-specific details that need to be configured appropriately.

Most importantly, since one of the recent Nix 2.x releases, sandboxing
during builds is enabled by default. This, however, requires kernel
privileges which commonly aren't available to containers.

Nixery's demo instance (for instance, hehe) is deployed on AppEngine
where this type of container configuration is difficult, hence this
change.

Specifically the following were changed:

* additional tools (such as tar/gzip) were introduced into the image
  because the builtins-toolset in Nix does not reference these tools
  via their store paths, which leads to them not being included
  automatically
* Nix sandboxing was disabled in the container image
* the users/groups required by Nix were added to the container setup.
  Note that these are being configured manually instead of via the
  tools from the 'shadow'-package, because the latter requires some
  user information (such as root) to be present already, which is not
  the case inside of the container
2019-07-24 17:46:39 +00:00
Vincent Ambo
23260e59d9 chore: Add license scaffolding & contribution guidelines 2019-07-23 23:32:56 +01:00
Vincent Ambo
5f471392cf feat(build): Add wrapper script & container image setup
Introduces a wrapper script which automatically sets the paths to the
required runtime data dependencies.

Additionally configures a container image derivation which will output
a derivation with Nixery, Nix and other dependencies.
2019-07-23 23:23:21 +01:00
Vincent Ambo
5d9b32977d feat(build): Introduce build configuration using Nix
Rather than migrating to Bazel, it seems more appropriate to use Nix
for this project.

The project is split into several different components (for data
dependencies and binaries). A derivation for building an image for
Nixery itself will be added.
2019-07-23 21:48:27 +01:00