Nixery is going to gain a new binary (used for building images without
a registry server); to prepare for this the server binary has moved to
cmd/server and the Nix build logic has been updated to wrap this
binary and set the required environment variables.
Change-Id: I9b4f49f47872ae76430463e2fcb8f68114070f72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5603
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Cleans up a whole bunch of things I wanted to get out of the door
right away:
* depot internal references to //third_party/nixery have been replaced
with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks
Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
This does not fully change the build structure of Nixery to be
depot-compatible yet, but should allow most targets to be built in
depot CI.
This contains some hacks to work around surface incompatibilities
which we'll clear away later.
Change-Id: I84e7734334abbe299983956f528c0897f49fa8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5485
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This absorbs a josh-filtered Nix subtree into depot, at
//tools/nixery.
This subtree was created through `josh-filter ':prefix=tools/nixery'`,
which allows a filter on tools/nixery to yield the same commit hashes
as the original Nixery repository (allowing for history continuity).
Change-Id: Icc1a99bf1248226b91f437b0a90361d36fb0d327
After the discussion in #116, this stores the blob content types
in extended attributes when using the filesystem backend.
If the underlying filesystem doesn't support extended attributes,
storing blobs won't work; also, if extended attributes get removed,
blobs won't be served anymore. We can relax this behavior if
needed (i.e. log errors but still accept to store or serve blobs).
However, since the Docker Engine (and possibly other container
engines) won't accept to pull images from a registry that doesn't
use correct content types for manifest files, it could be argued
that it's better to give a hard fail. (Otherwise, the container
engine gives cryptic error messages like "missing signature key".)
I can change that behavior (and log errors but still store/serve
blobs to the filesystem) if you think it's better.
Drops the go2nix configuration in favour of pkgs.buildGoModule.
Note that the go.sum file is bloated by issues with cyclic
dependencies in some Google projects, but this large number of
dependencies is not actually built.
This moves the pin from just being in the Travis configuration to also
being set in a nixpkgs-pin.nix file, which makes it trivial to build
at the right commit when performing local builds.
This gets rid of the package called "server" and instead moves
everything into the project root, such that Go actually builds us a
binary called `nixery`.
This is the first step towards factoring out CLI-based functionality
for Nixery.
Nixery itself is built with the buildLayeredImage system, which takes
some time to create large numbers of layers.
This adjusts the default number of image layers from 96 to 20.
Additionally Nixery's image is often loaded with `docker load -i`,
which ignores layer cache hits anyways.
Additionaly the CI build is configured to use only 1, which speeds up
CI runs.
Some Nix download mechanisms will add a second hash in the store path,
which had been added to the source hash output (breaking argument
interpolation).
Uses a hash of Nixery's sources as the version displayed when Nixery
launches or logs an error. This makes it possible to distinguish
between errors logged from different versions.
The source hashes should be reproducible between different checkouts
of the same source tree.
This introduces a new Nix derivation that, given an attribute set of
layer hashes mapped to store paths, will create a layer tarball for
each of the store paths.
This is going to be used by the builder to create layers that are not
present in the cache.
Relates to #50.
Simplifies the wrapper script used to invoke Nix builds from Nixery to
just contain the essentials, since the layer grouping logic is moving
into the server itself.
Fixes two launch script compatibility issues with other container
runtimes (such as gvisor):
* don't fail if /tmp already exists
* don't fail if the environment becomes unset
Introduces three new types representing each of the possible package
sources and moves the logic for specifying the package source to the
server.
Concrete changes:
* Determining whether a specified git reference is a commit vs. a
branch/tag is now done in the server, and is done more precisely by
using a regular expression.
* Package sources now have a new `CacheKey` function which can be used
to retrieve a key under which a build manifest can be cached *if*
the package source is not a moving target (i.e. a full git commit
hash of either nixpkgs or a private repository).
This function is not yet used.
* Users *must* now specify a package source, Nixery no longer defaults
to anything and will fail to launch if no source is configured.
This package is used by a variety of programs that users may want to
embed into Nixery in addition, for example cachix, but those packages
don't refer to it explicitly.
This is required by git in cases where Nixery is configured with a
custom git repository.
I've also added a shell back into the image to make debugging a
running Nixery easier. It turns out some of the dependencies already
pull in bash anyways, so this is just surfacing it to $PATH.
Some upcoming changes might require the Nix build to be split into
multiple separate nix-build invocations of different expressions, thus
splitting this out is useful.
It also fixes an issue where `build-image/default.nix` might be called
in an environment where no Nix channels are configured.
ALl the ones except for build-image.nix are considered trivial. On the
latter, nixfmt makes some useful changes but by-and-large it is not
ready for that code yet.
Instead of requiring the server component to be made aware of the
location of the Nix builder via environment variables, this commit
introduces a wrapper script for the builder that can simply exist on
the builders $PATH.
This is one step towards a slightly nicer out-of-the-box experience
when using `nix-build -A nixery-bin`.
This uses a significantly larger percentage of the total available
layers (125) than before, which means that cache hits for layers
become more likely between images.
Executes the previously added mdBook on the previously added book
source to yield a directory that can be served by Nixery on its index
page.
This is one of those 'I <3 Nix' things due to how easy it is to do.
Shiny, new domain is much better and eliminates the TLS redirect issue
because there is a HSTS preload for the entire .dev TLD (which, by the
way, is awesome!)
Adds git & SSH as part of the Nixery image, which are required to use
Nix's builtins.fetchGit.
The dependency on interactive tools is dropped, as it was only
required during development when debugging the image building process
itself.
When running on AppEngine, the image is expected to be configured with
a default entry point / command.
This sets the command to the wrapper script, so that the image can
actually run properly when deployed.
When running Nix inside of a container image, there are several
environment-specific details that need to be configured appropriately.
Most importantly, since one of the recent Nix 2.x releases, sandboxing
during builds is enabled by default. This, however, requires kernel
privileges which commonly aren't available to containers.
Nixery's demo instance (for instance, hehe) is deployed on AppEngine
where this type of container configuration is difficult, hence this
change.
Specifically the following were changed:
* additional tools (such as tar/gzip) were introduced into the image
because the builtins-toolset in Nix does not reference these tools
via their store paths, which leads to them not being included
automatically
* Nix sandboxing was disabled in the container image
* the users/groups required by Nix were added to the container setup.
Note that these are being configured manually instead of via the
tools from the 'shadow'-package, because the latter requires some
user information (such as root) to be present already, which is not
the case inside of the container
Introduces a wrapper script which automatically sets the paths to the
required runtime data dependencies.
Additionally configures a container image derivation which will output
a derivation with Nixery, Nix and other dependencies.
Rather than migrating to Bazel, it seems more appropriate to use Nix
for this project.
The project is split into several different components (for data
dependencies and binaries). A derivation for building an image for
Nixery itself will be added.
