Commit graph

245 commits

Author SHA1 Message Date
Vincent Ambo
a412791752 refactor(nixery): expose launch script derivation
Simplifies reusing the launch script in other use-cases than the
"official" Nixery image.

Relates to nixery#166

Change-Id: Iaf1dff385ce270792253551081c1b2fca6400037
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11046
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-28 20:21:04 +00:00
Vincent Ambo
bc06e4d99c fix(nixery): use set-default for setting WEB_DIR
Makes it possible for users to still override this using an envvar.

Relates to nixery#166

Change-Id: Ief2925e03cf2e4351bc38554bf553c8ee259f1f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11045
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-28 20:20:30 +00:00
Markus Rudy
c461595b7f fix(nixery): strictly adhere to OCI image spec
nixery.dev uses the vnd.docker.container.image.v1 format, which is
recognized by the OCI [1] and originally defined by Docker [2]. The
config field in this image format, which this commit is about, is
even portable between the Docker and OCI formats (the Docker Golang
library embeds the OCI definition [3]).

The attribute names in what's called ImageConfig in [3] are specified as
PascalCase, which effectively means that the names Env and Cmd used by
nixery need to be capitalized. The lowercase variant is not causing a
lot of issues because most container tooling is written in Golang, which
allows case-insensitive matches when deserializing JSON. Languages that
parse strictly either miss the configuration values, or fail due to
unknown attributes. This commit capitalizes Cmd and Env to accomodate
strict parsers.

[1]: https://github.com/opencontainers/image-spec/blob/365fa41/media-types.md?plain=1#L70
[2]: https://github.com/moby/moby/blob/v20.10.8/image/spec/v1.2.md#image-json-description
[3]: https://github.com/opencontainers/image-spec/blob/365fa41/specs-go/v1/config.go#L24

Change-Id: Ibee597a64d36c008dea83a3b7a0d8e59b8287d0d
Signed-off-by: Markus Rudy <webmaster@burgerdev.de>
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11012
Autosubmit: lukegb <lukegb@tvl.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2024-02-22 16:56:02 +00:00
Vincent Ambo
67fed3d76d chore(tools/depotfmt): use Go version from buildGo
This is required because Go 1.18 is actually being deleted. I've
applied the formatting breakage that it introduces (such as breaking
comment formatting), because I can't be bothered to try and work
around broken Go stuff.

Change-Id: Ica7cee0d01228845d6a766079fef36df99a3da96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9832
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-30 16:33:21 +00:00
guangwu
3e5279aeff docs(nixery): occurrences typo
Change-Id: I3798e1c23d6b0580b99b14bb4aae1c7cfc81fb6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9366
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-09-19 08:58:13 +00:00
Vincent Ambo
2464ea7303 fix(nixery): allow references to packages starting with numbers
These packages are invalid in Nix, and worked around in nixpkgs with
underscores, but the underscores are invalid in the Docker registry
protocol.

We work around this by detecting this case and adding the underscore
to yield the correct package reference. There is no case where this
workaround can break something, as there can be no valid package
matching the regular expression.

This relates to https://github.com/tazjin/nixery/issues/158

Change-Id: I7990cdb534a8e86c2ceee2c589a2636af70a4a03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8531
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-04-29 11:49:02 +00:00
Vincent Ambo
bdf93dcefe fix(nixery): fix link to nixery logo
Change-Id: Ib78659b971696feaff579bc0a31df7d8ee24e459
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8034
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-06 17:36:54 +00:00
Vincent Ambo
7c99e9e8e3 docs(nixery): replace the Nixery mdBook with a simple web page
Nixery's previous landing page was an mdBook that was basically
unmaintained and full of incorrect information. It also duplicated
some things (like nix-1p) which actually live elsewhere.

This commit removes the mdBook completely and reduces it down to a
simple TVL-style landing page. The landing page has been checked in
in its entirety because Nixery is frequently cloned through josh
without the entirety of depot, however the page has been created by
building it through depot's //web/tvl/template.

See also https://github.com/tazjin/nixery/issues/156

Change-Id: I20e1d58f1e6608377207e80345c169f7d92d3847
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6930
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-10-12 10:58:03 +00:00
Vincent Ambo
c9e4d9c06b chore(nixery): use ldflags parameter instead of buildFlagsArray
The latter has been deprecated in nixpkgs.

Relates to b/200

Change-Id: I42871ce3eb54ebf092909f033b43936b9610d982
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6836
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-02 13:39:28 +00:00
sterni
0c178a0ef6 chore(3p/sources): Bump channels & overlays
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.

* runCommandNoCC -> runCommand
* gmailieer -> lieer
  We also need to work around the fact that home-manager hasn't catched
  on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
  This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy

Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.

Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-09-28 08:02:31 +00:00
talyz
5b165e7318 fix(nixery): Set correct depot ref when fetching nix-1p
Change-Id: Iffa49a4e8fd38d0762ed1f60bf72b9a050594a3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6697
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-19 11:28:10 +00:00
talyz
02b6b6c564 fix(nixery): Discard string context before parsing with fromJSON
Discard string context in prepare-image.nix before parsing input read
with readFile with fromJSON. Required for compatibility with nix >2.3.

Change-Id: I3830707e80fd19a700551a15f1a96d2841d0b022
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6696
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-19 11:28:10 +00:00
talyz
28417afbb4 fix(nixery): Avoid race when the same image is fetched in parallel
Remove a race condition which appears when uploadHashLayer is called
with the same key from multiple threads simultaneously. This can
easily happen when the same image path is requested by multiple
clients at the same time. When it does, a 500 status is returned and
the following error message is logged:

{
  "context": {
    "filePath": "github.com/google/nixery/builder/builder.go",
    "lineNumber": 440,
    "functionName": "github.com/google/nixery/builder.uploadHashLayer"
  },
  "error": "rename /var/lib/nixery/staging/<hash> /var/lib/nixery/layers/<hash>: no such file or directory",
  "eventTime": "...",
  "layer": "<hash>",
  "message": "failed to move layer from staging",
  ...
}

To solve this issue, introduce a mutex keyed on the uploaded hash and
move all layer caching into uploadHashLayer. This could additionally
provide a small performance benefit when an already built image is
requested and NIXERY_PKGS_PATH is set, since symlink layers and config
layers are now also cached.

Change-Id: I50788a7ec7940cb5e5760f244692e361019a9bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6695
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-19 11:28:10 +00:00
Vincent Ambo
544d72189c chore(nixery): use nix-1p from within the depot
Since the source of nix-1p is checked in under //nix/nix-1p, we should
use it from there if Nixery is being built inside of depot.

Change-Id: Iddd54f7b93b398b2f909db6ee105366a9914a2ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5882
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-06-16 15:08:27 +00:00
Vincent Ambo
ac10907913 docs(nixery): dynamically display current nixpkgs commit
People occasionally ask what the current nixpkgs commit is on
nixery.dev (see e.g. https://github.com/tazjin/nixery/issues/153).

With this change, the commit is displayed on nixery.dev if Nixery is
built for the TVL deployment.

Change-Id: I795220214db5a367a126c9b4bd03754e9f144940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5881
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-06-16 15:08:26 +00:00
Vincent Ambo
385591d8bf chore(nixery): Bump Go dependencies
Change-Id: Id6ff48d66368732cba0b8af6e1cbab64b0f2afbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5671
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-26 10:05:55 +00:00
Vincent Ambo
85943eeed4 feat(nixery): Automatically mirror subtree to Github
This exports the `:/tools/nixery` subtree to Github automatically
after merges to `canon`.

Due to the way the project was imported this continues the existing
git history in the external repository.

Change-Id: Ie871c14ad5d8f1019f8be86adecbe9b130ffb01a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5667
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-26 08:41:57 +00:00
Vincent Ambo
f31edeec1b refactor(nixery): Modernise structure of binaries
Nixery is going to gain a new binary (used for building images without
a registry server); to prepare for this the server binary has moved to
cmd/server and the Nix build logic has been updated to wrap this
binary and set the required environment variables.

Change-Id: I9b4f49f47872ae76430463e2fcb8f68114070f72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5603
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-23 15:04:56 +00:00
Vincent Ambo
796ff086be refactor(nixery): Extract layering logic into separate package
This will be required for making a standalone, Nixery-style image
builder function usable from Nix.

Change-Id: I5e36348bd4c32d249d56f6628cd046916691319f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5601
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-23 15:04:56 +00:00
Vincent Ambo
119462d720 fix(nixery): Avoid impure reading of .git directory
Change-Id: I67405f9c9bd9cc8cb34fafff80e30b2fca53a2b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5502
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-04-21 17:48:14 +00:00
Vincent Ambo
6716bf018c chore(nixery): Housekeeping for depot compatibility
Cleans up a whole bunch of things I wanted to get out of the door
right away:

* depot internal references to //third_party/nixery have been replaced
  with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks

Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-04-20 15:31:16 +00:00
Vincent Ambo
70779f4e65 refactor(nixery): Adapt Nix build instructions for readTree
This does not fully change the build structure of Nixery to be
depot-compatible yet, but should allow most targets to be built in
depot CI.

This contains some hacks to work around surface incompatibilities
which we'll clear away later.

Change-Id: I84e7734334abbe299983956f528c0897f49fa8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5485
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-20 14:23:17 +00:00
Vincent Ambo
e459a6cf3b feat(tools/nixery): Absorb Nixery into depot
This absorbs a josh-filtered Nix subtree into depot, at
//tools/nixery.

This subtree was created through `josh-filter ':prefix=tools/nixery'`,
which allows a filter on tools/nixery to yield the same commit hashes
as the original Nixery repository (allowing for history continuity).

Change-Id: Icc1a99bf1248226b91f437b0a90361d36fb0d327
2022-04-20 16:04:17 +02:00
Raphael Borun Das Gupta
3d26ea9e63 docs: change references to repo URL
The Nixery main Git repo has moved
from https://github.com/google/nixery
to https://github.com/tazjin/nixery .

So change it in README and on the https://nixery.dev/ website.
2022-04-20 14:22:09 +02:00
Jérôme Petazzoni
dd7de32c36 feat: set SSL_CERT_FILE and provide a Cmd
Two minor "quality of life" improvements:
- automatically set SSL_CERT_FILE environment variable,
  so that programs relying on OpenSSL for certificate
  validation can actually validate certificates
  (the certificates are included no matter what since
  we add the "cacert" package to all iamges)
- if the requested image includes an interactive shell
  (e.g. if it includes the "shell" metapackage), set
  the image Cmd to "bash", which allows to execute
  "docker run nixery.dev/shell" and get a shell)

I'm happy to split this PR in two if you'd like, but
since both features touch the Config structure and are
rather small, I thought it would make sense to bundle
them together.
2021-12-27 11:26:54 +03:00
Jérôme Petazzoni
7433d620bb feat: add /tmp
Examples of programs that fail when /tmp doesn't exist:
- terraform
- anything using mktemp and similar helpers
2021-12-24 22:19:55 +03:00
Ethan Davidson
15f79e1364 docs: mention arm64 metapackage 2021-12-24 21:42:02 +03:00
Jérôme Petazzoni
aaf5370344 chore: fix env var name in error message
The error message shows the wrong variable name, which might
be confusing for new users.
2021-12-24 20:06:50 +03:00
Jérôme Petazzoni
1dd3421615 docs: update installation instructions
These instructions were not up-to-date (they didn't mention
the different storage backends, and some variables were
tagged as optional while they were mandatory). With this
update, they should (hopefully) be more accurate! :)

I also added instructions if someone wants to run Nixery
outside of the container image (I found it convenient when
working on Nixery's code).
2021-12-24 18:49:35 +03:00
Vincent Ambo
fc62b90514 chore: Bump all Go dependencies
Result of 'go get -u && go mod tidy'
2021-10-29 19:18:27 +02:00
Vincent Ambo
485b8aa929 chore: Bump nixpkgs pin to nixos-unstable 2021-10-29 2021-10-29 18:24:57 +02:00
Vincent Ambo
f4daffbb50 chore(docs): Bump included nix-1p version
... basically never updated this, oops.
2021-10-29 17:41:01 +02:00
Vincent Ambo
9929891f56 docs: Remove note about unsupported Google projects
I no longer work at Google and the repo has moved, so this is no
longer relevant.
2021-10-29 17:32:40 +02:00
Jérôme Petazzoni
dd778e7766 revert: "feat(storage): Add generic support for content-types"
This reverts commit 7db252f36a68d875429a25e06d88fbfc804d84fd.

Superseded by the implementation in #127.
2021-10-08 13:44:02 +03:00
Vincent Ambo
af337010e9 feat(prepare-image): Ensure /usr/bin/env is always present
This is required by common patterns in shell scripts.

There are some caveats around this. Adding logic to filter whether
coreutils is included in an image would slow down the Nix evaluation,
so the link is currently created even in cases where it doesn't point
to anything.

Fixes #109
2021-08-25 16:49:05 +03:00
Vincent Ambo
02455bd0fd chore(build): Allow passing in a specific commit hash when building
Required for builds where the full repository isn't available (e.g.
from a tarball).
2021-08-06 14:27:21 +03:00
Vincent Ambo
84fb380f57 docs: Update build badge in README
Moves the build badge to point at Github Actions, instead of the old (failing) Travis build
2021-07-15 16:23:42 +02:00
Jérôme Petazzoni
94e04a76b6 feat(storage): Store blob content-type in extended attributes
After the discussion in #116, this stores the blob content types
in extended attributes when using the filesystem backend.

If the underlying filesystem doesn't support extended attributes,
storing blobs won't work; also, if extended attributes get removed,
blobs won't be served anymore. We can relax this behavior if
needed (i.e. log errors but still accept to store or serve blobs).
However, since the Docker Engine (and possibly other container
engines) won't accept to pull images from a registry that doesn't
use correct content types for manifest files, it could be argued
that it's better to give a hard fail. (Otherwise, the container
engine gives cryptic error messages like "missing signature key".)

I can change that behavior (and log errors but still store/serve
blobs to the filesystem) if you think it's better.
2021-06-26 01:27:43 +02:00
Florian Klink
3efbbfcd4e feat(ci): don't mount /var/cache/nixery from tmpfs into docker container
With https://github.com/google/nixery/pull/127, nixery will use extended
attributes to store metadata (when using local storage).

Right now, our integration test mounts a tmpfs to /var/cache/nixery.
However, *user* xattrs aren't supported with tmpfs [1], so setting
xattrs would fail.

To workaround this, use a folder in the current working directory and
hope it's backed by something supporting user xattrs (which is the case
for GitHub Actions).

[1]: https://man7.org/linux/man-pages/man5/tmpfs.5.html#NOTES
2021-06-20 18:33:53 +02:00
Vincent Ambo
768f3986a9 feat(build): Run go vet as a step in the GitHub Actions workflow 2021-04-30 13:27:59 +02:00
Vincent Ambo
13d97c9e51 refactor(build): Pin dependencies using Go modules
Drops the go2nix configuration in favour of pkgs.buildGoModule.

Note that the go.sum file is bloated by issues with cyclic
dependencies in some Google projects, but this large number of
dependencies is not actually built.
2021-04-30 13:27:59 +02:00
Vincent Ambo
5c2db7b8ce chore(build): Use current git commit hash as build version 2021-04-30 13:27:59 +02:00
Vincent Ambo
7520f2cb96 chore: Update default NixOS channel to nixos-20.09 2021-04-30 12:28:17 +02:00
Vincent Ambo
8a1add9ef8 chore(ci): Remove unnecessary commands from new CI setup
* remove a step that was not supposed to be committed ("Do we have
  Docker?")
* remove setup of old temporary storage directory (now done in
  integration script test instead)
* skip creation of out-link for initial Nixery build (to avoid
  cache-busting on the second build)
2021-04-30 11:02:38 +02:00
Florian Klink
7e8295189b docs: document unset GOOGLE_APPLICATION_CREDENTIALS
In case the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is not
set, a redirect to storage.googleapis.com is issued, which means the
underlying bucket objects need to be publicly accessible.

This wasn't really obvious until now, so further clarify it.
2021-04-29 23:55:34 +02:00
Florian Klink
970f492235 feat(ci): add integration tests to GitHub Actions, remove .travis.yaml
This copies the integration tests from `.travis.yaml` into a script,
documents the assumptions it makes, and wires it into GitHub Actions.

Contrary to the travis version, we don't use Nixery's GCS backend, as
handing out access to the bucket used, especially for PRs, needs to be
done carefully.

Adding back GCS to the integration test can be done at a later point,
either by using a mock server, or by only exposing the credentials for
master builds (and have the test script decide on whether
GOOGLE_APPLICATION_CREDENTIALS is set or not).

The previous travis version had some complicated post-mortem log
gathering - instead of doing this, we can just `docker run` nixery, but
fork it into the background with the shell - causing it to still be able
to log its output as it's running.

An additional `--rm` is appended, so the container gets cleaned up on
termination - this allows subsequent runs on non-CI infrastructure (like
developer laptops), without having to manually clean up containers.

Fixes #119.
2021-04-29 23:44:42 +02:00
Florian Klink
ee48bd891c feat(ci): remove unneeded permissions: read-all
We don't intend to label, authenticate or whatever with the
GITHUB_TOKEN, so there's not really a reason to give any broader
permissions than the defaults.
2021-04-29 20:23:15 +02:00
Vincent Ambo
d2767bbe8a feat(ci): Configure initial GitHub Actions setup
Travis is being deprecated, and this might be the best option for now.
2021-04-28 22:50:58 +02:00
Jerome Petazzoni
f172107ef1 feat(storage): Add generic support for content-types
When serving a manifest, it is important to set the content-type
correctly (otherwise pulling an image is likely to give a cryptic
error message, "Error response from daemon: missing signature key").

This makes sure that we set the content-type properly for both
manifests and layers.
2021-04-27 15:39:58 +02:00
Jerome Petazzoni
954953d8ba chore(nix): update channel URL
It looks like NixPkgs channels have moved. Fixing this URL allows
using nixos-20.09, for instance.
2021-04-14 14:10:53 +02:00