mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore(nixery): Housekeeping for depot compatibility

Browse source 
Cleans up a whole bunch of things I wanted to get out of the door
right away:

* depot internal references to //third_party/nixery have been replaced
  with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks

Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
This commit is contained in:
Vincent Ambo 2022-04-20 16:41:20 +02:00 committed by clbot
parent 535ad8732a
commit 6716bf018c
32 changed files with 192 additions and 478 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
README.md
View file

@ -49,6 +49,9 @@ configuration is tracked in `//ops/{modules,machines}`.
* [`//nix/readTree`](https://cs.tvl.fyi/depot/-/blob/nix/readTree/README.md) * [`//nix/readTree`](https://cs.tvl.fyi/depot/-/blob/nix/readTree/README.md)
contains the Nix code which automatically registers projects in our Nix contains the Nix code which automatically registers projects in our Nix
attribute hierarchy based on their in-tree location attribute hierarchy based on their in-tree location
* [`//tools/nixery`](https://cs.tvl.fyi/depot/-/tree/tools/nixery)
contains the source code of [Nixery][], a container registry that
can build images ad-hoc from Nix packages
* `//nix/yants` contains **Y**et **A**nother **N**ix **T**ype **S**ystem, which * `//nix/yants` contains **Y**et **A**nother **N**ix **T**ype **S**ystem, which
we use for a variety of things throughout the repository we use for a variety of things throughout the repository
* `//nix/buildGo` implements a Nix library that can build Go software in the * `//nix/buildGo` implements a Nix library that can build Go software in the
@ -119,3 +122,4 @@ Hackint also provide a [web chat][tvl-webchat].
[hackint-xmpp]: https://hackint.org/transport/xmpp [hackint-xmpp]: https://hackint.org/transport/xmpp
[tvl-xmpp]: xmpp:#tvl@irc.hackint.org?join [tvl-xmpp]: xmpp:#tvl@irc.hackint.org?join
[tvl-webchat]: https://webirc.hackint.org/#ircs://irc.hackint.org/#tvl [tvl-webchat]: https://webirc.hackint.org/#ircs://irc.hackint.org/#tvl
[Nixery]: https://nixery.dev

2
ops/modules/nixery.nix
View file

@ -28,7 +28,7 @@ in
StateDirectory = "nixery"; StateDirectory = "nixery";
Restart = "always"; Restart = "always";
ExecStartPre = "${pkgs.coreutils}/bin/mkdir -p ${storagePath}"; ExecStartPre = "${pkgs.coreutils}/bin/mkdir -p ${storagePath}";
ExecStart = "${depot.third_party.nixery.nixery-bin}/bin/nixery"; ExecStart = "${depot.tools.nixery.nixery-bin}/bin/nixery";
}; };
environment = { environment = {

19
third_party/nixery/default.nix vendored
View file

@ -1,19 +0,0 @@
# Import the Nixery repository as-is, but pass our own package set
# instead of the pin it has.
{ depot, pkgs, ... }:
let
inherit (depot.nix.readTree) drvTargets;
commit = "601cd998077f77f257ad1a40fa488add8464650f";
src = pkgs.fetchFromGitHub {
owner = "google";
repo = "nixery";
rev = commit;
sha256 = "195rz25y3hfxcmniysajzjg7g69qhz7w06lql8fn0dbcdcxsq6g4";
};
in
drvTargets (import src {
inherit pkgs;
commitHash = _: commit;
})

3
tools/depotfmt.nix
View file

@ -24,8 +24,7 @@ let
includes = [ "*.nix" ] includes = [ "*.nix" ]
excludes = [ excludes = [
"third_party/nix/tests/*", "third_party/nix/tests/*",
"third_party/nix/src/tests/*", "third_party/nix/src/tests/*"
"tools/nixery/*"
] ]
[formatter.rust] [formatter.rust]

2
tools/nixery/.gitattributes vendored
View file

@ -1,2 +0,0 @@
# Ignore stylesheet modifications for the book in Linguist stats
*.css linguist-detectable=false

27
tools/nixery/.github/workflows/build-and-test.yaml vendored
View file

@ -1,27 +0,0 @@
# Build Nixery, spin up an instance and pull an image from it.
name: "Build and test Nixery"
on:
push:
branches:
- master
pull_request: {}
env:
NIX_PATH: "nixpkgs=https://github.com/NixOS/nixpkgs/archive/4263ba5e133cc3fc699c1152ab5ee46ef668e675.tar.gz"
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Install Nix
uses: cachix/install-nix-action@v13
- name: Checkout
uses: actions/checkout@v2.3.4
- name: Prepare environment
run: nix-env -f '<nixpkgs>' -iA go
- name: Check formatting
run: "test -z $(gofmt -l .)"
- name: Run `go vet`
run: "go vet ./..."
- name: Build Nixery
run: "nix-build --no-out-link"
- name: Run integration test
run: scripts/integration-test.sh

35
tools/nixery/CONTRIBUTING.md
View file

@ -1,35 +0,0 @@
# How to Contribute
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
## Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution;
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to <https://cla.developers.google.com/> to see
your current agreements on file or to sign a new one.
You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Commit messages
Commits in this repository follow the [Angular commit message
guidelines][commits].
## Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Community Guidelines
This project follows [Google's Open Source Community
Guidelines](https://opensource.google.com/conduct/).
[commits]: https://github.com/angular/angular/blob/master/CONTRIBUTING.md#commit

14
tools/nixery/README.md
View file

@ -4,7 +4,7 @@
----------------- -----------------
[![Build Status](https://github.com/tazjin/nixery/actions/workflows/build-and-test.yaml/badge.svg)](https://github.com/tazjin/nixery/actions/workflows/build-and-test.yaml) [![Build status](https://badge.buildkite.com/016bff4b8ae2704a3bbbb0a250784e6692007c582983b6dea7.svg?branch=refs/heads/canon)](https://buildkite.com/tvl/depot)
**Nixery** is a Docker-compatible container registry that is capable of **Nixery** is a Docker-compatible container registry that is capable of
transparently building and serving container images using [Nix][]. transparently building and serving container images using [Nix][].
@ -24,6 +24,15 @@ You can watch the NixCon 2019 [talk about
Nixery](https://www.youtube.com/watch?v=pOI9H4oeXqA) for more information about Nixery](https://www.youtube.com/watch?v=pOI9H4oeXqA) for more information about
the project and its use-cases. the project and its use-cases.
The canonical location of the Nixery source code is
[`//tools/nixery`][depot-link] in the [TVL](https://tvl.fyi)
monorepository. If cloning the entire repository is not desirable, the
Nixery subtree can be cloned like this:
git clone https://code.tvl.fyi/depot.git:/tools/nixery.git
The subtree is infrequently mirrored to `tazjin/nixery` on Github.
## Demo ## Demo
Click the image to see an example in which an image containing an interactive Click the image to see an example in which an image containing an interactive
@ -139,8 +148,9 @@ separate Nix function, which will make it possible to build images directly in
Nix builds. Nix builds.
[Nix]: https://nixos.org/ [Nix]: https://nixos.org/
[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html [layering strategy]: https://tazj.in/blog/nixery-layers
[gist]: https://gist.github.com/tazjin/08f3d37073b3590aacac424303e6f745 [gist]: https://gist.github.com/tazjin/08f3d37073b3590aacac424303e6f745
[buildLayeredImage]: https://grahamc.com/blog/nix-and-layered-docker-images [buildLayeredImage]: https://grahamc.com/blog/nix-and-layered-docker-images
[public]: https://nixery.dev [public]: https://nixery.dev
[depot-link]: https://cs.tvl.fyi/depot/-/tree/tools/nixery
[gcs]: https://cloud.google.com/storage/ [gcs]: https://cloud.google.com/storage/

15
tools/nixery/builder/archive.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
package builder package builder
// This file implements logic for walking through a directory and creating a // This file implements logic for walking through a directory and creating a

15
tools/nixery/builder/builder.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Package builder implements the logic for assembling container // Package builder implements the logic for assembling container
// images. It shells out to Nix to retrieve all required Nix-packages // images. It shells out to Nix to retrieve all required Nix-packages

15
tools/nixery/builder/builder_test.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
package builder package builder
import ( import (

15
tools/nixery/builder/cache.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
package builder package builder
import ( import (

15
tools/nixery/builder/layers.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// This package reads an export reference graph (i.e. a graph representing the // This package reads an export reference graph (i.e. a graph representing the
// runtime dependencies of a set of derivations) created by Nix and groups it in // runtime dependencies of a set of derivations) created by Nix and groups it in

15
tools/nixery/config/config.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Package config implements structures to store Nixery's configuration at // Package config implements structures to store Nixery's configuration at
// runtime as well as the logic for instantiating this configuration from the // runtime as well as the logic for instantiating this configuration from the

15
tools/nixery/config/pkgsource.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
package config package config
import ( import (

123
tools/nixery/default.nix
View file

@ -1,16 +1,5 @@
# Copyright 2019-2021 Google LLC # Copyright 2022 The TVL Contributors
# # SPDX-License-Identifier: Apache-2.0
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This function header aims to provide compatibility between builds of # This function header aims to provide compatibility between builds of
# Nixery taking place inside/outside of the TVL depot. # Nixery taking place inside/outside of the TVL depot.
@ -19,12 +8,13 @@
# build system and this will need some major adaptations to support # build system and this will need some major adaptations to support
# that. # that.
{ depot ? { nix.readTree.drvTargets = x: x; } { depot ? { nix.readTree.drvTargets = x: x; }
, pkgs ? import <nixpkgs> {} , pkgs ? import <nixpkgs> { }
, preLaunch ? "" , preLaunch ? ""
, extraPackages ? [] , extraPackages ? [ ]
, maxLayers ? 20 , maxLayers ? 20
, commitHash ? null , commitHash ? null
, ... }@args: , ...
}@args:
with pkgs; with pkgs;
@ -54,7 +44,8 @@ let
"-ldflags=-s -w -X main.version=${nixery-commit-hash}" "-ldflags=-s -w -X main.version=${nixery-commit-hash}"
]; ];
}; };
in depot.nix.readTree.drvTargets rec { in
depot.nix.readTree.drvTargets rec {
# Implementation of the Nix image building logic # Implementation of the Nix image building logic
nixery-prepare-image = import ./prepare-image { inherit pkgs; }; nixery-prepare-image = import ./prepare-image { inherit pkgs; };
@ -79,55 +70,57 @@ in depot.nix.readTree.drvTargets rec {
# Container image containing Nixery and Nix itself. This image can # Container image containing Nixery and Nix itself. This image can
# be run on Kubernetes, published on AppEngine or whatever else is # be run on Kubernetes, published on AppEngine or whatever else is
# desired. # desired.
nixery-image = let nixery-image =
# Wrapper script for the wrapper script (meta!) which configures let
# the container environment appropriately. # Wrapper script for the wrapper script (meta!) which configures
# # the container environment appropriately.
# Most importantly, sandboxing is disabled to avoid privilege
# issues in containers.
nixery-launch-script = writeShellScriptBin "nixery" ''
set -e
export PATH=${coreutils}/bin:$PATH
export NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
mkdir -p /tmp
# Create the build user/group required by Nix
echo 'nixbld:x:30000:nixbld' >> /etc/group
echo 'nixbld:x:30000:30000:nixbld:/tmp:/bin/bash' >> /etc/passwd
echo 'root:x:0:0:root:/root:/bin/bash' >> /etc/passwd
echo 'root:x:0:' >> /etc/group
# Disable sandboxing to avoid running into privilege issues
mkdir -p /etc/nix
echo 'sandbox = false' >> /etc/nix/nix.conf
# In some cases users building their own image might want to
# customise something on the inside (e.g. set up an environment
# for keys or whatever).
# #
# This can be achieved by setting a 'preLaunch' script. # Most importantly, sandboxing is disabled to avoid privilege
${preLaunch} # issues in containers.
nixery-launch-script = writeShellScriptBin "nixery" ''
set -e
export PATH=${coreutils}/bin:$PATH
export NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
mkdir -p /tmp
exec ${nixery-bin}/bin/nixery # Create the build user/group required by Nix
''; echo 'nixbld:x:30000:nixbld' >> /etc/group
in dockerTools.buildLayeredImage { echo 'nixbld:x:30000:30000:nixbld:/tmp:/bin/bash' >> /etc/passwd
name = "nixery"; echo 'root:x:0:0:root:/root:/bin/bash' >> /etc/passwd
config.Cmd = [ "${nixery-launch-script}/bin/nixery" ]; echo 'root:x:0:' >> /etc/group
inherit maxLayers; # Disable sandboxing to avoid running into privilege issues
contents = [ mkdir -p /etc/nix
bashInteractive echo 'sandbox = false' >> /etc/nix/nix.conf
cacert
coreutils # In some cases users building their own image might want to
git # customise something on the inside (e.g. set up an environment
gnutar # for keys or whatever).
gzip #
iana-etc # This can be achieved by setting a 'preLaunch' script.
nix ${preLaunch}
nixery-prepare-image
nixery-launch-script exec ${nixery-bin}/bin/nixery
openssh '';
zlib in
] ++ extraPackages; dockerTools.buildLayeredImage {
}; name = "nixery";
config.Cmd = [ "${nixery-launch-script}/bin/nixery" ];
inherit maxLayers;
contents = [
bashInteractive
cacert
coreutils
git
gnutar
gzip
iana-etc
nix
nixery-prepare-image
nixery-launch-script
openssh
zlib
] ++ extraPackages;
};
} }

18
tools/nixery/docs/default.nix
View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC # Copyright 2022 The TVL Contributors
# # SPDX-License-Identifier: Apache-2.0
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Builds the documentation page using the Rust project's 'mdBook' # Builds the documentation page using the Rust project's 'mdBook'
# tool. # tool.
@ -27,7 +16,8 @@ let
rev = "9f0baf5e270128d9101ba4446cf6844889e399a2"; rev = "9f0baf5e270128d9101ba4446cf6844889e399a2";
sha256 = "1pf9i90gn98vz67h296w5lnwhssk62dc6pij983dff42dbci7lhj"; sha256 = "1pf9i90gn98vz67h296w5lnwhssk62dc6pij983dff42dbci7lhj";
}; };
in runCommand "nixery-book" { } '' in
runCommand "nixery-book" { } ''
mkdir -p $out mkdir -p $out
cp -r ${./.}/* . cp -r ${./.}/* .
chmod -R a+w src chmod -R a+w src

6
tools/nixery/docs/src/nixery.md
View file

@ -68,10 +68,6 @@ production project we recommend setting up a private instance. The public Nixery
at `nixery.dev` is run on a best-effort basis and we make no guarantees about at `nixery.dev` is run on a best-effort basis and we make no guarantees about
availability. availability.
### Is this an official Google project?
**No.** Nixery is not officially supported by Google.
### Who made this? ### Who made this?
Nixery was written by [tazjin][], but many people have contributed to Nix over Nixery was written by [tazjin][], but many people have contributed to Nix over
@ -81,4 +77,4 @@ time, maybe you could become one of them?
[Nix]: https://nixos.org/nix [Nix]: https://nixos.org/nix
[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html [layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
[layers]: https://grahamc.com/blog/nix-and-layered-docker-images [layers]: https://grahamc.com/blog/nix-and-layered-docker-images
[tazjin]: https://github.com/tazjin [tazjin]: https://tazj.in

17
tools/nixery/docs/src/run-your-own.md
View file

@ -65,13 +65,17 @@ use it with your own packages. There are three options available:
### 2.1. With a container image ### 2.1. With a container image
The easiest way to run Nixery is to build a container image. The easiest way to run Nixery is to build a container image. This
This section assumes that the container runtime used is Docker, section assumes that the container runtime used is Docker, please
please modify instructions accordingly if modify instructions accordingly if you are using something else.
you are using something else.
With a working Nix installation, building Nixery is done by invoking `nix-build With a working Nix installation, you can clone and build the Nixery
-A nixery-image` from a checkout of the [Nixery repository][repo]. image like this:
```
git clone https://code.tvl.fyi/depot.git:/tools/nixery.git
nix-build -A nixery-image
```
This will create a `result`-symlink which points to a tarball containing the This will create a `result`-symlink which points to a tarball containing the
image. In Docker, this tarball can be loaded by using `docker load -i result`. image. In Docker, this tarball can be loaded by using `docker load -i result`.
@ -184,7 +188,6 @@ If the directory doesn't exist, Nixery will run fine but serve 404.
[nixery#4]: https://github.com/tazjin/nixery/issues/4 [nixery#4]: https://github.com/tazjin/nixery/issues/4
[Nix]: https://nixos.org/nix [Nix]: https://nixos.org/nix
[gcs]: https://cloud.google.com/storage/ [gcs]: https://cloud.google.com/storage/
[repo]: https://github.com/tazjin/nixery
[signed-urls]: under-the-hood.html#5-image-layers-are-requested [signed-urls]: under-the-hood.html#5-image-layers-are-requested
[ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically [ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
[nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html [nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html

15
tools/nixery/logs/logs.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
package logs package logs
// This file configures different log formatters via logrus. The // This file configures different log formatters via logrus. The

15
tools/nixery/main.go
View file

@ -1,16 +1,5 @@
// Copyright 2019-2020 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// The nixery server implements a container registry that transparently builds // The nixery server implements a container registry that transparently builds
// container images based on Nix derivations. // container images based on Nix derivations.

15
tools/nixery/manifest/manifest.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Package image implements logic for creating the image metadata // Package image implements logic for creating the image metadata
// (such as the image manifest and configuration). // (such as the image manifest and configuration).

15
tools/nixery/popcount/default.nix
View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC # Copyright 2022 The TVL Contributors
# # SPDX-License-Identifier: Apache-2.0
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{ buildGoPackage }: { buildGoPackage }:

15
tools/nixery/popcount/popcount.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Popcount fetches popularity information for each store path in a // Popcount fetches popularity information for each store path in a
// given Nix channel from the upstream binary cache. // given Nix channel from the upstream binary cache.

17
tools/nixery/prepare-image/default.nix
View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC # Copyright 2022 The TVL Contributors
# # SPDX-License-Identifier: Apache-2.0
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file builds a wrapper script called by Nixery to ask for the # This file builds a wrapper script called by Nixery to ask for the
# content information for a given image. # content information for a given image.
@ -18,7 +7,7 @@
# The purpose of using a wrapper script is to ensure that the paths to # The purpose of using a wrapper script is to ensure that the paths to
# all required Nix files are set correctly at runtime. # all required Nix files are set correctly at runtime.
{ pkgs ? import <nixpkgs> {} }: { pkgs ? import <nixpkgs> { } }:
pkgs.writeShellScriptBin "nixery-prepare-image" '' pkgs.writeShellScriptBin "nixery-prepare-image" ''
exec ${pkgs.nix}/bin/nix-build \ exec ${pkgs.nix}/bin/nix-build \

21
tools/nixery/prepare-image/load-pkgs.nix
View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC # Copyright 2022 The TVL Contributors
# # SPDX-License-Identifier: Apache-2.0
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Load a Nix package set from one of the supported source types # Load a Nix package set from one of the supported source types
# (nixpkgs, git, path). # (nixpkgs, git, path).
@ -24,7 +13,8 @@ let
let let
url = url =
"https://github.com/NixOS/nixpkgs/archive/${channel}.tar.gz"; "https://github.com/NixOS/nixpkgs/archive/${channel}.tar.gz";
in import (fetchTarball url) importArgs; in
import (fetchTarball url) importArgs;
# If a git repository is requested, it is retrieved via # If a git repository is requested, it is retrieved via
# builtins.fetchGit which defaults to the git configuration of the # builtins.fetchGit which defaults to the git configuration of the
@ -35,7 +25,8 @@ let
# No special handling is used for paths, so users are expected to pass one # No special handling is used for paths, so users are expected to pass one
# that will work natively with Nix. # that will work natively with Nix.
importPath = path: import (toPath path) importArgs; importPath = path: import (toPath path) importArgs;
in if srcType == "nixpkgs" then in
if srcType == "nixpkgs" then
fetchImportChannel srcArgs fetchImportChannel srcArgs
else if srcType == "git" then else if srcType == "git" then
fetchImportGit (fromJSON srcArgs) fetchImportGit (fromJSON srcArgs)

116
tools/nixery/prepare-image/prepare-image.nix
View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC # Copyright 2022 The TVL Contributors
# # SPDX-License-Identifier: Apache-2.0
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This file contains a derivation that outputs structured information # This file contains a derivation that outputs structured information
# about the runtime dependencies of an image with a given set of # about the runtime dependencies of an image with a given set of
@ -23,13 +12,13 @@
{ {
# Description of the package set to be used (will be loaded by load-pkgs.nix) # Description of the package set to be used (will be loaded by load-pkgs.nix)
srcType ? "nixpkgs", srcType ? "nixpkgs"
srcArgs ? "nixos-20.09", , srcArgs ? "nixos-20.09"
system ? "x86_64-linux", , system ? "x86_64-linux"
importArgs ? { }, , importArgs ? { }
# Path to load-pkgs.nix , # Path to load-pkgs.nix
loadPkgs ? ./load-pkgs.nix, loadPkgs ? ./load-pkgs.nix
# Packages to install by name (which must refer to top-level attributes of , # Packages to install by name (which must refer to top-level attributes of
# nixpkgs). This is passed in as a JSON-array in string form. # nixpkgs). This is passed in as a JSON-array in string form.
packages ? "[]" packages ? "[]"
}: }:
@ -77,24 +66,28 @@ let
# `deepFetch haskellpackages.stylish-haskell` retrieves # `deepFetch haskellpackages.stylish-haskell` retrieves
# `haskellPackages.stylish-haskell`. # `haskellPackages.stylish-haskell`.
deepFetch = with lib; s: n: deepFetch = with lib; s: n:
let path = splitString "." n; let
err = { error = "not_found"; pkg = n; }; path = splitString "." n;
# The most efficient way I've found to do a lookup against err = { error = "not_found"; pkg = n; };
# case-differing versions of an attribute is to first construct a # The most efficient way I've found to do a lookup against
# mapping of all lowercased attribute names to their differently cased # case-differing versions of an attribute is to first construct a
# equivalents. # mapping of all lowercased attribute names to their differently cased
# # equivalents.
# This map is then used for a second lookup if the top-level #
# (case-sensitive) one does not yield a result. # This map is then used for a second lookup if the top-level
hasUpper = str: (match ".*[A-Z].*" str) != null; # (case-sensitive) one does not yield a result.
allUpperKeys = filter hasUpper (attrNames s); hasUpper = str: (match ".*[A-Z].*" str) != null;
lowercased = listToAttrs (map (k: { allUpperKeys = filter hasUpper (attrNames s);
lowercased = listToAttrs (map
(k: {
name = toLower k; name = toLower k;
value = k; value = k;
}) allUpperKeys); })
caseAmendedPath = map (v: if hasAttr v lowercased then lowercased."${v}" else v) path; allUpperKeys);
fetchLower = attrByPath caseAmendedPath err s; caseAmendedPath = map (v: if hasAttr v lowercased then lowercased."${v}" else v) path;
in attrByPath path fetchLower s; fetchLower = attrByPath caseAmendedPath err s;
in
attrByPath path fetchLower s;
# allContents contains all packages successfully retrieved by name # allContents contains all packages successfully retrieved by name
# from the package set, as well as any errors encountered while # from the package set, as well as any errors encountered while
@ -105,27 +98,30 @@ let
# Folds over the results of 'deepFetch' on all requested packages to # Folds over the results of 'deepFetch' on all requested packages to
# separate them into errors and content. This allows the program to # separate them into errors and content. This allows the program to
# terminate early and return only the errors if any are encountered. # terminate early and return only the errors if any are encountered.
let splitter = attrs: res: let
if hasAttr "error" res splitter = attrs: res:
then attrs // { errors = attrs.errors ++ [ res ]; } if hasAttr "error" res
else attrs // { contents = attrs.contents ++ [ res ]; }; then attrs // { errors = attrs.errors ++ [ res ]; }
init = { contents = []; errors = []; }; else attrs // { contents = attrs.contents ++ [ res ]; };
fetched = (map (deepFetch pkgs) (fromJSON packages)); init = { contents = [ ]; errors = [ ]; };
in foldl' splitter init fetched; fetched = (map (deepFetch pkgs) (fromJSON packages));
in
foldl' splitter init fetched;
# Contains the export references graph of all retrieved packages, # Contains the export references graph of all retrieved packages,
# which has information about all runtime dependencies of the image. # which has information about all runtime dependencies of the image.
# #
# This is used by Nixery to group closures into image layers. # This is used by Nixery to group closures into image layers.
runtimeGraph = runCommand "runtime-graph.json" { runtimeGraph = runCommand "runtime-graph.json"
__structuredAttrs = true; {
exportReferencesGraph.graph = allContents.contents; __structuredAttrs = true;
PATH = "${coreutils}/bin"; exportReferencesGraph.graph = allContents.contents;
builder = toFile "builder" '' PATH = "${coreutils}/bin";
. .attrs.sh builder = toFile "builder" ''
cp .attrs.json ''${outputs[out]} . .attrs.sh
''; cp .attrs.json ''${outputs[out]}
} ""; '';
} "";
# Create a symlink forest into all top-level store paths of the # Create a symlink forest into all top-level store paths of the
# image contents. # image contents.
@ -151,7 +147,7 @@ let
# Image layer that contains the symlink forest created above. This # Image layer that contains the symlink forest created above. This
# must be included in the image to ensure that the filesystem has a # must be included in the image to ensure that the filesystem has a
# useful layout at runtime. # useful layout at runtime.
symlinkLayer = runCommand "symlink-layer.tar" {} '' symlinkLayer = runCommand "symlink-layer.tar" { } ''
cp -r ${contentsEnv}/ ./layer cp -r ${contentsEnv}/ ./layer
tar --transform='s|^\./||' -C layer --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=0 --group=0 -cf $out . tar --transform='s|^\./||' -C layer --sort=name --mtime="@$SOURCE_DATE_EPOCH" --owner=0 --group=0 -cf $out .
''; '';
@ -159,9 +155,10 @@ let
# Metadata about the symlink layer which is required for serving it. # Metadata about the symlink layer which is required for serving it.
# Two different hashes are computed for different usages (inclusion # Two different hashes are computed for different usages (inclusion
# in manifest vs. content-checking in the layer cache). # in manifest vs. content-checking in the layer cache).
symlinkLayerMeta = fromJSON (readFile (runCommand "symlink-layer-meta.json" { symlinkLayerMeta = fromJSON (readFile (runCommand "symlink-layer-meta.json"
buildInputs = [ coreutils jq openssl ]; {
}'' buildInputs = [ coreutils jq openssl ];
} ''
tarHash=$(sha256sum ${symlinkLayer} | cut -d ' ' -f1) tarHash=$(sha256sum ${symlinkLayer} | cut -d ' ' -f1)
layerSize=$(stat --printf '%s' ${symlinkLayer}) layerSize=$(stat --printf '%s' ${symlinkLayer})
@ -181,7 +178,8 @@ let
error = "not_found"; error = "not_found";
pkgs = map (err: err.pkg) allContents.errors; pkgs = map (err: err.pkg) allContents.errors;
}; };
in writeText "build-output.json" (if (length allContents.errors) == 0 in
then toJSON buildOutput writeText "build-output.json" (if (length allContents.errors) == 0
else toJSON errorOutput then toJSON buildOutput
else toJSON errorOutput
) )

15
tools/nixery/shell.nix
View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC # Copyright 2022 The TVL Contributors
# # SPDX-License-Identifier: Apache-2.0
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configures a shell environment that builds required local packages to # Configures a shell environment that builds required local packages to
# run Nixery. # run Nixery.

15
tools/nixery/storage/filesystem.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Filesystem storage backend for Nixery. // Filesystem storage backend for Nixery.
package storage package storage

15
tools/nixery/storage/gcs.go
View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Google Cloud Storage backend for Nixery. // Google Cloud Storage backend for Nixery.
package storage package storage

15
tools/nixery/storage/storage.go
View file

@ -1,16 +1,5 @@
// Copyright 2019-2020 Google LLC // Copyright 2022 The TVL Contributors
// // SPDX-License-Identifier: Apache-2.0
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Package storage implements an interface that can be implemented by // Package storage implements an interface that can be implemented by
// storage backends, such as Google Cloud Storage or the local // storage backends, such as Google Cloud Storage or the local

6
users/tazjin/blog/posts/nixery-layers.md
View file

@ -260,13 +260,13 @@ TIP: This is implemented in [popcount][] in Nixery.
Hopefully this detailed design review was useful to you. You can also watch [my Hopefully this detailed design review was useful to you. You can also watch [my
NixCon talk][talk] about Nixery for a review of some of this, and some demos. NixCon talk][talk] about Nixery for a review of some of this, and some demos.
[Nixery]: https://github.com/google/nixery [Nixery]: https://cs.tvl.fyi/depot/-/tree/tools/nixery
[grhmc]: https://grahamc.com/blog/nix-and-layered-docker-images [grhmc]: https://grahamc.com/blog/nix-and-layered-docker-images
[Nix]: https://nixos.org/nix [Nix]: https://nixos.org/nix
[registry protocols]: https://github.com/opencontainers/distribution-spec/blob/master/spec.md [registry protocols]: https://github.com/opencontainers/distribution-spec/blob/master/spec.md
[nixery.dev]: https://nixery.dev [nixery.dev]: https://nixery.dev
[dominator trees]: https://en.wikipedia.org/wiki/Dominator_(graph_theory) [dominator trees]: https://en.wikipedia.org/wiki/Dominator_(graph_theory)
[gonum/graph]: https://godoc.org/gonum.org/v1/gonum/graph [gonum/graph]: https://godoc.org/gonum.org/v1/gonum/graph
[layers.go]: https://github.com/google/nixery/blob/master/builder/layers.go [layers.go]: https://cs.tvl.fyi/depot/-/blob/tools/nixery/builder/layers.go
[popcount]: https://github.com/google/nixery/tree/master/popcount [popcount]: https://cs.tvl.fyi/depot/-/tree/tools/nixery/popcount
[talk]: https://www.youtube.com/watch?v=pOI9H4oeXqA [talk]: https://www.youtube.com/watch?v=pOI9H4oeXqA