chore(nixery): Housekeeping for depot compatibility

Cleans up a whole bunch of things I wanted to get out of the door
right away:

* depot internal references to //third_party/nixery have been replaced
  with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks

Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
This commit is contained in:
Vincent Ambo 2022-04-20 16:41:20 +02:00 committed by clbot
parent 535ad8732a
commit 6716bf018c
32 changed files with 192 additions and 478 deletions

View file

@ -49,6 +49,9 @@ configuration is tracked in `//ops/{modules,machines}`.
* [`//nix/readTree`](https://cs.tvl.fyi/depot/-/blob/nix/readTree/README.md)
contains the Nix code which automatically registers projects in our Nix
attribute hierarchy based on their in-tree location
* [`//tools/nixery`](https://cs.tvl.fyi/depot/-/tree/tools/nixery)
contains the source code of [Nixery][], a container registry that
can build images ad-hoc from Nix packages
* `//nix/yants` contains **Y**et **A**nother **N**ix **T**ype **S**ystem, which
we use for a variety of things throughout the repository
* `//nix/buildGo` implements a Nix library that can build Go software in the
@ -119,3 +122,4 @@ Hackint also provide a [web chat][tvl-webchat].
[hackint-xmpp]: https://hackint.org/transport/xmpp
[tvl-xmpp]: xmpp:#tvl@irc.hackint.org?join
[tvl-webchat]: https://webirc.hackint.org/#ircs://irc.hackint.org/#tvl
[Nixery]: https://nixery.dev

View file

@ -28,7 +28,7 @@ in
StateDirectory = "nixery";
Restart = "always";
ExecStartPre = "${pkgs.coreutils}/bin/mkdir -p ${storagePath}";
ExecStart = "${depot.third_party.nixery.nixery-bin}/bin/nixery";
ExecStart = "${depot.tools.nixery.nixery-bin}/bin/nixery";
};
environment = {

View file

@ -1,19 +0,0 @@
# Import the Nixery repository as-is, but pass our own package set
# instead of the pin it has.
{ depot, pkgs, ... }:
let
inherit (depot.nix.readTree) drvTargets;
commit = "601cd998077f77f257ad1a40fa488add8464650f";
src = pkgs.fetchFromGitHub {
owner = "google";
repo = "nixery";
rev = commit;
sha256 = "195rz25y3hfxcmniysajzjg7g69qhz7w06lql8fn0dbcdcxsq6g4";
};
in
drvTargets (import src {
inherit pkgs;
commitHash = _: commit;
})

View file

@ -24,8 +24,7 @@ let
includes = [ "*.nix" ]
excludes = [
"third_party/nix/tests/*",
"third_party/nix/src/tests/*",
"tools/nixery/*"
"third_party/nix/src/tests/*"
]
[formatter.rust]

View file

@ -1,2 +0,0 @@
# Ignore stylesheet modifications for the book in Linguist stats
*.css linguist-detectable=false

View file

@ -1,27 +0,0 @@
# Build Nixery, spin up an instance and pull an image from it.
name: "Build and test Nixery"
on:
push:
branches:
- master
pull_request: {}
env:
NIX_PATH: "nixpkgs=https://github.com/NixOS/nixpkgs/archive/4263ba5e133cc3fc699c1152ab5ee46ef668e675.tar.gz"
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Install Nix
uses: cachix/install-nix-action@v13
- name: Checkout
uses: actions/checkout@v2.3.4
- name: Prepare environment
run: nix-env -f '<nixpkgs>' -iA go
- name: Check formatting
run: "test -z $(gofmt -l .)"
- name: Run `go vet`
run: "go vet ./..."
- name: Build Nixery
run: "nix-build --no-out-link"
- name: Run integration test
run: scripts/integration-test.sh

View file

@ -1,35 +0,0 @@
# How to Contribute
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
## Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution;
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to <https://cla.developers.google.com/> to see
your current agreements on file or to sign a new one.
You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Commit messages
Commits in this repository follow the [Angular commit message
guidelines][commits].
## Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Community Guidelines
This project follows [Google's Open Source Community
Guidelines](https://opensource.google.com/conduct/).
[commits]: https://github.com/angular/angular/blob/master/CONTRIBUTING.md#commit

View file

@ -4,7 +4,7 @@
-----------------
[![Build Status](https://github.com/tazjin/nixery/actions/workflows/build-and-test.yaml/badge.svg)](https://github.com/tazjin/nixery/actions/workflows/build-and-test.yaml)
[![Build status](https://badge.buildkite.com/016bff4b8ae2704a3bbbb0a250784e6692007c582983b6dea7.svg?branch=refs/heads/canon)](https://buildkite.com/tvl/depot)
**Nixery** is a Docker-compatible container registry that is capable of
transparently building and serving container images using [Nix][].
@ -24,6 +24,15 @@ You can watch the NixCon 2019 [talk about
Nixery](https://www.youtube.com/watch?v=pOI9H4oeXqA) for more information about
the project and its use-cases.
The canonical location of the Nixery source code is
[`//tools/nixery`][depot-link] in the [TVL](https://tvl.fyi)
monorepository. If cloning the entire repository is not desirable, the
Nixery subtree can be cloned like this:
git clone https://code.tvl.fyi/depot.git:/tools/nixery.git
The subtree is infrequently mirrored to `tazjin/nixery` on Github.
## Demo
Click the image to see an example in which an image containing an interactive
@ -139,8 +148,9 @@ separate Nix function, which will make it possible to build images directly in
Nix builds.
[Nix]: https://nixos.org/
[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
[layering strategy]: https://tazj.in/blog/nixery-layers
[gist]: https://gist.github.com/tazjin/08f3d37073b3590aacac424303e6f745
[buildLayeredImage]: https://grahamc.com/blog/nix-and-layered-docker-images
[public]: https://nixery.dev
[depot-link]: https://cs.tvl.fyi/depot/-/tree/tools/nixery
[gcs]: https://cloud.google.com/storage/

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package builder
// This file implements logic for walking through a directory and creating a

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Package builder implements the logic for assembling container
// images. It shells out to Nix to retrieve all required Nix-packages

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package builder
import (

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package builder
import (

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// This package reads an export reference graph (i.e. a graph representing the
// runtime dependencies of a set of derivations) created by Nix and groups it in

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Package config implements structures to store Nixery's configuration at
// runtime as well as the logic for instantiating this configuration from the

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package config
import (

View file

@ -1,16 +1,5 @@
# Copyright 2019-2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# This function header aims to provide compatibility between builds of
# Nixery taking place inside/outside of the TVL depot.
@ -24,7 +13,8 @@
, extraPackages ? [ ]
, maxLayers ? 20
, commitHash ? null
, ... }@args:
, ...
}@args:
with pkgs;
@ -54,7 +44,8 @@ let
"-ldflags=-s -w -X main.version=${nixery-commit-hash}"
];
};
in depot.nix.readTree.drvTargets rec {
in
depot.nix.readTree.drvTargets rec {
# Implementation of the Nix image building logic
nixery-prepare-image = import ./prepare-image { inherit pkgs; };
@ -79,7 +70,8 @@ in depot.nix.readTree.drvTargets rec {
# Container image containing Nixery and Nix itself. This image can
# be run on Kubernetes, published on AppEngine or whatever else is
# desired.
nixery-image = let
nixery-image =
let
# Wrapper script for the wrapper script (meta!) which configures
# the container environment appropriately.
#
@ -110,7 +102,8 @@ in depot.nix.readTree.drvTargets rec {
exec ${nixery-bin}/bin/nixery
'';
in dockerTools.buildLayeredImage {
in
dockerTools.buildLayeredImage {
name = "nixery";
config.Cmd = [ "${nixery-launch-script}/bin/nixery" ];

View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# Builds the documentation page using the Rust project's 'mdBook'
# tool.
@ -27,7 +16,8 @@ let
rev = "9f0baf5e270128d9101ba4446cf6844889e399a2";
sha256 = "1pf9i90gn98vz67h296w5lnwhssk62dc6pij983dff42dbci7lhj";
};
in runCommand "nixery-book" { } ''
in
runCommand "nixery-book" { } ''
mkdir -p $out
cp -r ${./.}/* .
chmod -R a+w src

View file

@ -68,10 +68,6 @@ production project we recommend setting up a private instance. The public Nixery
at `nixery.dev` is run on a best-effort basis and we make no guarantees about
availability.
### Is this an official Google project?
**No.** Nixery is not officially supported by Google.
### Who made this?
Nixery was written by [tazjin][], but many people have contributed to Nix over
@ -81,4 +77,4 @@ time, maybe you could become one of them?
[Nix]: https://nixos.org/nix
[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
[layers]: https://grahamc.com/blog/nix-and-layered-docker-images
[tazjin]: https://github.com/tazjin
[tazjin]: https://tazj.in

View file

@ -65,13 +65,17 @@ use it with your own packages. There are three options available:
### 2.1. With a container image
The easiest way to run Nixery is to build a container image.
This section assumes that the container runtime used is Docker,
please modify instructions accordingly if
you are using something else.
The easiest way to run Nixery is to build a container image. This
section assumes that the container runtime used is Docker, please
modify instructions accordingly if you are using something else.
With a working Nix installation, building Nixery is done by invoking `nix-build
-A nixery-image` from a checkout of the [Nixery repository][repo].
With a working Nix installation, you can clone and build the Nixery
image like this:
```
git clone https://code.tvl.fyi/depot.git:/tools/nixery.git
nix-build -A nixery-image
```
This will create a `result`-symlink which points to a tarball containing the
image. In Docker, this tarball can be loaded by using `docker load -i result`.
@ -184,7 +188,6 @@ If the directory doesn't exist, Nixery will run fine but serve 404.
[nixery#4]: https://github.com/tazjin/nixery/issues/4
[Nix]: https://nixos.org/nix
[gcs]: https://cloud.google.com/storage/
[repo]: https://github.com/tazjin/nixery
[signed-urls]: under-the-hood.html#5-image-layers-are-requested
[ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
[nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
package logs
// This file configures different log formatters via logrus. The

View file

@ -1,16 +1,5 @@
// Copyright 2019-2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// The nixery server implements a container registry that transparently builds
// container images based on Nix derivations.

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Package image implements logic for creating the image metadata
// (such as the image manifest and configuration).

View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
{ buildGoPackage }:

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Popcount fetches popularity information for each store path in a
// given Nix channel from the upstream binary cache.

View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# This file builds a wrapper script called by Nixery to ask for the
# content information for a given image.

View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# Load a Nix package set from one of the supported source types
# (nixpkgs, git, path).
@ -24,7 +13,8 @@ let
let
url =
"https://github.com/NixOS/nixpkgs/archive/${channel}.tar.gz";
in import (fetchTarball url) importArgs;
in
import (fetchTarball url) importArgs;
# If a git repository is requested, it is retrieved via
# builtins.fetchGit which defaults to the git configuration of the
@ -35,7 +25,8 @@ let
# No special handling is used for paths, so users are expected to pass one
# that will work natively with Nix.
importPath = path: import (toPath path) importArgs;
in if srcType == "nixpkgs" then
in
if srcType == "nixpkgs" then
fetchImportChannel srcArgs
else if srcType == "git" then
fetchImportGit (fromJSON srcArgs)

View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# This file contains a derivation that outputs structured information
# about the runtime dependencies of an image with a given set of
@ -23,13 +12,13 @@
{
# Description of the package set to be used (will be loaded by load-pkgs.nix)
srcType ? "nixpkgs",
srcArgs ? "nixos-20.09",
system ? "x86_64-linux",
importArgs ? { },
# Path to load-pkgs.nix
loadPkgs ? ./load-pkgs.nix,
# Packages to install by name (which must refer to top-level attributes of
srcType ? "nixpkgs"
, srcArgs ? "nixos-20.09"
, system ? "x86_64-linux"
, importArgs ? { }
, # Path to load-pkgs.nix
loadPkgs ? ./load-pkgs.nix
, # Packages to install by name (which must refer to top-level attributes of
# nixpkgs). This is passed in as a JSON-array in string form.
packages ? "[]"
}:
@ -77,7 +66,8 @@ let
# `deepFetch haskellpackages.stylish-haskell` retrieves
# `haskellPackages.stylish-haskell`.
deepFetch = with lib; s: n:
let path = splitString "." n;
let
path = splitString "." n;
err = { error = "not_found"; pkg = n; };
# The most efficient way I've found to do a lookup against
# case-differing versions of an attribute is to first construct a
@ -88,13 +78,16 @@ let
# (case-sensitive) one does not yield a result.
hasUpper = str: (match ".*[A-Z].*" str) != null;
allUpperKeys = filter hasUpper (attrNames s);
lowercased = listToAttrs (map (k: {
lowercased = listToAttrs (map
(k: {
name = toLower k;
value = k;
}) allUpperKeys);
})
allUpperKeys);
caseAmendedPath = map (v: if hasAttr v lowercased then lowercased."${v}" else v) path;
fetchLower = attrByPath caseAmendedPath err s;
in attrByPath path fetchLower s;
in
attrByPath path fetchLower s;
# allContents contains all packages successfully retrieved by name
# from the package set, as well as any errors encountered while
@ -105,19 +98,22 @@ let
# Folds over the results of 'deepFetch' on all requested packages to
# separate them into errors and content. This allows the program to
# terminate early and return only the errors if any are encountered.
let splitter = attrs: res:
let
splitter = attrs: res:
if hasAttr "error" res
then attrs // { errors = attrs.errors ++ [ res ]; }
else attrs // { contents = attrs.contents ++ [ res ]; };
init = { contents = [ ]; errors = [ ]; };
fetched = (map (deepFetch pkgs) (fromJSON packages));
in foldl' splitter init fetched;
in
foldl' splitter init fetched;
# Contains the export references graph of all retrieved packages,
# which has information about all runtime dependencies of the image.
#
# This is used by Nixery to group closures into image layers.
runtimeGraph = runCommand "runtime-graph.json" {
runtimeGraph = runCommand "runtime-graph.json"
{
__structuredAttrs = true;
exportReferencesGraph.graph = allContents.contents;
PATH = "${coreutils}/bin";
@ -159,7 +155,8 @@ let
# Metadata about the symlink layer which is required for serving it.
# Two different hashes are computed for different usages (inclusion
# in manifest vs. content-checking in the layer cache).
symlinkLayerMeta = fromJSON (readFile (runCommand "symlink-layer-meta.json" {
symlinkLayerMeta = fromJSON (readFile (runCommand "symlink-layer-meta.json"
{
buildInputs = [ coreutils jq openssl ];
} ''
tarHash=$(sha256sum ${symlinkLayer} | cut -d ' ' -f1)
@ -181,7 +178,8 @@ let
error = "not_found";
pkgs = map (err: err.pkg) allContents.errors;
};
in writeText "build-output.json" (if (length allContents.errors) == 0
in
writeText "build-output.json" (if (length allContents.errors) == 0
then toJSON buildOutput
else toJSON errorOutput
)

View file

@ -1,16 +1,5 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright 2022 The TVL Contributors
# SPDX-License-Identifier: Apache-2.0
# Configures a shell environment that builds required local packages to
# run Nixery.

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Filesystem storage backend for Nixery.
package storage

View file

@ -1,16 +1,5 @@
// Copyright 2019 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Google Cloud Storage backend for Nixery.
package storage

View file

@ -1,16 +1,5 @@
// Copyright 2019-2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may not
// use this file except in compliance with the License. You may obtain a copy of
// the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations under
// the License.
// Copyright 2022 The TVL Contributors
// SPDX-License-Identifier: Apache-2.0
// Package storage implements an interface that can be implemented by
// storage backends, such as Google Cloud Storage or the local

View file

@ -260,13 +260,13 @@ TIP: This is implemented in [popcount][] in Nixery.
Hopefully this detailed design review was useful to you. You can also watch [my
NixCon talk][talk] about Nixery for a review of some of this, and some demos.
[Nixery]: https://github.com/google/nixery
[Nixery]: https://cs.tvl.fyi/depot/-/tree/tools/nixery
[grhmc]: https://grahamc.com/blog/nix-and-layered-docker-images
[Nix]: https://nixos.org/nix
[registry protocols]: https://github.com/opencontainers/distribution-spec/blob/master/spec.md
[nixery.dev]: https://nixery.dev
[dominator trees]: https://en.wikipedia.org/wiki/Dominator_(graph_theory)
[gonum/graph]: https://godoc.org/gonum.org/v1/gonum/graph
[layers.go]: https://github.com/google/nixery/blob/master/builder/layers.go
[popcount]: https://github.com/google/nixery/tree/master/popcount
[layers.go]: https://cs.tvl.fyi/depot/-/blob/tools/nixery/builder/layers.go
[popcount]: https://cs.tvl.fyi/depot/-/tree/tools/nixery/popcount
[talk]: https://www.youtube.com/watch?v=pOI9H4oeXqA