docs(caching): Add information about Nixery's caching strategies

tools/nixery/docs/src/SUMMARY.md

@@ -2,6 +2,7 @@
 
 - [Nixery](./nixery.md)
   - [Under the hood](./under-the-hood.md)
+  - [Caching](./caching.md)
   - [Run your own Nixery](./run-your-own.md)
 - [Nix](./nix.md)
   - [Nix, the language](./nix-1p.md)

tools/nixery/docs/src/caching.md

@@ -0,0 +1,70 @@
+# Caching in Nixery
+
+This page gives a quick overview over the caching done by Nixery. All cache data
+is written to Nixery's storage bucket and is based on deterministic identifiers
+or content-addressing, meaning that cache entries under the same key *never
+change*.
+
+## Manifests
+
+Manifests of builds are cached at `$BUCKET/manifests/$KEY`. The effect of this
+cache is that multiple instances of Nixery do not need to rebuild the same
+manifest from scratch.
+
+Since the manifest cache is populated only *after* layers are uploaded, Nixery
+can immediately return the manifest to its clients without needing to check
+whether layers have been uploaded already.
+
+`$KEY` is generated by creating a SHA1 hash of the requested content of a
+manifest plus the package source specification.
+
+Manifests are *only* cached if the package source specification is *not* a
+moving target.
+
+Manifest caching *only* applies in the following cases:
+
+* package source specification is a specific git commit
+* package source specification is a specific NixOS/nixpkgs commit
+
+Manifest caching *never* applies in the following cases:
+
+* package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`)
+* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-19.03`)
+* package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`)
+
+It is thus always preferable to request images from a fully-pinned package
+source.
+
+Manifests can be removed from the manifest cache without negative consequences.
+
+## Layer tarballs
+
+Layer tarballs are the files that Nixery clients retrieve from the storage
+bucket to download an image.
+
+They are stored content-addressably at `$BUCKET/layers/$SHA256HASH` and layer
+requests sent to Nixery will redirect directly to this storage location.
+
+The effect of this cache is that Nixery does not need to upload identical layers
+repeatedly. When Nixery notices that a layer already exists in GCS, it will use
+the object metadata to compare its MD5-hash with the locally computed one and
+skip uploading.
+
+Removing layers from the cache is *potentially problematic* if there are cached
+manifests or layer builds referencing those layers.
+
+To clean up layers, a user must ensure that no other cached resources still
+reference these layers.
+
+## Layer builds
+
+Layer builds are cached at `$BUCKET/builds/$HASH`, where `$HASH` is a SHA1 of
+the Nix store paths included in the layer.
+
+The content of the cached entries is a JSON-object that contains the MD5 and
+SHA256 hashes of the built layer.
+
+The effect of this cache is that different instances of Nixery will not build,
+hash and upload layers that have identical contents across different instances.
+
+Layer builds can be removed from the cache without negative consequences.