Remove a race condition which appears when uploadHashLayer is called
with the same key from multiple threads simultaneously. This can
easily happen when the same image path is requested by multiple
clients at the same time. When it does, a 500 status is returned and
the following error message is logged:
{
"context": {
"filePath": "github.com/google/nixery/builder/builder.go",
"lineNumber": 440,
"functionName": "github.com/google/nixery/builder.uploadHashLayer"
},
"error": "rename /var/lib/nixery/staging/<hash> /var/lib/nixery/layers/<hash>: no such file or directory",
"eventTime": "...",
"layer": "<hash>",
"message": "failed to move layer from staging",
...
}
To solve this issue, introduce a mutex keyed on the uploaded hash and
move all layer caching into uploadHashLayer. This could additionally
provide a small performance benefit when an already built image is
requested and NIXERY_PKGS_PATH is set, since symlink layers and config
layers are now also cached.
Change-Id: I50788a7ec7940cb5e5760f244692e361019a9bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6695
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
After the discussion in #116, this stores the blob content types
in extended attributes when using the filesystem backend.
If the underlying filesystem doesn't support extended attributes,
storing blobs won't work; also, if extended attributes get removed,
blobs won't be served anymore. We can relax this behavior if
needed (i.e. log errors but still accept to store or serve blobs).
However, since the Docker Engine (and possibly other container
engines) won't accept to pull images from a registry that doesn't
use correct content types for manifest files, it could be argued
that it's better to give a hard fail. (Otherwise, the container
engine gives cryptic error messages like "missing signature key".)
I can change that behavior (and log errors but still store/serve
blobs to the filesystem) if you think it's better.
Drops the go2nix configuration in favour of pkgs.buildGoModule.
Note that the go.sum file is bloated by issues with cyclic
dependencies in some Google projects, but this large number of
dependencies is not actually built.
