Commit graph

7 commits

Author SHA1 Message Date
talyz
28417afbb4 fix(nixery): Avoid race when the same image is fetched in parallel
Remove a race condition which appears when uploadHashLayer is called
with the same key from multiple threads simultaneously. This can
easily happen when the same image path is requested by multiple
clients at the same time. When it does, a 500 status is returned and
the following error message is logged:

{
  "context": {
    "filePath": "github.com/google/nixery/builder/builder.go",
    "lineNumber": 440,
    "functionName": "github.com/google/nixery/builder.uploadHashLayer"
  },
  "error": "rename /var/lib/nixery/staging/<hash> /var/lib/nixery/layers/<hash>: no such file or directory",
  "eventTime": "...",
  "layer": "<hash>",
  "message": "failed to move layer from staging",
  ...
}

To solve this issue, introduce a mutex keyed on the uploaded hash and
move all layer caching into uploadHashLayer. This could additionally
provide a small performance benefit when an already built image is
requested and NIXERY_PKGS_PATH is set, since symlink layers and config
layers are now also cached.

Change-Id: I50788a7ec7940cb5e5760f244692e361019a9bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6695
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-19 11:28:10 +00:00
Vincent Ambo
796ff086be refactor(nixery): Extract layering logic into separate package
This will be required for making a standalone, Nixery-style image
builder function usable from Nix.

Change-Id: I5e36348bd4c32d249d56f6628cd046916691319f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5601
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-23 15:04:56 +00:00
Vincent Ambo
6716bf018c chore(nixery): Housekeeping for depot compatibility
Cleans up a whole bunch of things I wanted to get out of the door
right away:

* depot internal references to //third_party/nixery have been replaced
  with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks

Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-04-20 15:31:16 +00:00
Jérôme Petazzoni
dd7de32c36 feat: set SSL_CERT_FILE and provide a Cmd
Two minor "quality of life" improvements:
- automatically set SSL_CERT_FILE environment variable,
  so that programs relying on OpenSSL for certificate
  validation can actually validate certificates
  (the certificates are included no matter what since
  we add the "cacert" package to all iamges)
- if the requested image includes an interactive shell
  (e.g. if it includes the "shell" metapackage), set
  the image Cmd to "bash", which allows to execute
  "docker run nixery.dev/shell" and get a shell)

I'm happy to split this PR in two if you'd like, but
since both features touch the Config structure and are
rather small, I thought it would make sense to bundle
them together.
2021-12-27 11:26:54 +03:00
Vincent Ambo
cc35bf0fc3 feat(storage): Add support for content-types (GCS only)
Extends storage.Persist to accept a Content-Type argument, which in
the GCS backend is persisted with the object to ensure that the object
is served back with this content-type.

This is not yet implemented for the filesystem backend, where the
parameter is simply ignored.

This should help in the case of clients which expect the returned
objects to have content-types set when, for example, fetching layers
by digest.
2020-10-29 17:07:52 +01:00
Vincent Ambo
1031d890ec fix(builder): Fix minor logging switcharoo 2020-01-19 07:44:36 +00:00
Vincent Ambo
2b82f1b71a refactor: Reshuffle file structure for better code layout
This gets rid of the package called "server" and instead moves
everything into the project root, such that Go actually builds us a
binary called `nixery`.

This is the first step towards factoring out CLI-based functionality
for Nixery.
2019-11-27 14:12:38 +00:00
Renamed from tools/nixery/server/builder/builder.go (Browse further)