I think migrating the execline scripts over to bash makes sense:
1. Ever since nixpkgs-fmt, execline scripts in depot have become a huge
pain to write and edit and I can't think of a satisfying solution to
this problem.
2. The scripts here require remembering things across loop cycles (i. e.
the status variable) which is not possible in pure execline. As a a
workaround we used to read the entire report into memory first and
check if it was empty (tying us to the argv limit for the report
length).
Change-Id: I954b08b982ef947f9014a685676d2b83a2aec4d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5259
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
* //ops/machines/whitby: Disable grafana, since the grafana module was
changed upstream in a way that our configuration no longer works.
Since the OpenSSL security update is relatively pressing, adapting the
grafana configuration beforehand is not a hard requirement. See
https://github.com/NixOS/nixpkgs/pull/191768.
* //tools/depotfmt: keep Go at version 1.18 to forgo a reformat of the
tree.
* //nix/buildGo: keep Go at version 1.18, as 1.19 changed the CLI
interface (?) in a way that breaks buildGo.
* //3p/overlays/tvl: drop upstreamed tdlib upgrade.
* //3p/overlays/tvl: patch buf to work around breakage due to git 2.38.1
TODO items for Go are tracked in b/215.
Change-Id: Ie08fef49cf3db12e6b5225a8b992a990ddc5b642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7141
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
The version of buf used is quite old.
nixpkgs provides a more recent version, but it requires us to migrate
config to the latest version.
depot_scanner.proto doesn't honor some of the conventions, so we need
allow_comment_ignores and drop a bunch of comments in there.
Change-Id: Ic978fe92fb7c8471f58c137497528f18aad8f3ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7053
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
Nixery's previous landing page was an mdBook that was basically
unmaintained and full of incorrect information. It also duplicated
some things (like nix-1p) which actually live elsewhere.
This commit removes the mdBook completely and reduces it down to a
simple TVL-style landing page. The landing page has been checked in
in its entirety because Nixery is frequently cloned through josh
without the entirety of depot, however the page has been created by
building it through depot's //web/tvl/template.
See also https://github.com/tazjin/nixery/issues/156
Change-Id: I20e1d58f1e6608377207e80345c169f7d92d3847
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6930
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The latter has been deprecated in nixpkgs.
Relates to b/200
Change-Id: I42871ce3eb54ebf092909f033b43936b9610d982
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6836
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.
* runCommandNoCC -> runCommand
* gmailieer -> lieer
We also need to work around the fact that home-manager hasn't catched
on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy
Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.
Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Discard string context in prepare-image.nix before parsing input read
with readFile with fromJSON. Required for compatibility with nix >2.3.
Change-Id: I3830707e80fd19a700551a15f1a96d2841d0b022
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6696
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Remove a race condition which appears when uploadHashLayer is called
with the same key from multiple threads simultaneously. This can
easily happen when the same image path is requested by multiple
clients at the same time. When it does, a 500 status is returned and
the following error message is logged:
{
"context": {
"filePath": "github.com/google/nixery/builder/builder.go",
"lineNumber": 440,
"functionName": "github.com/google/nixery/builder.uploadHashLayer"
},
"error": "rename /var/lib/nixery/staging/<hash> /var/lib/nixery/layers/<hash>: no such file or directory",
"eventTime": "...",
"layer": "<hash>",
"message": "failed to move layer from staging",
...
}
To solve this issue, introduce a mutex keyed on the uploaded hash and
move all layer caching into uploadHashLayer. This could additionally
provide a small performance benefit when an already built image is
requested and NIXERY_PKGS_PATH is set, since symlink layers and config
layers are now also cached.
Change-Id: I50788a7ec7940cb5e5760f244692e361019a9bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6695
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds scaffolding code for running the Nix language test suite.
The majority of eval-okay-* tests should eventually be runnable as-is
by Tvix, however the eval-fail-* tests might not as we intend to have
more useful error messages than upstream Nix.
Change-Id: I4f3227f0889c55e4274b804a3072850fb78dd1bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6126
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
This adds a new function (intentionally bound to a rare key (Q)) in
the push menu which can push a *private* change to Gerrit.
A private change is one that, until submitted, is only visible to its
owner and all explicitly added people (reviewers, CC).
Change-Id: I6ee13dbbad099584475d3efac96e5d9b86efbc26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6061
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Since the source of nix-1p is checked in under //nix/nix-1p, we should
use it from there if Nixery is being built inside of depot.
Change-Id: Iddd54f7b93b398b2f909db6ee105366a9914a2ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5882
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
People occasionally ask what the current nixpkgs commit is on
nixery.dev (see e.g. https://github.com/tazjin/nixery/issues/153).
With this change, the commit is displayed on nixery.dev if Nixery is
built for the TVL deployment.
Change-Id: I795220214db5a367a126c9b4bd03754e9f144940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5881
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This can be re-used across Terraform environments.
Change-Id: I3d964a17d1cda1aff1df12bd4c0c3ee84b7f7748
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5850
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
In order to run this the secrets needs to be sourced, e.g.:
eval $(age --decrypt -i ~/.ssh/id_ed25519 $(git rev-parse --show-toplevel)/ops/secrets/tf-buildkite.age)
Change-Id: I9f6a02c0dac22f584181635861ddbb06cf849f14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5838
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
`mg repl` is essentially a shortcut for nix repl $(mg path //) which
comes up often enough for me. Launching a repl only really makes sense
in the repository root with how readTree works at the moment, so I think
this is a convenient addition.
Change-Id: I32b695885c2e6eaecdcc656c7249afa504439913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5822
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This exports the `:/tools/nixery` subtree to Github automatically
after merges to `canon`.
Due to the way the project was imported this continues the existing
git history in the external repository.
Change-Id: Ie871c14ad5d8f1019f8be86adecbe9b130ffb01a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5667
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This adds an extra step definition which can push the result of
running a josh filter on the repository to Github.
Change-Id: I1f93ae78e1bf452fbd1b21ce943a60acc85c944f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5666
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Nixery is going to gain a new binary (used for building images without
a registry server); to prepare for this the server binary has moved to
cmd/server and the Nix build logic has been updated to wrap this
binary and set the required environment variables.
Change-Id: I9b4f49f47872ae76430463e2fcb8f68114070f72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5603
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This will be required for making a standalone, Nixery-style image
builder function usable from Nix.
Change-Id: I5e36348bd4c32d249d56f6628cd046916691319f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5601
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The camelCase variant of `rebuild-system` doesn't exist, but the
kebab-case version does.
Side note: this `lazy-dispatch` upgrade is pretty cool. TIL `direnv`
supports `watch_file` and `PATH_add`.
Change-Id: Idc9109a9b0de327ddf7b9c6a4368b7bebb551196
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5565
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: wpcarro <wpcarro@gmail.com>
nix-shell pollutes the environment with all sorts of variables. Let's
just add the tools to the PATH?
This also papers over the various differences in users `use_nix`
implementations by not using it at all.
Change-Id: If4282531fd6b7453b3611fe50217beacadc08bb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5524
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This modifies the envrc configuration to add the result of building
//tools/depot-deps to $PATH, instead of dispatching through the
manually maintained list of symlinks.
While at it, I've cleaned up some stuff from that list that is no
longer actually used.
Change-Id: If345c44da75b23c06b7c7f435be0cb02f99aaac5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5513
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
in some cases, users might want to pass through flags for
nix-build (such as `-j`).
magrathea now accepts these as arguments to `mg build`, as long as
they are separated by `--`.
the arguments passed to `mg build` are parsed into a proper record,
which enables us to show users very clear error messages in case they
forget to use the `--` separator and keeping us future-compatible with
more potential arguments to magrathea itself.
Change-Id: I81f5d9db52779a5cc3b8bbdd975316274fffe5fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5507
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: asmundo <asmundo@gmail.com>
Cleans up a whole bunch of things I wanted to get out of the door
right away:
* depot internal references to //third_party/nixery have been replaced
with //tools/nixery
* cleaned up files from Github
* fixed SPDX & Copyright headers
* code formatting and inclusion in //tools/depotfmt checks
Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
This does not fully change the build structure of Nixery to be
depot-compatible yet, but should allow most targets to be built in
depot CI.
This contains some hacks to work around surface incompatibilities
which we'll clear away later.
Change-Id: I84e7734334abbe299983956f528c0897f49fa8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5485
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This absorbs a josh-filtered Nix subtree into depot, at
//tools/nixery.
This subtree was created through `josh-filter ':prefix=tools/nixery'`,
which allows a filter on tools/nixery to yield the same commit hashes
as the original Nixery repository (allowing for history continuity).
Change-Id: Icc1a99bf1248226b91f437b0a90361d36fb0d327
This tool has been replaced by niv.
Change-Id: I011059b7d8890d0456b22f066e723584cc1d9a2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5329
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This has been superseded by magrathea.
Change-Id: Ief4a3d1b81e51e7a9c9a0112584fa7efc8aca63f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5328
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This command builds the desired target and runs the executable produced
by it. If a directory is produced, it looks for a single (!) executable
in the bin directory. Dot files are ignored, so wrappers should
generally work. In the future we could provide a flag to select one of
multiple executables.
All arguments following the target are passed to the executable as is.
Examples:
mg run ops/mq_cli ls
mg run web/bubblegum:examples
Change-Id: I6490668af68e028520973196d9daa5f1d58969ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5277
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Instead of the strict check-all-our-crates, generate a fake Cargo.lock
and add it to the report generated by check-all-our-lock-files.
check-all-our-crates was a reimplementation of cargo-audit anyways and
prevented us from updating the advisory db due to its strict
model (failing on any advisory).
Change-Id: I264a7f1a5058a527cbc46d26225352ecd437a22b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5230
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Rename check-all-our-lock-files to tree-lock-file-report and pull out
all the buildkite-specific code which makes the code less awkward.
check-all-our-lock-files is then only executed in extraSteps and runs
tree-lock-file-report on depot, adding it as a warning to the pipeline
if it is non-empty.
Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This script is somewhat usable by humans (it even has a help screen!)
and can be reused in //users/sterni/nixpkgs-crate-holes. We are using
bash since that allows us to exit with the actual exit code of
cargo-audit - something that's not possible in execline.
Change-Id: I3331ae8222a20e23b8e30dc920ab48af78f0247c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5228
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
I want to add a shortcut to build and run e.g. scripts that are depot
targets - for which it would be useful to not have stdout polluted by
magrathea itself.
Change-Id: Ic58fe28eafb4d0715e53beae041bfaa5d1745812
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5276
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The rustfmt configuration of the depot is moved to `rustfmt.toml` (it
is recognised more reliably from this path than from the hidden
.rustfmt.toml).
Nested configuration is theoretically possible, but detection of
nested config files is flaky. Paths with nested config files need to
be disabled in the top-level check (I've excluded my user directory).
Change-Id: I385ce3ef529bda28fac03bfba86fc204c81b8a61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5241
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
apparently the cache can get out-of-sync or something; either way we
had a build where it missed a file that was misformated.
Change-Id: I2967aec99ee1c7c8b978a3dfdfed4ff213bb6591
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5249
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
