These packages are invalid in Nix, and worked around in nixpkgs with
underscores, but the underscores are invalid in the Docker registry
protocol.
We work around this by detecting this case and adding the underscore
to yield the correct package reference. There is no case where this
workaround can break something, as there can be no valid package
matching the regular expression.
This relates to https://github.com/tazjin/nixery/issues/158
Change-Id: I7990cdb534a8e86c2ceee2c589a2636af70a4a03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8531
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Usually the current behavior is best: You are dropped in a REPL with the
package(s) you are working on already available. As you are working on
them, you recompile individual files and your changes become available.
However, I've found that there are some occasions when this is not
desireable, e.g.: When you are working on something and have broken the
test suite intermittently, it becomes impossible to start a new REPL.
Not sure how the yes-or-no-p question should be phrased, its negation
may be better?
Change-Id: I6a37ebc02f3121f628fc9206e0de650851824cd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8415
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Instead of prepending :unsign to all URLs in josh-proxy, and for all
calls to filteredGitPush, explicitly use it only in the filter we use
for the `export-kit` extraStep.
This means, people cloning tvl-kit via
> https://code.tvl.fyi/depot.git:workspace=views/kit.git
now need to update the URL to point to
> https://code.tvl.fyi/depot.git:unsign:workspace=views/kit.git
instead.
git@github.com:tvlfyi/kit.git will keep the same hashes, as it's updated
to export the unsigned workspace view of it.
This is less invasive than dooming every josh workspace to have to strip
signatures.
Change-Id: I6de05182fad4c3695081388c3bbf37306521d255
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8369
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
We need to followup on cl/8186, and use the same workaround for josh-
filter that we use for josh-proxy, so we can push again.
This could potentially become an optional function argument, if new
subtrees are to be carved out, and we want to keep signatures on commits
in there.
Change-Id: I7a9b821a7365767e8a4188e8200fa3c37463142c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8368
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: flokli <flokli@flokli.de>
Upgrade to syntect 5.0 and load the new kind of syntax set
serialisation with the new helper function for that purpose.
Includes other minor API fixes as well, note that the things that are
now calling `expect` previously failed internally at those points and
we're reasonably confident they don't fail in production.
This has been waiting for a long time ...
Change-Id: I8af4fef995ff64bfbe24e1f13917fa50ecb6e4ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7787
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is a semi-manual bump of all dependencies (except syntect, which
is more complex to update).
Change-Id: I8c678a16d779f3f896b95f7d161710ac39d38e88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7786
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
I think what might be going on with b/231 is that the annotations
somehow started conflicting because they don't have contexts set.
Lets try setting a context and see if it changs anything ...
Change-Id: I62ed57f9e24f08e4e7215f05d35cfa769e2e2c24
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7640
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This script fetches the inbox for depot@tvl.su into the specified
directory in maildir format.
The layout of the folder follows the structure generated by public
inbox, i.e. the directory containing the current maildir will be
`$TARGET/su.tvl.depot.0`, but most mail clients (e.g notmuch) will
figure this out on their own.
----
In addition, we would ideally find a CLI mail client that can be
pointed at an arbitrary maildir (or an IMAP server) and works with
local `sendmail` config so that people can have a single command entry
point to interacting with depot@tvl.su.
Change-Id: Iaf9fcce73e9caa2f202327488c43d0394be26ca6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7644
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is the new address which leads to the public inbox at inbox.tvl.su
Change-Id: I45d98a373b8acda49b05c4f74669ffb9ad1f1a3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7632
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
These are both unused things from a long time ago, which we don't need
to keep around anymore.
Their design doc has been marked as archived.
Change-Id: Icd2744e511e78ec95ec8f39e5f79ed1fe98e9e4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7639
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The //tvix README already steers to `mg run //tvix:crate2nixGenerate
--`, there's no point in /also/ having a non-formatting version of
crate2nix in ``$PATH`.
Change-Id: Idc6409799ae5f0629376eef6eeff6eb9eaa4fb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7613
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
As we start using this in more parts of depot, it makes sense to add
it to the available tools.
Change-Id: I148902714167b36bc51aeca4a241c79ad8a59285
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7562
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
With this, we can disambiguate the following `mg run` invocations:
```shell
$ mg run :foo # run the virtual target, foo, with no args
$ mg run -- :foo # run the empty target and pass the arg, :foo, to it
```
Change-Id: Id6395b36a4d8ef3f325937e322e1c27b8630b556
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7408
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Something I missed last time reading through the process documentation
is that you can use a combination of `process` and `process-wait` to
determine the exit status of a child process *and* read from its
standard output. With `process*` we could even capture stderr, but we
probably want it mounted to the parent process' stderr anyways.
Change-Id: I9840f607df465caa80d28109e344e5fc1402949d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7259
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Profpatsch originally implemented an advisory checker from scratch in
Rust. We now ended up just using cargo-audit for the global checks
exposed via CI and the custom implementation is unused. To clean up
//tools/rust-crates-advisory a bit, we can move the unused parts to his
user directory.
Change-Id: Iacbd27c163edd07c804220fd1b3569c23aebd3e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7171
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
I've elected to split the check-all-our-lock-files script into two new
scripts: One very simple script which generates the report by invoking
lock-file-report on the fake lock file for //third_party/rust-crates and
all lock files in depot, and one which executes this and adds it as a
buildkite annotation if there are any warnings (which is reported by the
report generating script using a non zero exit code).
The latter script could become the basis for generalizing buildkite
annotations, a slight attempt at making it easily reusable in the future
has been made. So far we expect a report generating script to exit non
zero if a report should be made and to print commonmark to stdout. In
the future we may want to use a JSON format for generating the report,
allowing us to filter it by buildkite target (using the drvmap to
exclude certain reports, potentially).
Change-Id: I1df9e440509d69adff5b8e6304105a45dc62c018
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5260
Reviewed-by: kn <klemens@posteo.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
I think migrating the execline scripts over to bash makes sense:
1. Ever since nixpkgs-fmt, execline scripts in depot have become a huge
pain to write and edit and I can't think of a satisfying solution to
this problem.
2. The scripts here require remembering things across loop cycles (i. e.
the status variable) which is not possible in pure execline. As a a
workaround we used to read the entire report into memory first and
check if it was empty (tying us to the argv limit for the report
length).
Change-Id: I954b08b982ef947f9014a685676d2b83a2aec4d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5259
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
* //ops/machines/whitby: Disable grafana, since the grafana module was
changed upstream in a way that our configuration no longer works.
Since the OpenSSL security update is relatively pressing, adapting the
grafana configuration beforehand is not a hard requirement. See
https://github.com/NixOS/nixpkgs/pull/191768.
* //tools/depotfmt: keep Go at version 1.18 to forgo a reformat of the
tree.
* //nix/buildGo: keep Go at version 1.18, as 1.19 changed the CLI
interface (?) in a way that breaks buildGo.
* //3p/overlays/tvl: drop upstreamed tdlib upgrade.
* //3p/overlays/tvl: patch buf to work around breakage due to git 2.38.1
TODO items for Go are tracked in b/215.
Change-Id: Ie08fef49cf3db12e6b5225a8b992a990ddc5b642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7141
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
The version of buf used is quite old.
nixpkgs provides a more recent version, but it requires us to migrate
config to the latest version.
depot_scanner.proto doesn't honor some of the conventions, so we need
allow_comment_ignores and drop a bunch of comments in there.
Change-Id: Ic978fe92fb7c8471f58c137497528f18aad8f3ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7053
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
Nixery's previous landing page was an mdBook that was basically
unmaintained and full of incorrect information. It also duplicated
some things (like nix-1p) which actually live elsewhere.
This commit removes the mdBook completely and reduces it down to a
simple TVL-style landing page. The landing page has been checked in
in its entirety because Nixery is frequently cloned through josh
without the entirety of depot, however the page has been created by
building it through depot's //web/tvl/template.
See also https://github.com/tazjin/nixery/issues/156
Change-Id: I20e1d58f1e6608377207e80345c169f7d92d3847
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6930
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
The latter has been deprecated in nixpkgs.
Relates to b/200
Change-Id: I42871ce3eb54ebf092909f033b43936b9610d982
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6836
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.
* runCommandNoCC -> runCommand
* gmailieer -> lieer
We also need to work around the fact that home-manager hasn't catched
on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy
Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.
Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Discard string context in prepare-image.nix before parsing input read
with readFile with fromJSON. Required for compatibility with nix >2.3.
Change-Id: I3830707e80fd19a700551a15f1a96d2841d0b022
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6696
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Remove a race condition which appears when uploadHashLayer is called
with the same key from multiple threads simultaneously. This can
easily happen when the same image path is requested by multiple
clients at the same time. When it does, a 500 status is returned and
the following error message is logged:
{
"context": {
"filePath": "github.com/google/nixery/builder/builder.go",
"lineNumber": 440,
"functionName": "github.com/google/nixery/builder.uploadHashLayer"
},
"error": "rename /var/lib/nixery/staging/<hash> /var/lib/nixery/layers/<hash>: no such file or directory",
"eventTime": "...",
"layer": "<hash>",
"message": "failed to move layer from staging",
...
}
To solve this issue, introduce a mutex keyed on the uploaded hash and
move all layer caching into uploadHashLayer. This could additionally
provide a small performance benefit when an already built image is
requested and NIXERY_PKGS_PATH is set, since symlink layers and config
layers are now also cached.
Change-Id: I50788a7ec7940cb5e5760f244692e361019a9bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6695
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds scaffolding code for running the Nix language test suite.
The majority of eval-okay-* tests should eventually be runnable as-is
by Tvix, however the eval-fail-* tests might not as we intend to have
more useful error messages than upstream Nix.
Change-Id: I4f3227f0889c55e4274b804a3072850fb78dd1bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6126
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
This adds a new function (intentionally bound to a rare key (Q)) in
the push menu which can push a *private* change to Gerrit.
A private change is one that, until submitted, is only visible to its
owner and all explicitly added people (reviewers, CC).
Change-Id: I6ee13dbbad099584475d3efac96e5d9b86efbc26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6061
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Since the source of nix-1p is checked in under //nix/nix-1p, we should
use it from there if Nixery is being built inside of depot.
Change-Id: Iddd54f7b93b398b2f909db6ee105366a9914a2ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5882
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
People occasionally ask what the current nixpkgs commit is on
nixery.dev (see e.g. https://github.com/tazjin/nixery/issues/153).
With this change, the commit is displayed on nixery.dev if Nixery is
built for the TVL deployment.
Change-Id: I795220214db5a367a126c9b4bd03754e9f144940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5881
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This can be re-used across Terraform environments.
Change-Id: I3d964a17d1cda1aff1df12bd4c0c3ee84b7f7748
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5850
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
In order to run this the secrets needs to be sourced, e.g.:
eval $(age --decrypt -i ~/.ssh/id_ed25519 $(git rev-parse --show-toplevel)/ops/secrets/tf-buildkite.age)
Change-Id: I9f6a02c0dac22f584181635861ddbb06cf849f14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5838
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
`mg repl` is essentially a shortcut for nix repl $(mg path //) which
comes up often enough for me. Launching a repl only really makes sense
in the repository root with how readTree works at the moment, so I think
this is a convenient addition.
Change-Id: I32b695885c2e6eaecdcc656c7249afa504439913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5822
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This exports the `:/tools/nixery` subtree to Github automatically
after merges to `canon`.
Due to the way the project was imported this continues the existing
git history in the external repository.
Change-Id: Ie871c14ad5d8f1019f8be86adecbe9b130ffb01a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5667
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This adds an extra step definition which can push the result of
running a josh filter on the repository to Github.
Change-Id: I1f93ae78e1bf452fbd1b21ce943a60acc85c944f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5666
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Nixery is going to gain a new binary (used for building images without
a registry server); to prepare for this the server binary has moved to
cmd/server and the Nix build logic has been updated to wrap this
binary and set the required environment variables.
Change-Id: I9b4f49f47872ae76430463e2fcb8f68114070f72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5603
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This will be required for making a standalone, Nixery-style image
builder function usable from Nix.
Change-Id: I5e36348bd4c32d249d56f6628cd046916691319f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5601
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The camelCase variant of `rebuild-system` doesn't exist, but the
kebab-case version does.
Side note: this `lazy-dispatch` upgrade is pretty cool. TIL `direnv`
supports `watch_file` and `PATH_add`.
Change-Id: Idc9109a9b0de327ddf7b9c6a4368b7bebb551196
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5565
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: wpcarro <wpcarro@gmail.com>
nix-shell pollutes the environment with all sorts of variables. Let's
just add the tools to the PATH?
This also papers over the various differences in users `use_nix`
implementations by not using it at all.
Change-Id: If4282531fd6b7453b3611fe50217beacadc08bb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5524
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This modifies the envrc configuration to add the result of building
//tools/depot-deps to $PATH, instead of dispatching through the
manually maintained list of symlinks.
While at it, I've cleaned up some stuff from that list that is no
longer actually used.
Change-Id: If345c44da75b23c06b7c7f435be0cb02f99aaac5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5513
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
in some cases, users might want to pass through flags for
nix-build (such as `-j`).
magrathea now accepts these as arguments to `mg build`, as long as
they are separated by `--`.
the arguments passed to `mg build` are parsed into a proper record,
which enables us to show users very clear error messages in case they
forget to use the `--` separator and keeping us future-compatible with
more potential arguments to magrathea itself.
Change-Id: I81f5d9db52779a5cc3b8bbdd975316274fffe5fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5507
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: asmundo <asmundo@gmail.com>
