Commit graph

594 commits

Author SHA1 Message Date
Vincent Ambo
eeca6cd891 feat(emacs-pkgs/treecrumbs): support "JSON-style" YAML syntax
Makes crumbs work correctly for weird stuff like:

```yaml
foo:
  - { "bar": [ { baz: 1 }]}
```

Would be nice to have tests for this at some point.

Change-Id: I034e83318435404ef8613e439313dbb08865f228
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11105
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-03-10 19:27:49 +00:00
Vincent Ambo
13cced5b84 feat(tools/emacs-pkgs): configure FSF_OWNERS file
This adds a separate owners file that can be imported in projects with
FSF copyright assignment, so that only people with copyright paperwork
on file can commit (or verify commits before approving).

Change-Id: Ifbe07792572b9d6bf5f7d47c41135892bddea46b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11108
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2024-03-10 19:27:49 +00:00
Vincent Ambo
0be7b938c2 feat(emacs-pkgs/treecrumbs): tree-sitter based breadcrumbs
Adds a new 'treecrumbs' Emacs package that allows displaying fast,
tree-sitter based breadcrumbs. The initial version only supports
YAML (which is what I needed this for!).

The package is documented, please go read the code and comments for
more information.

I assign copyright over this package to the FSF, paperwork is on file.
This means that I can only accept contributions from people with the
right paperwork. If this package ends up being useful, I intend to
eventually submit it to GNU Emacs.

Change-Id: If1297a080f63f402f83b6cc57fa7c970d32d3695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11104
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2024-03-10 19:27:49 +00:00
Vincent Ambo
a412791752 refactor(nixery): expose launch script derivation
Simplifies reusing the launch script in other use-cases than the
"official" Nixery image.

Relates to nixery#166

Change-Id: Iaf1dff385ce270792253551081c1b2fca6400037
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11046
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-28 20:21:04 +00:00
Vincent Ambo
bc06e4d99c fix(nixery): use set-default for setting WEB_DIR
Makes it possible for users to still override this using an envvar.

Relates to nixery#166

Change-Id: Ief2925e03cf2e4351bc38554bf553c8ee259f1f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11045
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-28 20:20:30 +00:00
Markus Rudy
c461595b7f fix(nixery): strictly adhere to OCI image spec
nixery.dev uses the vnd.docker.container.image.v1 format, which is
recognized by the OCI [1] and originally defined by Docker [2]. The
config field in this image format, which this commit is about, is
even portable between the Docker and OCI formats (the Docker Golang
library embeds the OCI definition [3]).

The attribute names in what's called ImageConfig in [3] are specified as
PascalCase, which effectively means that the names Env and Cmd used by
nixery need to be capitalized. The lowercase variant is not causing a
lot of issues because most container tooling is written in Golang, which
allows case-insensitive matches when deserializing JSON. Languages that
parse strictly either miss the configuration values, or fail due to
unknown attributes. This commit capitalizes Cmd and Env to accomodate
strict parsers.

[1]: https://github.com/opencontainers/image-spec/blob/365fa41/media-types.md?plain=1#L70
[2]: https://github.com/moby/moby/blob/v20.10.8/image/spec/v1.2.md#image-json-description
[3]: https://github.com/opencontainers/image-spec/blob/365fa41/specs-go/v1/config.go#L24

Change-Id: Ibee597a64d36c008dea83a3b7a0d8e59b8287d0d
Signed-off-by: Markus Rudy <webmaster@burgerdev.de>
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11012
Autosubmit: lukegb <lukegb@tvl.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2024-02-22 16:56:02 +00:00
Florian Klink
4497ac41ab refactor(tools): move crate2nix generate here
Having something running the depot crate2nix and formatting it with
depotfmt is useful outside of tvix too.

Change-Id: Iecc8f207da38cc6995747c5ea48d3911433fd416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10658
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-18 15:02:15 +00:00
Luke Granger-Brown
e20848ecf1 chore(depot): update OWNERS files for aspen
Change-Id: Id94b646a6ea035782298c421d6667530da6fc5b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10384
Tested-by: BuildkiteCI
Owners-Override: lukegb <lukegb@tvl.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2023-12-20 18:35:58 +00:00
Aspen Smith
02c6621813 feat(tools/magrathea): Allow running commands in a shell
Add support for running a command inside a `mg shell`, specified as an
extra argument after the target to the shell command

Change-Id: Icbbd9cf4e1f099fcd7e6b13655b8447775a236d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10247
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 11:44:46 +00:00
Vincent Ambo
47e24e6e8e chore(tools/cheddar): bump cargo dependencies
Fixes:

* RUSTSEC-2023-0018

Change-Id: If4b5ea9edacc6f1e8664387e96e7abc24618b1a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10016
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-11-12 22:42:04 +00:00
Evgeny Zemtsov
e3fe6826f8 fix(tools/magrathea): isolate from environment packages
This makes magrathea use the repostory version of nix and git. This is
done in the pursuit of enforcing guaranteed unified experience of
magrathea tool across all users of a `tvl-kit`-based
repository. Especially among ubuntu users with uncontrolled set of
packages and versions installed on their system.

Not having this was giving build problems for `mg build` as one of the
users has 2.17 version of nix that had inconsistent hash computation
with 2.3.

Change-Id: I3182faf4c545ac61f6cc1cc862dc23d51c1cd397
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9892
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-03 13:54:36 +00:00
Vincent Ambo
67fed3d76d chore(tools/depotfmt): use Go version from buildGo
This is required because Go 1.18 is actually being deleted. I've
applied the formatting breakage that it introduces (such as breaking
comment formatting), because I can't be bothered to try and work
around broken Go stuff.

Change-Id: Ica7cee0d01228845d6a766079fef36df99a3da96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9832
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-30 16:33:21 +00:00
guangwu
3e5279aeff docs(nixery): occurrences typo
Change-Id: I3798e1c23d6b0580b99b14bb4aae1c7cfc81fb6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9366
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-09-19 08:58:13 +00:00
Vincent Ambo
6e1ea80e5b docs(REVIEWS): point people to signup.tvl.fyi for account creation
Rather than invoking the old shell script, people can now use the web
UI to create the entry.

Change-Id: Ic1b065d219fde9977bb1d8ee59ce7ac5f27e2e29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9306
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-12 13:59:34 +00:00
Vincent Ambo
87d63e4a1b feat(tools/cheddar): allow disabling tagfilter extension
Makes it possible to do things like embedding YouTube videos in blog
posts rendered through Cheddar.

Change-Id: I6aed943c7bec0167b9f009d36dd067c52c6d3083
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9275
Tested-by: BuildkiteCI
Reviewed-by: Mark Shevchenko <markshevchenko@gmail.com>
2023-09-08 13:51:26 +00:00
Vincent Ambo
6d5d96ecea fix(emacs-pkgs/term-switcher): apply tramp workaround to first vterm
Change-Id: Idc01001856cb01104c20b62dc4221c0ad2ab7a3f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9153
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-08-25 15:34:04 +00:00
Vincent Ambo
561a9fa45b refactor(emacs-pkgs/term-switcher): un-dash term-switcher.el
The builtin `seq' has everything I need, and this way bpalmer will be
less annoyed.

Change-Id: Ic8e5ac07d5214f36d77e9b577a3f805cdf89f220
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9146
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-08-23 20:22:25 +00:00
Vincent Ambo
4b4ec86114 fix(emacs-pkgs/term-switcher): switch buffers by object, not name
The terminal switcher uses ivy to select buffers from a list of
buffer *names*, however this can cause weird situations if, for
example, two `vterm` sessions are in the same folder and buffer name
uniquification is active.

This commit implements a corrected solution, which constructs an
association list of buffer names to their actual buffer object, and
retrieves the buffer object from that list after the user has made
their selection. This way, changes in buffer names during terminal
selection do not lead to confusing results.

Change-Id: I3ab3d6b715b32606cf771dabc31d9d4507c8b856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9145
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-08-23 20:22:25 +00:00
sterni
ffc1fb8f5c chore(3p/sources): Bump channels & overlays
* //users/grfn: vendor ddclient module and package into depot

  //3p/ddclient now contains the removed package expression and
  NixOS module with the following changes:

  - Include former uid/gid settings from ids.nix which have been removed
    by upstream with the ddclient module.

  - Rename to deprecate-ddclient, since it is impossible at the moment
    to prevent the corresponding mkRemovedOptionModule from being
    imported (https://github.com/NixOS/nixpkgs/issues/245265).
    I wrote a patch for nixpkgs that would at least allow individual
    mkRemovedOptionModule to be disable, but it is stuck for now:
    https://github.com/NixOS/nixpkgs/pull/245274.

* //tools/magrathea:

  We need to pass -host to csc due to
  https://github.com/NixOS/nixpkgs/pull/246923 now. I don't fully grasp
  what this means, but it works and we are hardly cross-compiling, so it
  should be fine until I can get some answers from the change author.

* //3p/nixpkgs:nixos-option: provide latest Nix as input

  https://github.com/NixOS/nixpkgs/pull/237442 adapted nixos-option to
  API changes in Nix's libraries which means it needs to be built
  against Nix 2.15, not 2.3. Let's hope it stays up to date with the
  latest Nix version in the future, so we can keep this override as is.

  Sadly this means that machines in depot will depend on two
  versions of Nix going forward.

* //3p/nixpkgs:tdlib: update to match emacs-overlay

Change-Id: Iac4dba58a076ecf25e8647fd9a06cbabf2f7809e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9004
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-08-08 20:33:01 +00:00
Vincent Ambo
fa8288823b fix(emacs-pkgs/term-switcher): don't open vterm over tramp
vterm doesn't really seem to support this, but it also doesn't stop
the user from doing it, resulting in weirdly broken terminals.

Change-Id: I5251306a2fb72e7aae4e6c70e522bad240222301
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9017
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2023-08-08 10:52:24 +00:00
sterni
a72e67c8af feat(tools/git-r): git subcommand to display r/numbers for commits
Sadly, this can't quite be an alias (which would be difficult to
automatically set up anyways), since we want to check if an r/number is
part of the (upstream) canon branch.

The test script for the subcommand doubles up as a soundness check for
our pipelines ref creation.

Change-Id: I840af6556e50187c69490668bd8a18dd7dc25a86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8844
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
2023-07-05 12:37:09 +00:00
sterni
c3628af8cc refactor(tools/magrathea): introduce read-chomping helper function
Change-Id: I2ee6903686fd210755c40eb9555c938e8c1ab52b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8843
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-23 16:47:06 +00:00
sterni
227dc9421f fix(tools/magrathea): allow specifying just a target for mg run
Resolves b/232.

Change-Id: I4a32e4fe90c819e3fc98da35d370c84b2f3d3722
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8842
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-23 16:47:06 +00:00
Vincent Ambo
2464ea7303 fix(nixery): allow references to packages starting with numbers
These packages are invalid in Nix, and worked around in nixpkgs with
underscores, but the underscores are invalid in the Docker registry
protocol.

We work around this by detecting this case and adding the underscore
to yield the correct package reference. There is no case where this
workaround can break something, as there can be no valid package
matching the regular expression.

This relates to https://github.com/tazjin/nixery/issues/158

Change-Id: I7990cdb534a8e86c2ceee2c589a2636af70a4a03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8531
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-04-29 11:49:02 +00:00
sterni
0cb2057a76 feat(emacs-pkgs/tvl): allow starting sly only including dependencies
Usually the current behavior is best: You are dropped in a REPL with the
package(s) you are working on already available. As you are working on
them, you recompile individual files and your changes become available.

However, I've found that there are some occasions when this is not
desireable, e.g.: When you are working on something and have broken the
test suite intermittently, it becomes impossible to start a new REPL.

Not sure how the yes-or-no-p question should be phrased, its negation
may be better?

Change-Id: I6a37ebc02f3121f628fc9206e0de650851824cd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8415
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-07 09:32:38 +00:00
Florian Klink
e9686f84d9 fix(views/kit): communicate :unsign in the tvl-kit URL directly
Instead of prepending :unsign to all URLs in josh-proxy, and for all
calls to filteredGitPush, explicitly use it only in the filter we use
for the `export-kit` extraStep.

This means, people cloning tvl-kit via

> https://code.tvl.fyi/depot.git:workspace=views/kit.git

now need to update the URL to point to

> https://code.tvl.fyi/depot.git:unsign:workspace=views/kit.git

instead.

git@github.com:tvlfyi/kit.git will keep the same hashes, as it's updated
to export the unsigned workspace view of it.

This is less invasive than dooming every josh workspace to have to strip
signatures.

Change-Id: I6de05182fad4c3695081388c3bbf37306521d255
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8369
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-31 08:46:01 +00:00
Florian Klink
e8fd2aabb3 fix(tools/releases): add :unsign to filteredGitPush
We need to followup on cl/8186, and use the same workaround for josh-
filter that we use for josh-proxy, so we can push again.

This could potentially become an optional function argument, if new
subtrees are to be carved out, and we want to keep signatures on commits
in there.

Change-Id: I7a9b821a7365767e8a4188e8200fa3c37463142c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8368
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: flokli <flokli@flokli.de>
2023-03-30 14:28:39 +00:00
Vincent Ambo
bdf93dcefe fix(nixery): fix link to nixery logo
Change-Id: Ib78659b971696feaff579bc0a31df7d8ee24e459
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8034
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-06 17:36:54 +00:00
Vincent Ambo
f04829a1bb chore(tools/cheddar): bump to syntect 5.0
Upgrade to syntect 5.0 and load the new kind of syntax set
serialisation with the new helper function for that purpose.

Includes other minor API fixes as well, note that the things that are
now calling `expect` previously failed internally at those points and
we're reasonably confident they don't fail in production.

This has been waiting for a long time ...

Change-Id: I8af4fef995ff64bfbe24e1f13917fa50ecb6e4ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7787
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-01-07 08:02:37 +00:00
Vincent Ambo
0efe78b3a7 chore(tools/cheddar): bump dependencies
This is a semi-manual bump of all dependencies (except syntect, which
is more complex to update).

Change-Id: I8c678a16d779f3f896b95f7d161710ac39d38e88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7786
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-07 08:02:37 +00:00
Vincent Ambo
ee7a724b60 fix(ops/pipelines): explicitly set contexts for annotations
I think what might be going on with b/231 is that the annotations
somehow started conflicting because they don't have contexts set.

Lets try setting a context and see if it changs anything ...

Change-Id: I62ed57f9e24f08e4e7215f05d35cfa769e2e2c24
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7640
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 16:35:20 +00:00
Vincent Ambo
fe4cdff442 feat(tools/fetch-depot-inbox): wrapper script to fetch depot maildir
This script fetches the inbox for depot@tvl.su into the specified
directory in maildir format.

The layout of the folder follows the structure generated by public
inbox, i.e. the directory containing the current maildir will be
`$TARGET/su.tvl.depot.0`, but most mail clients (e.g notmuch) will
figure this out on their own.

----

In addition, we would ideally find a CLI mail client that can be
pointed at an arbitrary maildir (or an IMAP server) and works with
local `sendmail` config so that people can have a single command entry
point to interacting with depot@tvl.su.

Change-Id: Iaf9fcce73e9caa2f202327488c43d0394be26ca6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7644
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-12-28 08:17:45 +00:00
Vincent Ambo
81fd9caf3e docs: change email address mentions to depot@tvl.su
This is the new address which leads to the public inbox at inbox.tvl.su

Change-Id: I45d98a373b8acda49b05c4f74669ffb9ad1f1a3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7632
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-12-27 19:46:11 +00:00
Vincent Ambo
c6cb138565 chore(tools): remove depot-scanner & tvlc
These are both unused things from a long time ago, which we don't need
to keep around anymore.

Their design doc has been marked as archived.

Change-Id: Icd2744e511e78ec95ec8f39e5f79ed1fe98e9e4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7639
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-12-26 11:16:39 +00:00
Florian Klink
4714f8b939 feat(tools/depot-deps): drop crate2nix
The //tvix README already steers to `mg run //tvix:crate2nixGenerate
--`, there's no point in /also/ having a non-formatting version of
crate2nix in ``$PATH`.

Change-Id: Idc6409799ae5f0629376eef6eeff6eb9eaa4fb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7613
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-22 15:26:41 +00:00
Vincent Ambo
dcc6d5db63 chore(tools/depot-deps): add crate2nix
As we start using this in more parts of depot, it makes sense to add
it to the available tools.

Change-Id: I148902714167b36bc51aeca4a241c79ad8a59285
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7562
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-16 12:00:35 +00:00
William Carroll
1ea97c288e fix(tools/mg): Support applying args to mg run (empty-target)
With this, we can disambiguate the following `mg run` invocations:

```shell
$ mg run :foo    # run the virtual target, foo, with no args
$ mg run -- :foo # run the empty target and pass the arg, :foo, to it
```

Change-Id: Id6395b36a4d8ef3f325937e322e1c27b8630b556
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7408
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-11-26 15:20:24 +00:00
sterni
982022826d fix(tools/magrathea): pass through nix-build exit status
Something I missed last time reading through the process documentation
is that you can use a combination of `process` and `process-wait` to
determine the exit status of a child process *and* read from its
standard output. With `process*` we could even capture stderr, but we
probably want it mounted to the parent process' stderr anyways.

Change-Id: I9840f607df465caa80d28109e344e5fc1402949d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7259
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-20 20:33:24 +00:00
sterni
8699370fae chore(tools/rust-crates-advisory): move custom checker to user dir
Profpatsch originally implemented an advisory checker from scratch in
Rust. We now ended up just using cargo-audit for the global checks
exposed via CI and the custom implementation is unused. To clean up
//tools/rust-crates-advisory a bit, we can move the unused parts to his
user directory.

Change-Id: Iacbd27c163edd07c804220fd1b3569c23aebd3e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7171
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-11-06 18:40:59 +00:00
sterni
bf18e65719 refactor(rust-crates-advisory): redo the buildkite report in bash
I've elected to split the check-all-our-lock-files script into two new
scripts: One very simple script which generates the report by invoking
lock-file-report on the fake lock file for //third_party/rust-crates and
all lock files in depot, and one which executes this and adds it as a
buildkite annotation if there are any warnings (which is reported by the
report generating script using a non zero exit code).

The latter script could become the basis for generalizing buildkite
annotations, a slight attempt at making it easily reusable in the future
has been made. So far we expect a report generating script to exit non
zero if a report should be made and to print commonmark to stdout. In
the future we may want to use a JSON format for generating the report,
allowing us to filter it by buildkite target (using the drvmap to
exclude certain reports, potentially).

Change-Id: I1df9e440509d69adff5b8e6304105a45dc62c018
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5260
Reviewed-by: kn <klemens@posteo.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-04 22:06:53 +00:00
sterni
ffec3c70f4 refactor(rust-crates-advisory): redo tree-lock-file-report in bash
I think migrating the execline scripts over to bash makes sense:

1. Ever since nixpkgs-fmt, execline scripts in depot have become a huge
   pain to write and edit and I can't think of a satisfying solution to
   this problem.

2. The scripts here require remembering things across loop cycles (i. e.
   the status variable) which is not possible in pure execline. As a a
   workaround we used to read the entire report into memory first and
   check if it was empty (tying us to the argv limit for the report
   length).

Change-Id: I954b08b982ef947f9014a685676d2b83a2aec4d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5259
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-11-04 21:29:09 +00:00
sterni
57cf952ea9 chore(3p/sources): Bump channels & overlays (OpenSSL edition)
* //ops/machines/whitby: Disable grafana, since the grafana module was
  changed upstream in a way that our configuration no longer works.
  Since the OpenSSL security update is relatively pressing, adapting the
  grafana configuration beforehand is not a hard requirement. See
  https://github.com/NixOS/nixpkgs/pull/191768.

* //tools/depotfmt: keep Go at version 1.18 to forgo a reformat of the
  tree.

* //nix/buildGo: keep Go at version 1.18, as 1.19 changed the CLI
  interface (?) in a way that breaks buildGo.

* //3p/overlays/tvl: drop upstreamed tdlib upgrade.

* //3p/overlays/tvl: patch buf to work around breakage due to git 2.38.1

TODO items for Go are tracked in b/215.

Change-Id: Ie08fef49cf3db12e6b5225a8b992a990ddc5b642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7141
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-03 15:10:39 +00:00
Florian Klink
6025242fc7 chore(buf): Use nixpkgs-provided buf
The version of buf used is quite old.

nixpkgs provides a more recent version, but it requires us to migrate
config to the latest version.

depot_scanner.proto doesn't honor some of the conventions, so we need
allow_comment_ignores and drop a bunch of comments in there.

Change-Id: Ic978fe92fb7c8471f58c137497528f18aad8f3ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7053
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
2022-10-21 18:39:03 +00:00
Vincent Ambo
7c99e9e8e3 docs(nixery): replace the Nixery mdBook with a simple web page
Nixery's previous landing page was an mdBook that was basically
unmaintained and full of incorrect information. It also duplicated
some things (like nix-1p) which actually live elsewhere.

This commit removes the mdBook completely and reduces it down to a
simple TVL-style landing page. The landing page has been checked in
in its entirety because Nixery is frequently cloned through josh
without the entirety of depot, however the page has been created by
building it through depot's //web/tvl/template.

See also https://github.com/tazjin/nixery/issues/156

Change-Id: I20e1d58f1e6608377207e80345c169f7d92d3847
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6930
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-10-12 10:58:03 +00:00
Vincent Ambo
c9e4d9c06b chore(nixery): use ldflags parameter instead of buildFlagsArray
The latter has been deprecated in nixpkgs.

Relates to b/200

Change-Id: I42871ce3eb54ebf092909f033b43936b9610d982
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6836
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-02 13:39:28 +00:00
sterni
0c178a0ef6 chore(3p/sources): Bump channels & overlays
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.

* runCommandNoCC -> runCommand
* gmailieer -> lieer
  We also need to work around the fact that home-manager hasn't catched
  on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
  This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy

Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.

Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-09-28 08:02:31 +00:00
talyz
5b165e7318 fix(nixery): Set correct depot ref when fetching nix-1p
Change-Id: Iffa49a4e8fd38d0762ed1f60bf72b9a050594a3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6697
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-19 11:28:10 +00:00
talyz
02b6b6c564 fix(nixery): Discard string context before parsing with fromJSON
Discard string context in prepare-image.nix before parsing input read
with readFile with fromJSON. Required for compatibility with nix >2.3.

Change-Id: I3830707e80fd19a700551a15f1a96d2841d0b022
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6696
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-19 11:28:10 +00:00
talyz
28417afbb4 fix(nixery): Avoid race when the same image is fetched in parallel
Remove a race condition which appears when uploadHashLayer is called
with the same key from multiple threads simultaneously. This can
easily happen when the same image path is requested by multiple
clients at the same time. When it does, a 500 status is returned and
the following error message is logged:

{
  "context": {
    "filePath": "github.com/google/nixery/builder/builder.go",
    "lineNumber": 440,
    "functionName": "github.com/google/nixery/builder.uploadHashLayer"
  },
  "error": "rename /var/lib/nixery/staging/<hash> /var/lib/nixery/layers/<hash>: no such file or directory",
  "eventTime": "...",
  "layer": "<hash>",
  "message": "failed to move layer from staging",
  ...
}

To solve this issue, introduce a mutex keyed on the uploaded hash and
move all layer caching into uploadHashLayer. This could additionally
provide a small performance benefit when an already built image is
requested and NIXERY_PKGS_PATH is set, since symlink layers and config
layers are now also cached.

Change-Id: I50788a7ec7940cb5e5760f244692e361019a9bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6695
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-19 11:28:10 +00:00
Luke Granger-Brown
f190712b7f chore(gerrit): migrate OWNERS files to code-owners style
Change-Id: Iacc521dfdd4b4a2d5cef3920cf8189bcce35a488
2022-09-19 11:13:28 +00:00