Commit graph

46 commits

Author SHA1 Message Date
Vincent Ambo
7f2f5d07f2 fix(ops/besadii): Pass Build.Author to Buildkite
Extracts author information from the flags passed by Gerrit and moves
them along to Buildkite. This should display the owners of builds
correctly in the UI, rather than marking everything as coming from me.

Change-Id: If9efe5553a13f0dbdb8bf3936c1d341ae5922318
2021-12-06 17:42:57 +03:00
Vincent Ambo
b679bb4034 refactor(ops/besadii): Get config from home directory by default
Slightly more ergonomic in some setups.

Change-Id: I565f2d242852ffd299ef5d5740a47520187dd4b4
2021-12-02 17:13:49 +03:00
Vincent Ambo
dcb2410982 refactor(ops/besadii): Generalise for use with non-TVL URLs
This makes it possible to use besadii for any TVL-ish setup using
Gerrit and Buildkite, with the same hook functionality as for TVL.

Change-Id: I1144b68d7ec01c4c8e34f7bee4da590f2ff8c53c
2021-12-02 13:10:21 +03:00
Vincent Ambo
e48ae26e8e feat(ops/besadii): Add other missing configuration keys
Adds configuration keys and rudimentary validation for all other
besadii settings that are currently hardcoded.

This adds the config options:

* repository: Name of the repository in Gerrit.
* branch: Name of the HEAD branch in the repository.
* gerritUrl: Base URL of the Gerrit instance
* gerritUser: Username of the Gerrit user
* gerritPassword: Password of the Gerrit user
* buildkiteOrg: Name of the Buildkite organisation
* buildkiteProject: Name of the pipeline inside the Buildkite
  organisation
* buildkiteToken: Auth token for Buildkite access

All of these configuration options are required.

Change-Id: Ie6b109de9cd8484a3773c6351d7fd140f39a49ed
2021-12-02 13:10:21 +03:00
Vincent Ambo
168114df52 refactor(ops/besadii): Move Sourcegraph config to a file
Initial step towards moving besadii away from hardcoded values and
onto config files. This is required because I want to reuse besadii
outside of the TVL context.

Change-Id: Id4fa7a49c5d4f876a02b202f04a421ab5ba0dcc4
2021-12-02 13:10:20 +03:00
Vincent Ambo
c1aab56a02 feat(besadii): Support invocation as different Gerrit hooks
Removes besadii support for the previously used 'ref-updated' hook and
instead introduces support for the 'change-merged' and
'patchset-created' hooks.

These hooks more accurately capture the semantics of when besadii
should trigger CI builds and using them will avoid problems such as
skipping 'canon' builds if chains of CLs are submitted together.

Change-Id: Ib90356c069780bf0c0250e56b927e46a5b31ce7f
2021-12-01 12:49:31 +03:00
Vincent Ambo
eca2bc572e refactor(besadii): Rename refUpdated -> buildTrigger
We are changing the Gerrit hooks which invoke besadii, but this
structure will be used for both kinds.

Change-Id: Idb1cb0c640d2c42db8e7af39f3ab372a97bfef91
2021-11-29 13:54:47 +00:00
Vincent Ambo
104f002a07 fix(ops/besadii): Trim whitespace of auth tokens
This is causing failures when trying to update Sourcegraph at least,
for good measure I've trimmed both.

Change-Id: I40266ee83b4e266ffe50f16bb365eb2e51952513
2021-11-28 10:34:36 +00:00
Vincent Ambo
98e4d4b18f feat(besadii): Link to started builds in CL comments
This makes it easier to click through to a build from Gerrit after
submitting a CL.

Change-Id: Ic5c6eeb81c87bc4ea23c5c5ca25704434b081fd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3572
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-18 14:28:26 +00:00
Vincent Ambo
ae93094ebf refactor(besadii): Extract logic for posting review comments
Currently besadii only posts comments when builds succeed, but it
might be very useful to also have a link to a build when the build is
started.

This just shuffles code around. The only functional change is that the
`labels` field in the review input is marked as `omitempty`, as this
will not be needed when posting the build start comment.

Change-Id: Id4a43fad8817c9a15da02f01ab2b781d48b46978
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3571
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-18 14:28:26 +00:00
Griffin Smith
330b7067a0 feat(besadii): Tag gerrit comments as autogenerated
This is a bit of an under-documented feature, but if the "tag" field for
a gerrit review starts with the string
"autogenerated:<something>~<something-else>", only the last comment per
instance of <something> will be shown by default on the CL page (with
the rest viewable by toggling the "Show all entries" switch). The idea
behind the "<something-else>" tag is to be used for the "type" of
comment within a particular system - gerrit's documentation gives the
example of one tag for "the build is running" and another for "the build
has finished, here's the result".

Change-Id: I9199a6ed97beca1b3a51ec5d6230c6c8358ba2b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3374
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 13:00:59 +00:00
Vincent Ambo
e09b44bdec chore(besadii): Stop passing explicit messages to Buildkite
Dropping the message field will make Buildkite use the commit messages
instead, which makes for much more readable build logs.

Change-Id: I1849f811632526893b700f117c9f6cf64888c329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2949
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-12 17:42:36 +00:00
Vincent Ambo
d34c527372 refactor: Replace some uses of builtins.toFile with pkgs.writeText
I'm looking at removing some of these because they can cause
unnecessary build steps during CI pipeline generation.

Change-Id: I84742968918090c050d2eedab8a1b42692632a42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2655
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-03-25 19:14:36 +00:00
sterni
e91d5e4e61 fix(config): remove ciBuilds inherit
The ciBuilds attribute seems to no longer exist and it breaks the
evaluation of the config attribute. It's only appearance was in
besadii which doesn't actually use the attribute.

Removing the ciBuilds inherit fixes these issues.

Change-Id: Ibbf3413ba6efe10ad868cf57cf0711d574860f97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2487
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-02-06 17:54:18 +00:00
Vincent Ambo
8f57ca92bd chore(3p|nix): Remove typed Go
Nobody has actually done any experimentation with typed Go, so we're
getting rid of it for now - it's causing annoying IFD during build
graph generation.

Change-Id: Ibac3dea98ebed1b3ee08acda184d24c500cf695d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2458
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-01-30 08:20:45 +00:00
Vincent Ambo
88317aea0d fix(besadii): Remove unused Nix code from the derivation
This was used previously when build granularity was besadii's task,
which it no longer is.

Change-Id: I6df2db1ed4730a7953199b7b48aa9ad916418b22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1853
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
f7687b1ea7 docs: Update README for the repository itself
Adds a lot more information about what is actually going on here, as
well as links to relevant things such as our Monorepo document.

I'm aware that I didn't make all `//...` links clickable, but I
realised at some point that it might be easier to just update cheddar
to support highlighting those natively :-)

Change-Id: Icbb212a6c07a5cf38ab8e65d83a64bec783eb8d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1768
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-17 21:52:58 +00:00
Luke Granger-Brown
1094e356df fix(besadii): Don't add people back to attention set.
This uses the not very documented ignore_default_attention_set_rules review API.

Change-Id: I650777bbbd24a1922f26967fbbd7da06d14b6782
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1513
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-02 01:18:37 +00:00
Vincent Ambo
10180e70ec chore(besadii): Stop adding Code-Review label on CLs
We now use the actual 'Verified' label instead of Code-Review from
Buildkite, this workaround is no longer required.

This reverts commit d3f9cb0ec3.

Change-Id: Ib8c1680eae844cb7b45bf8837acf2af03d4ed344
Reviewed-on: https://cl.tvl.fyi/c/depot/+/909
Reviewed-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-03 23:17:46 +00:00
Vincent Ambo
03076c3977 fix(besadii): Do not pass on update values for meta refs
Before this change, besadii would skip further processing of meta refs (which happen for every CL metadata change), but it would still schedule a build by returning an update - which would then inevitably fail.

This change makes besadii skip meta refs the same way it skips non-depot builds, i.e. completely.

Move *on* from meta refs, do *not* collect $100.

Change-Id: I269d2299f4d3cb1f9c041da8c92fa00ae7794b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/825
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: BuildkiteCI
Tested-by: BuildkiteCI
2020-07-01 16:51:50 +00:00
Vincent Ambo
0380841eb1 fix(besadii): Don't log errors for /meta refs
These are updated for all sorts of things and should just be silently
ignored by besadii.

Change-Id: I0a6de373b21d6bef5fd31d0a1d3f72c501073bba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/801
Reviewed-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-06-30 02:51:45 +00:00
Vincent Ambo
d3f9cb0ec3 feat(besadii): Temporarily add Code-Review labels on CLs
Besadii already adds 'Verified'-labels, which are used to signal CI
status on CLs, however we don't actually use these labels (yet) which
also means that they are not displayed in the Gerrit UI.

This change temporarily introduces the Code-Review label *in
addition* (with the same values as Verified), providing a build status
signal on the CL but without being required for submission.

Change-Id: I2c3a37c59aceb426815ad4e400c80ab85be482dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/781
Tested-by: BuildkiteCI
Reviewed-by: ericvolp12 <ericvolp12@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-29 19:06:37 +00:00
Vincent Ambo
6d3a9e7b5f feat(besadii): Implement support for Buildkite's post-command hook
This hook is invoked by Buildkite (on the runner) after every build
stage. This change adds support in Besadii to run as this hook and
update the build status on a Gerrit CL.

Change-Id: Ie07a94d9b41645a77681cf42f6969d218abf93c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/761
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-06-29 15:15:19 +00:00
Vincent Ambo
0bb24ed700 feat(besadii): Propagate Gerrit change ID & patchset to Buildkite
I previously implemented this in a CL that ended up being abandoned,
but it turns out we need it for the hook setup, anyways.

These environment variables become available during the build and,
crucially, to the post-build hooks.

Change-Id: Id6c1657947995e8bae1fa7b76184dd8be4c01525
Reviewed-on: https://cl.tvl.fyi/c/depot/+/739
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-06-29 01:19:51 +00:00
Vincent Ambo
15335e7b3d feat(besadii): Enable automatic builds for CLs
This expands builds to also be triggered for updates to CL refs.

The message displayed on Buildkite will contain a link back to the
CL (& patchset) from which the build was triggered.

Change-Id: Ib36dee454aeb11d623b89c78b384359ee7ea3477
Reviewed-on: https://cl.tvl.fyi/c/depot/+/708
Reviewed-by: ericvolp12 <ericvolp12@gmail.com>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-06-28 17:56:10 +00:00
Vincent Ambo
d3284112f6 refactor(besadii): Rename branchUpdate -> refUpdated
The name of the hook this type represents is 'refUpdated'. Since we're
adding support for additional hooks, it makes sense to to rename this
accordingly.

Change-Id: Ia568c85493813f5e754c77d0b993aaf246d3d595
Reviewed-on: https://cl.tvl.fyi/c/depot/+/667
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-28 17:20:22 +00:00
Vincent Ambo
1380d5998a feat(besadii): Trigger builds on Buildkite instead of Sourcehut
These builds run on runners that we control and disk space is (less
of) an issue there.

Change-Id: Id0a1436b2368418e447f6d5298ab474f829d4c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/628
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-27 18:02:04 +00:00
Vincent Ambo
6edf69da4b feat(besadii): Trigger separate builds for build target sets
In CL/570 we split up the build targets into different buckets, with
the idea that this should help us avoid the disk space issues on
Sourcehut.

This commit changes Besadii to read the list of target sets from a
file and trigger a separate build for each one of them.

Change-Id: If280fda5f40cd130c534c40911072e47c2d8f2be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/608
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-26 22:50:29 +00:00
Vincent Ambo
a37b4fbef3 feat(besadii): Add Sourcegraph index update triggers
Sourcegraph has a heuristic for determining when to update the
repository that doesn't work for the TVL repository, where commits are
often in irregular short bursts.

This changes besadii to trigger Sourcegraph index updates when invoked
by Gerrit.

Change-Id: Ifcfc25406d9e25bbdc93e79c23608ce4c6b10ba8
2020-06-19 01:35:11 +01:00
Vincent Ambo
2183cfe5de chore(besadii): Make besadii a Go2 project
Change-Id: Id5aef73b131252fb9c880dd95fc72bfe083f1b75
2020-06-18 02:33:23 +01:00
Vincent Ambo
b094e65bfc feat(besadii): Adapt into a Gerrit ref-updated-hook
Besadii was previously invoked as a git post-update hook, but Gerrit
does not use these hooks and instead has its own concept of hooks.

This change adapts besadii to be compatible with the way Gerrit hooks
are invoked (arguments being passed as flags, rather than via stdin).

Change-Id: I487b3a9e15810583bc5442fdc024ee2771c580cb
2020-06-13 06:03:45 +01:00
Vincent Ambo
37bbc43146 feat(camden): Move cgit to code.tvl.fyi
Moves the host at which cgit is served to 'code.tvl.fyi'.

Also updates related projects that link to this, most importantly:

* Hound's & Gerrit's cgit link bases have been updated
* besadii is updated to request CI builds for the new location

Change-Id: I44e3e584010ac29cc913ebb1a197c996eb024d80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/71
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-12 01:14:21 +00:00
Vincent Ambo
5faa737ead fix(ops/besadii): Remove branch tag after checking for it
... oops. The problem with working on besadii is that testing it
always lags one commit behind.
2020-02-21 23:35:42 +00:00
Vincent Ambo
a2bb8a7e1d fix(ops/besadii): Do not trigger builds for deleted branches 2020-02-21 23:30:05 +00:00
Vincent Ambo
28560fcf8a fix(ops/besadii): Include branch name in build note 2020-02-21 23:27:41 +00:00
Vincent Ambo
ca7c8fe9f0 fix(ops/besadii): Clone from git.tazj.in, not git.camden.* 2020-02-21 23:20:22 +00:00
Vincent Ambo
5ed68f0f6b fix(ops/besadii): Only trigger builds for branches 2020-02-21 23:16:28 +00:00
Vincent Ambo
8fe90430ee chore(ops/besadii): Pin git version used in besadii 2020-02-21 23:13:53 +00:00
Vincent Ambo
c689df0dc7 fix(ops/besadii): Replace slashes in branch names
Submitting a build with a branch containing a slash (which is common
for my branches) returns this error:

    Invalid tag name, tags must use lowercase alphanumeric characters,
    underscores, dashes, or dots

This commit replaces all slashes with underscores to work around that.
2020-02-21 23:06:19 +00:00
Vincent Ambo
21b76cb023 feat(ops/besadii): Run 'git update-server-info' at startup
Since besadii is effectively the entire post-receive hook, it also
needs to do the entire job of the hook.
2020-02-21 22:58:34 +00:00
Vincent Ambo
8377fd48f5 fix(ops/besadii): Send auth token in correct format 2020-02-21 22:51:40 +00:00
Vincent Ambo
59d02771b5 refactor(ops/besadii): Log to syslog instead of stdout 2020-02-21 22:46:34 +00:00
Vincent Ambo
dcbe3d1f9b feat(ops/besadii): Use post-receive hook input to trigger builds
Parses the input passed to besadii from git to extract ref updates and
trigger builds.
2020-02-21 22:32:23 +00:00
Vincent Ambo
5058f3928a feat(ops/besadii): Read sourcehut token from secrets file on disk 2020-02-21 22:31:57 +00:00
Vincent Ambo
0a34810e27 chore(ops/besadii): Fail if sourcehut token is unset 2020-02-21 22:09:23 +00:00
Vincent Ambo
80c6680eda feat(ops/besadii): Refactored tool to trigger sourcehut builds
Refactors //ops/sync-gcsr which was previously responsible for
synchronising the git repository between GCSR and the git.tazj.in cgit
instance to simply be responsible for triggering builds on sourcehut.

This program is intended to run as a git post-update hook.

Note: Not yet feature complete, as interpolation of concrete git
values and also sourcehut secrets is missing.
2020-02-21 22:05:43 +00:00