Commit graph

285 commits

Author SHA1 Message Date
sterni
d5b6704d3d chore: move protoCheck into extraStep of //nix/bufCheck
Resolves b/385. I have a feeling bufCheck should be moved to
//tools as well.

Change-Id: I2a2b63d135a2f8bcc982aa1138ff3213c6012f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11152
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-15 11:55:54 +00:00
sterni
16a7c4a1be feat(buildkite): avoid building extraSteps in pipeline construction
In principle we don't want to build any (later) pipeline target during
pipeline evaluation insofar they appear in extraSteps. For this reason,
we have the needsOutput mechanism which prevents the parent target of an
extraStep from being built in 🦙.

Unfortunately, this mechanism is not general purpose enough, as we use
other (i.e. non parent) targets from depot in extraSteps. As a
consequence, kind of expensive builds need to happen during pipeline
construction at the moment. The solution is to use the fact that the
command script we want to run is exposed via the readTree interface to
depot and build the script proper only when the extra step is executed.

To facilitate this, some prerequisite changes need to be made:

- We need to use a symlink different to result in case needsOutput is
  true which needs support in mkBuildCommand. We also need to avoid this
  symlink being picked up by git, as many extra steps check whether the
  tree is dirty or not. (Is there a way to have it outside the depot
  tree?)

- Since we rely on the build command printing a single store path we
  store in $command_script, we need to avoid it printing two paths
  in cases where nix-store(1) is used (nix-store(1) prints the symlink
  and readlink(1) would print the store path in a separate line).

Future work would be to remove/deprecate the needsOutput mechanism:
After this change the parent target wouldn't be built right away even if
it appeared in the script via string interpolation. Thus we could,
instead of expecting the target being available as `./result`, make our
extra steps nix-ier.

Change-Id: Idd2e88a865eadabe229ce1e05406e8cc4cb63f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10850
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-03-07 15:39:56 +00:00
sterni
1608f935aa refactor(nix/buildkite): unify target attr path calculation
After the previous refactor that made mkBuildExpr use attribute paths,
mkStep and mkDrvMap can now share the code for calculating the attribute
path of any given target.

Change-Id: I50f9687e85c3b4407e4a55ff7e645db882bb80d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10849
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-03-02 23:39:50 +00:00
sterni
206742949b refactor(nix/buildkite): use attr path over target drv in mkBuild*
mkBuild* is independent of the target abstraction, we can just use
attribute paths. That is useful because we'll want to reuse that code in
order to avoid building derivations (apart from unavoidable IfD) during
pipeline construction for extraSteps.

Breaking Change for mkBuildExpr and mkBuildCommand.

Change-Id: I03646310192087d3e50f358a714472d1ac1a652f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10848
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-02 23:32:38 +00:00
sterni
aa14e36a91 refactor(nix/buildkite): make more use of formal arguments
When changing the buildkite pipeline code I found that some functions
have a lot of arguments where the order is not necessarily clear. To
ease further refactors / new features, I've ported them over to taking
attribute sets.

Note that this technically is a breaking change, as these functions are
all exposed. Not sure how often they'd be called from the outside,
though.

Change-Id: I118c8c5242922403d12f6e5a61beaf68f636b40a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10847
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-02 23:01:05 +00:00
sterni
77a6eb4f51 feat(nix/writeTree): don't require IfD for drvs in tree
As far as I can tell we can handle files and directories using the same
cp(1) invocation, so we no longer need to potentially IfD derivations in
the tree to figure out whether they are files or directories.

Change-Id: Iabe648c30a747fa42768558715e388552024764a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10996
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-02-21 20:52:20 +00:00
Aspen Smith
944483ef5e feat(nix/writeTree): init
Add //nix/writeTree, a function to make a derivation to build a
directory structure from a Nix attribute set.

Co-authored-by: sterni <sternenseemann@systemli.org>
Change-Id: I9c0fc91611a55a20ad33de6f2b27abde4b6abd21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10963
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: aspen <root@gws.fyi>
2024-02-21 20:52:20 +00:00
Luke Granger-Brown
0c6813ccf8 chore(nix/buildGo): add support for Go 1.20+
https://github.com/golang/go/issues/51225 and other changes mean that importcfgs are now basically required for Go 1.20+; we also separately compile the Go stdlib, since it looks like pkgs.go no longer actually has the compiled version of the stdlib shipped, just the source.

Change-Id: Ibf5ee7d43f7800c6dd1e0dec6c7a6d35ef50b7b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10801
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-02-12 15:58:58 +00:00
Vincent Ambo
8ebfc02fb6 docs(nix-1p): clarify wording for what is an expression
See https://github.com/tazjin/nix-1p/issues/19

Change-Id: Ie78eb00192b595ffa59040a4f6d311023621939a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10737
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2024-02-02 16:39:44 +00:00
Aspen Smith
1481066cb7 fix(nix/readTree): Fix typo in docs
Change-Id: I65d6d1d99813d849a02369a0693893ced4681e7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10169
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-11-30 14:40:44 +00:00
Vincent Ambo
67fed3d76d chore(tools/depotfmt): use Go version from buildGo
This is required because Go 1.18 is actually being deleted. I've
applied the formatting breakage that it introduces (such as breaking
comment formatting), because I can't be bothered to try and work
around broken Go stuff.

Change-Id: Ica7cee0d01228845d6a766079fef36df99a3da96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9832
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-30 16:33:21 +00:00
Florian Klink
5c91ddbeef chore(nix/buildGo): bump to go 1.19
Reading from lukegb's tea leaves.

Change-Id: I5705fcb3212943e62de6e74cf8460171383121d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9796
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-24 20:36:39 +00:00
Vincent Ambo
ba9a62ca9f feat(nix/buildkite): fetch drvmap from *any* default branch builds
With this change, the query we make to Buildkite changes from
explicitly querying for a specific number of *ancestor* builds, to
any latest builds of the default branch that have a drvmap.

This is not really supported by Buildkite, it seems, and the query to
do it feels very wonky and requires a lot of fiddling with `jq` to get
the output into the right shape.

We lose the information about which build we downloaded this from in
the output. Adding that information back would make the `jq` query
much more complex.

Change-Id: I9e7cecdffa9ac09f9e0339eb24d98c0e8dd82292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9168
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2023-10-22 12:29:52 +00:00
Florian Klink
c4ea205387 chore(nix/bufCheck): remove git status check
buf lint already happens in other individual targets, and as no
generation happens in here anymore either, there's no need to check for
git status to record changes anymore.

Change-Id: Ieb8fc4760a61dd0f0e03f0ec388062dd2303c37a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9789
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 19:51:24 +00:00
Florian Klink
1b26bf21e3 chore(tvix): move store golang bindings to tvix/store-go
Similar to the castore-go CL before, this also updates the store-go
bindings to the new layout.

Change-Id: Id73d7ad43f7d70171ab021728e303300c5db71f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9788
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 19:51:24 +00:00
Florian Klink
e38733a955 chore(tvix): move castore golang bindings to tvix/castore-go
Have `tvix/castore/protos` only contain the protos, no go noise.

Make the `.pb.go` file generation a pure Nix build
at `//tvix/castore/protos:go-bindings`, and have a script at
`//tvix:castore-go-generate` (TBD) that copies the results to
`tvix/castore-go`.

`//tvix:castore-go`, with sources in `tvix/castore-go` now contains the
tooling around the generated bindings, and the generated bindings
themselves (So go mod replace workflows still work).

An additional CI step is added from there to ensure idempotenty of
the .pb.go files.

The code.tvl.fyi webserver config is updated to the new source code
path. I'm still unsure if we want to also update the go.mod name. While
being a backwards-incompatible change, it'll probbaly make it easier
where to find these files, and the amount of external consumers is still
low enough.

Part of b/323.

Change-Id: I2edadd118c22ec08e57c693f6cc2ef3261c62489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-17 19:51:24 +00:00
Vincent Ambo
da648b4707 docs(nix-1p): add section about the merge operator
Without the examples, some behaviour of the merge operator might not
be clear from the previous description. Due to how pervasively the
operator is used, I think it warrants a separate section.

This fixes https://github.com/tazjin/nix-1p/issues/16

Change-Id: Iecba5f1cb749bef0a4987b3fc5642832a92c18d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9599
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 11:18:25 +00:00
Vincent Ambo
0ad059ddfe docs(nix-1p): fix formatting of operator table
Change-Id: I19a10114bf4a816e3573eb86df208b68548884a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9598
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 11:18:25 +00:00
Jan Hrček
4ab0a9498f docs(nix-1p): update manual links
I found some links to nix manual are not working (they redirect me to
manual homepage and one github link led to 404 page), so I fixed all
the links for which I was able to find appropriate target.

Imported from https://github.com/tazjin/nix-1p/pull/15

Change-Id: Ie933951e9743649d7794a6bd2a82c3a0ed328336
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9501
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-01 14:19:38 +00:00
Florian Klink
32f41458c0 refactor(tvix): move castore into tvix-castore crate
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.

Notable changes:
 - `fixtures` and `utils` had to be moved out of the `test` cfg, so they
   can be imported from tvix-store.
 - Some ad-hoc fixtures in the test were moved to proper fixtures in the
   same step.
 - The protos are now created by a (more static) recipe in the protos/
   directory.

The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)

Bug: https://b.tvl.fyi/issues/301

Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:51:21 +00:00
Vincent Ambo
c5770e65dd feat(nix/renderMarkdown): add optional tagfilter argument
Makes it possible to disable tag filtering for rendered content (on by
default, of course).

Change-Id: I74ecfee97eaa7abf32049172b28705e7a3f21548
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9276
Tested-by: BuildkiteCI
Reviewed-by: Mark Shevchenko <markshevchenko@gmail.com>
2023-09-08 13:51:26 +00:00
Vincent Ambo
fcda068235 feat(nix/sparseTree): add optional name argument
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.

I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.

Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-08 20:11:56 +00:00
Vincent Ambo
9f762bf623 docs(nix-1p): explain scope of with
This fixes https://github.com/tazjin/nix-1p/issues/13

Due to with's use of a semicolon instead of some kind of bracing
statement, people can get confused about what the scope of a `with`
is.

Hopefully this change should make it more explicit.

Change-Id: Ifaff28e09a2e8b8b244a9c11d44b2780dee284ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8951
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-01 19:53:31 +00:00
Florian Klink
8b870c7ec9 feat(nix/buildkite): add meta.ci.buildkiteExtraStepArgs
This allows setting / overwriting arbitrary step arguments, such as setting
[Retry attributes](https://buildkite.com/docs/pipelines/command-step#retry-attributes)
for a specific readTree target.

It's intended to be used for cases where modelling each and every option
in a custom meta.ci attribute would be unfeasible.

Change-Id: I3352d5353b26a41a16760a7df37cd5ffee1665bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8983
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-07-24 20:51:14 +00:00
Florian Klink
f41b611761 feat(nix/buildkite): add meta.ci.buildkiteExtraDeps
This allows setting a buildkiteExtraDeps in `meta.ci`, which will get
added to `depends_on` (which can also be a list).

It allows explicitly describing a dependency on another buildkite step,
either generated by nix/buildkite, or part of the static pipeline.

At some point we might want to expose our key calculation function too,
similar to how readTree exposes mkLabel already, but that's left for
a followup.

Change-Id: I793170401ccd3907ad8bf232b80ca7a492e1c942
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8980
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-24 20:51:14 +00:00
Florian Klink
728de762fd feat(nix/buildkite): support meta.timeout
This uses the nixpkgs convention of meta.timeout, and adds a
timeout_in_minutes field to the pipeline step in case its set.

Fixes https://b.tvl.fyi/issues/285.

Change-Id: Ia72e3832f14bf9172319bce070c5b0944f1c96fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8970
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-07-18 16:47:58 +00:00
Florian Klink
6c630aadc8 feat(nix/buildkite): drop postBuild logic
This has been scheduled for removal in 2022-10-01, but it didn't happen
so far. Let's remove it now, better late than never.

Closes https://b.tvl.fyi/issues/286.

Change-Id: I26fe07360c694c1c93418f3310b4067051e08c87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8969
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-18 16:47:58 +00:00
Vincent Ambo
5b1c327c82 docs(nix-1p): suggest using nix repl to experiment with commands
This fixes https://github.com/tazjin/nix-1p/issues/14

I guess some people also don't discover the REPL for a while unless
told about it.

Change-Id: I76366c1cb02b5612a5aa87fe33b40b7bb0db3ff3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8950
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-07-11 16:06:23 +00:00
Vincent Ambo
09486d483c docs(nix-1p): import is a builtin, not a keyword
Pointed out by @Gaelan on GH in
https://github.com/tazjin/nix-1p/issues/12

As noted there, some Nix syntax highlighters will colour `import`
differently (e.g. the Emacs nix-mode), but it's not technically a
keyword in the language.

Change-Id: I1bbd36261cda31deb9ba47380a2810e64ba03ea1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8832
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-19 22:20:39 +00:00
Florian Klink
4ddfdc2ae0 feat(nix/buildkite): extraSteps: include parentLabel into hash
Contrary to the normal steps, extra steps only show the attribute name
in their label, not the whole attribute path.

To make sure these are still unique, also incorporate the parentLabel
into the string that's hashed.

Change-Id: Ic7a1f76aaac159ab5bd30054e422234882947990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8708
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-03 14:09:28 +00:00
Florian Klink
ed159e3a1d feat(nix/buildkite): add step key to release steps
This uses the same logic ("sha1sum of label") as already used in the
"build" chunks.

Fixes https://b.tvl.fyi/issues/276.

Change-Id: I76e59a141bd05f7d1d997a6471ea65e8cc7b4c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8707
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-03 13:47:22 +00:00
Florian Klink
4cf0ee8ea3 docs(nix/buildkite): escape "release" in warning
When using the buildkite tooling with `postBuild` parameter set, the
following error message is shown:

```
This replaces the previous boolean postBuild API in extra
step definitions. Please remove the postBuild parameter from
this step and instead set phase = release;.
```

However, `release` is no symbol, this should be a string.

Update the message to reflect that.

Change-Id: I341c81d862204db06946512428910cada28481ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8690
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-01 10:34:12 +00:00
sterni
830bc9961a chore: remove comments formerly meant for alignment
/**/ is a nice way to align if statements which doesn't work with
nixpkgs-fmt, since it'll reflow the comment to the line preceding the
if. Consequently, we can delete these comments now.

Change-Id: Ifa5327f846a903e07607b21f8eedbc32fc36f758
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8689
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-05-31 10:55:07 +00:00
Vincent Ambo
159646787a feat(nix/lazy-deps): add override pattern for deps
Introduces a `.overrideDeps` attribute with which additional tools can
be supplied. This works like `.override` in nixpkgs.

Change-Id: I69a009b51f7f073a2d030eda5e3b5310e0f8e883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8491
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-04-24 10:56:40 +00:00
Vincent Ambo
44a5e14200 feat(nix/readTree): implement .skip-tree marker for subtree ignoring
With this change, readTree gains the ability to notice a `.skip-tree`
marker in addition to the `.skip-subtree` marker.

The behaviour of the new marker will completely ignore the folder that
the marker is located in (i.e. no node will be present for it in the
parent at all).

To make this work, the recursive function in readTree had to be
modified to return a sentinel value (noting that a tree has requested
to be skipped) which is then filtered out when constructing the list
of children. The actual `readTree` function is now a wrapper around
this inner, sentinel-yielding implementation which unwraps the result
set.

For obvious reasons, `.skip-tree` is not allowed at the top-level and
readTree will throw an error if it encounters it there.

Fixes: b/244
Change-Id: Ica731bc1af356e881fd3d31c7109f62ffd2762ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8185
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-28 12:15:33 +00:00
Florian Klink
7df27dad12 feat(nix/bufCheck): always run from repo root
This doesn't work when run from subdirectories, and going there manually before
running `mg run //nix/bufCheck` is annoying.

Change-Id: Icd30a7596ff0dfe2781f7cfa1b4085cbfdebd6ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7324
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2022-12-27 13:27:40 +00:00
Florian Klink
a2ac8a66c1 feat(nix/bufCheck): ensure .pb.go is up to date
Change-Id: I03e1c3aced266966f6959dbce05c1f1fb1480b5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7323
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 13:27:40 +00:00
Florian Klink
54642c822d chore(nix/buildGo): drop thing.proto from example
This was missed from cl/7536.

Change-Id: I12c12bf91bad00c30bb63518d65c320a61cbf442
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7641
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
7fbac93940 chore(nix/buildGo): drop buildGo.proto and buildGo.grpc
As described in  https://b.tvl.fyi/issues/221#comment-344, buildGo.proto was a
mistake and should be removed.

Change-Id: Ic588a5e8eea58e83e3ec9a37ac681ce526028718
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7536
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-26 11:17:22 +00:00
zimbatm
66a8ce5900 feat(nix/lazy-deps): add passthru.devShell
Allows to easily convert a lazy-deps into a nix-shell compatible
derivation.

Change-Id: I65a0609642f12e0d00658c69cc394a73dddccd16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5855
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 10:31:34 +00:00
zimbatm
99030d10ce refactor(nix/lazy-deps): use runCommand
writeTextFile is nice, but not flexible enough to allow the passthru
argument needed for a follow-up change.

Change-Id: I4f0cffd0f29b2c06b0155101d3806c9c5745c37a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5854
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 10:31:34 +00:00
sterni
19ddb1466e revert(nix/utils): onlyDrvPath to get the drvPath w/o the outputs
This reverts commit 6813598c17.

Reason for revert: This function is a re-implementation of
builtins.unsafeDiscardOutputDependency which I missed at the time.

Change-Id: I5bb52bfd5e8d51defaf90ee795b0fe99be84f6db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7265
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-09 15:36:29 +00:00
sterni
57cf952ea9 chore(3p/sources): Bump channels & overlays (OpenSSL edition)
* //ops/machines/whitby: Disable grafana, since the grafana module was
  changed upstream in a way that our configuration no longer works.
  Since the OpenSSL security update is relatively pressing, adapting the
  grafana configuration beforehand is not a hard requirement. See
  https://github.com/NixOS/nixpkgs/pull/191768.

* //tools/depotfmt: keep Go at version 1.18 to forgo a reformat of the
  tree.

* //nix/buildGo: keep Go at version 1.18, as 1.19 changed the CLI
  interface (?) in a way that breaks buildGo.

* //3p/overlays/tvl: drop upstreamed tdlib upgrade.

* //3p/overlays/tvl: patch buf to work around breakage due to git 2.38.1

TODO items for Go are tracked in b/215.

Change-Id: Ie08fef49cf3db12e6b5225a8b992a990ddc5b642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7141
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-03 15:10:39 +00:00
William Carroll
1cd7678bdb feat(ops/buildkite): Support "soft failures"
Example Usage:

```nix
{ pkgs, ... }:

(pkgs.writeText "foo" "bar").overrideAttrs (_: {
  meta.ci.extraSteps.bar = {
    label = "testing soft_fail";
    softFail = [
      { exit_status = 253; }
      { exit_status = 251; }
    ];
    command = pkgs.writeShellScript "fail" ''
      # Trigger a soft failure by exiting 253 or 251
      exit 253
    '';
  };
})
```

Buildkite Documentation: https://buildkite.com/changelog/56-command-steps-can-now-be-made-to-soft-fail

Change-Id: Idfce15a8d9cde568aa24ebd03956ada399bba5d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7076
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-01 22:11:05 +00:00
Florian Klink
6025242fc7 chore(buf): Use nixpkgs-provided buf
The version of buf used is quite old.

nixpkgs provides a more recent version, but it requires us to migrate
config to the latest version.

depot_scanner.proto doesn't honor some of the conventions, so we need
allow_comment_ignores and drop a bunch of comments in there.

Change-Id: Ic978fe92fb7c8471f58c137497528f18aad8f3ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7053
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
2022-10-21 18:39:03 +00:00
sterni
5753fc14f5 refactor(nix/dependency-analyzer): use unsafeDiscardOutputDependency
Turns out that pathContextDrvPath already exists as a builtin which
is very convenient. Actually somewhat embarassing that I missed this
for so long.

Change-Id: Ieb5e113d70dec548b3053911ff9dbe9ed48402be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7050
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 18:21:25 +00:00
Evgeny Zemtsov
b5b968e103 fix(nix/buildkite) don't fail if gated step has no branches
Change-Id: Id3365c18bc3904b42e8b129a513cff4480a2cb07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7051
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 15:27:31 +00:00
sterni
8149ca5c38 fix(nix/dependency-analyzer): don't read from non-drv store paths
Emitting dependencies on non-drv store paths from drv directDrvDeps is
fine and actually correct, even though the Nix 2.3 version can't do it
at the moment (but this would change when the placeholder implementation
is replaced using a drv parser). However, we can't necessarily determine
the dependencies of non-drv store paths because such store paths may be
binary files that can't be read in by readFile due to NUL bytes.

Change-Id: Ifbd101adaee4f32f10c010fa79e19b9b1127fc6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6945
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-11 18:35:50 +00:00
sterni
81fadbe6eb fix(nix/buildGo): fix for Nix >= 2.6 readFile changes
The string context retained by readFile would leak into attribute keys
in fromJSON which may not have string context in any Nix version. We
don't need Nix >= 2.6 support, but buildGo may have external users and
this change is simple enough.

Change-Id: I593f1ef513502691119428d26d508a5f4d378543
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6946
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-11 17:38:42 +00:00
Evgeny Zemtsov
b319e00831 fix(nix/buildkite): split extra steps based on all known phases
Fix a bug introduced by 2ca153141 (I merged the wrong patchset).

Issue happens when pipeline is split by phases into independent
evals (e.g. build/release/deploy).

Splitting extra steps requires knowledge of all known phases,
otherwise pipeline evaluation fails due to extra steps from inactive
phases.

Change-Id: Iab0f2dc3eadda281e483055e26f00a95442e15b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6942
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-11 09:17:25 +00:00