Commit graph

242 commits

Author SHA1 Message Date
William Carroll
1cd7678bdb feat(ops/buildkite): Support "soft failures"
Example Usage:

```nix
{ pkgs, ... }:

(pkgs.writeText "foo" "bar").overrideAttrs (_: {
  meta.ci.extraSteps.bar = {
    label = "testing soft_fail";
    softFail = [
      { exit_status = 253; }
      { exit_status = 251; }
    ];
    command = pkgs.writeShellScript "fail" ''
      # Trigger a soft failure by exiting 253 or 251
      exit 253
    '';
  };
})
```

Buildkite Documentation: https://buildkite.com/changelog/56-command-steps-can-now-be-made-to-soft-fail

Change-Id: Idfce15a8d9cde568aa24ebd03956ada399bba5d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7076
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-01 22:11:05 +00:00
Florian Klink
6025242fc7 chore(buf): Use nixpkgs-provided buf
The version of buf used is quite old.

nixpkgs provides a more recent version, but it requires us to migrate
config to the latest version.

depot_scanner.proto doesn't honor some of the conventions, so we need
allow_comment_ignores and drop a bunch of comments in there.

Change-Id: Ic978fe92fb7c8471f58c137497528f18aad8f3ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7053
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: tazjin <tazjin@tvl.su>
2022-10-21 18:39:03 +00:00
sterni
5753fc14f5 refactor(nix/dependency-analyzer): use unsafeDiscardOutputDependency
Turns out that pathContextDrvPath already exists as a builtin which
is very convenient. Actually somewhat embarassing that I missed this
for so long.

Change-Id: Ieb5e113d70dec548b3053911ff9dbe9ed48402be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7050
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 18:21:25 +00:00
Evgeny Zemtsov
b5b968e103 fix(nix/buildkite) don't fail if gated step has no branches
Change-Id: Id3365c18bc3904b42e8b129a513cff4480a2cb07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7051
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 15:27:31 +00:00
sterni
8149ca5c38 fix(nix/dependency-analyzer): don't read from non-drv store paths
Emitting dependencies on non-drv store paths from drv directDrvDeps is
fine and actually correct, even though the Nix 2.3 version can't do it
at the moment (but this would change when the placeholder implementation
is replaced using a drv parser). However, we can't necessarily determine
the dependencies of non-drv store paths because such store paths may be
binary files that can't be read in by readFile due to NUL bytes.

Change-Id: Ifbd101adaee4f32f10c010fa79e19b9b1127fc6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6945
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-11 18:35:50 +00:00
sterni
81fadbe6eb fix(nix/buildGo): fix for Nix >= 2.6 readFile changes
The string context retained by readFile would leak into attribute keys
in fromJSON which may not have string context in any Nix version. We
don't need Nix >= 2.6 support, but buildGo may have external users and
this change is simple enough.

Change-Id: I593f1ef513502691119428d26d508a5f4d378543
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6946
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-11 17:38:42 +00:00
Evgeny Zemtsov
b319e00831 fix(nix/buildkite): split extra steps based on all known phases
Fix a bug introduced by 2ca153141 (I merged the wrong patchset).

Issue happens when pipeline is split by phases into independent
evals (e.g. build/release/deploy).

Splitting extra steps requires knowledge of all known phases,
otherwise pipeline evaluation fails due to extra steps from inactive
phases.

Change-Id: Iab0f2dc3eadda281e483055e26f00a95442e15b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6942
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-11 09:17:25 +00:00
Griffin Smith
b0547ccfa5 feat(nix/readTree): Handle a builtins w/o scopedImport
We (might) not want to implement scopedImport in tvix given it's
considered a bit of a misfeature; this makes readTree work with a
`builtins` set that doesn't have it (and if we decide we do want tvix to
have scopedImport, we can revert this pretty easily).

Change-Id: Ia3bbc847514672063a607d977ce167d489fa1131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6915
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-10 17:42:48 +00:00
Evgeny Zemtsov
2ca153141d feat(nix/buildkite): allow custom phases
This change automatically extends the list of known phases as soon as
they are added to active phase list.

This is great when a user wants to design pipelines with multiple
groups of dynamic steps.

For example in Resoptima we want to design deployment pipeline where
first only staging k8s namespaces are updated/tested and only after,
we update production.

Change-Id: Iab0f2dc3eadda281e483055e26f00a95442e15b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6923
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-10 15:40:12 +00:00
sterni
5174c21637 fix(nix/tag): correct no match check in discr
It uses discrDef internally, but passes `null` as the default tag name,
causing Nix to drop the attribute and return an empty attribute set if
the default case is hit. Consequently we need to check for the empty
attribute set, not `null` to figure out if there was no match found.

We can also test this behavior using `assertThrows` which was introduced
after the tag library was originally written.

Change-Id: I45adb2f9602762dfc867956323fb3f5ae4c8bd1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6904
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2022-10-09 13:25:14 +00:00
sterni
70427fd934 refactor(nix/stateMonad): optimize after
This should save on one function application which can be a big deal for
bigger for_ loops, I suspect. It's not really complicated, so why not.

Change-Id: I2bfcd254e55f1bea366b09de294b2bef9f5b5dda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6834
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-08 10:59:45 +00:00
sterni
3fab3b780b feat(nix/dependency-analyzer): improved directDrvDeps for Nix >= 2.6
This codepath will basically never be used in depot, but I want to add
it as kind of a note to myself. It's kind of a neat feature, although
I'm not quite sure it is going to stick around.

Change-Id: If0e26ef47bdedc6dbf3d048ad4fc9a3a1fd6c5a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6833
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-10-08 10:59:45 +00:00
sterni
57d5988b34 feat(nix/dependency-analyzer): find deps among a list of known drvs
This was written with the same intention (and reuses a little of its
code) as cl/5060 and cl/5063: We want to be able to emit dependencies
between //nix/buildkite pipeline steps, so that no agent is occupied
with waiting on locks for derivations built by a different agent.

This dependency information is already available to the Nix store
implementation (e.g. via `nix-store --query --references`) and can also
be obtained in the Nix language which is important, since the pipeline
is generated at evaluation time. (Note: For Nix 2.3, you either need a
strong convention about how derivations expose their dependencies (which
we don't) or rely on store implementation internals (drv files).
For Nix 2.6 there is a better trick, but it also relies on the existence
of drv files.)

The actual task can be formulated as follows: Given a set of
derivations, calculate the the closest derivations also in the input
each derivation depends on. (We call these (next) known dependencies.)
This is crucial because pipeline step often depend on each other only
indirectly with any number of intermediate derivations. For cl/5064 I
determined that 6 intermediate layers is quite common for dependencies
that are perceived to be “direct”.

This problem is solved as follows:

1. Calculate the dependency graph of the combined dependency closure of
   all input derivations. This is quite easy and fairly quick thanks to
   the C++ implementation of builtins.genericClosure. One weak point of
   the current implementation is that the function to determine the
   direct derivation dependencies for Nix < 2.6 is quite hacky.

2. Take the graph from 1. and calculate a dependency graph that only
   connects the known derivations of the input, but retains all
   connections between them (minus intermediate nodes).

In practice the dependency graph is represented as an attribute set
mapping derivation paths to a list of derivation paths it depends on.
The second step is performed by adding a second list of known derivation
paths it depends on.

The main improvements over the previous concept (cl/5060 and cl/5063):

* We only try to find the closest known dependencies in the dependency
  graph whereas we would traverse emit dependencies for the entire
  dependency closure.

* We immediately store the calculation of the closest known dependency
  in the dependency graph, even for intermediate nodes. This avoids
  recalculating the connection (which was a big drawback of the previous
  approach) and makes the calculation itself cheaper.

You can run `mg build //nix/dependency-analyzer:example` to build a
visualization of the internal dependencies between `depot.ci.targets` as
discovered by dependency-analyzer.

Change-Id: If8c0cdfc8470d4b337336257d9818aaa0d51110f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6832
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-08 10:59:45 +00:00
vpfeiffer
55ea15ab83 docs(nix/nix-1p): Update to use final and prev
The convention has changed to use `final` instead of `self` and `prev`
instead of `super`. This new convention is much easier to understand
especially for users that are new to Nix and Nixpkgs.

This change is notable in the Nixpkgs project code. I added a note
mentioning that the official documentation hasn't been update with
this information. The documentation is currently undergoing an
overhaul and that is why I believe this hasn't been reflected in the
nixpkgs documentation as of yet.

Imported-From: https://github.com/tazjin/nix-1p/pull/9/commits
Change-Id: I901df69a564969b1784b3d7ac629cfaf85093acb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6859
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-05 11:14:38 +00:00
sterni
5e097aa8e9 feat(nix/stateMonad): simple Nix state monad implementation
In the absence of do syntactic sugar relatively tedious to write, but
useful to express certain types of algorithms. I found it useful to
memoize intermediate results as they are calculated in order to avoid
recomputing them later in a drv dependency analyzer I've written.

Change-Id: I47cf3c644a96952c70276c9fa4cb3190b1c1e027
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6828
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-01 17:47:07 +00:00
Evgeny Zemtsov
335bf6900d fix(nix/buildkite): follow parent skip behavior in extra steps
We found a bug after updating to latest tvl-kit which broke
incremental releases.

Bug was related to the fact that extra steps skip attribute had
precedence over parent configuration. This is a desired behavior when
extra step is explicitly set to `skip=true` but otherwise it must
follow parent.

Due to extra step normalization skip parameter is always set to false
if not explicitly set.

Along the way, I'm adding support for setting skip attribute on extra
steps as string so that people can define skip reasons.

The bug was introduced by commit:
b9d79109d feat(ops/buildkite): Allow skip of individual steps

Change-Id: I8a46d0926a749d2434412b309c661b749e9dbf37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6827
Autosubmit: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-30 13:14:54 +00:00
sterni
0c178a0ef6 chore(3p/sources): Bump channels & overlays
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.

* runCommandNoCC -> runCommand
* gmailieer -> lieer
  We also need to work around the fact that home-manager hasn't catched
  on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
  This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy

Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.

Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-09-28 08:02:31 +00:00
sterni
f39a4e4fa7 refactor(nix/buildLisp/tests/argv0): use derivation for tests
There is no need to use an extraStep, actually, and using derivations
reduces noise on CI.

Change-Id: I897c3c3f7e0acee8f051fcc01450ff57176726f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6573
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-20 10:43:54 +00:00
Luke Granger-Brown
f190712b7f chore(gerrit): migrate OWNERS files to code-owners style
Change-Id: Iacc521dfdd4b4a2d5cef3920cf8189bcce35a488
2022-09-19 11:13:28 +00:00
Evgeny Zemtsov
8fa3bc7137 feat(buildkite/mkPipeline): allow fast failure for build steps
This is supposed to help with resource wasting in situations when
multiple dynamic steps depend on a failing derivation. Cost of
failure currently is `C = T * D` where:
 - T -> time to compile the failing derivation
 - D -> amount of CI targets depending on a failing derivation

Switching to `Fail Fast` limits it to just T (time of a single
failure). Which helps a lot, especially while upgrading nixpkgs.

Fast fail at this moment is in preview, so to enable it:
 - Enable `Fail Fast` feature at pipeline or organization level
 - Set `cancelOnBuildFailing` parameter to true for `mkPipeline`

Change-Id: I4373a46633522d21e94cfa8bac35243b4eeb0b9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6243
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-01 14:35:03 +00:00
sterni
049ec22943 feat(nix/buildLisp): re-enable CCL
The problem went away once again, let's see how long it'll last this time.

As it turns out, CCL has a Unicode Standard conforming string
implementation that doesn't allow the use of (lone) surrogate code
points, requiring us to disable a test in cl-json which tested the
behavior of en- and decoding of such a (technically illegal) string.

Change-Id: I8bfa482934bbf94f86cecdde02d5c3d4e77950a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6204
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-08-30 17:51:35 +00:00
Jean-François Roche
282ad0015b fix(ops/buildkite): do not set branch if empty
To run an extra step for all branches, user don't set the
`branches` attribute. This change avoid setting `branches` to null in such a case.

Change-Id: Iabf2f3d0411b037ece5584f30b29c7e65420b63f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5975
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 14:41:39 +00:00
Jean-François Roche
45c794f1b3 feat(ops/buildkite): Allow agents targeting
Using the `agents` attribute, it lets the user target specific agents to run a step.

Change-Id: Id6fc0981d4879b77598854e1f296dffffb33a405
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5974
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 09:29:58 +00:00
Jean-François Roche
b9d79109dd feat(ops/buildkite): Allow skip of individual steps
Using the `skip` attribute, it lets the user decide to temporarily skip a step.

Change-Id: If63ce60a2b4148f041655912730acc4ae9e28534
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5971
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-07-25 09:29:58 +00:00
Vincent Ambo
544d72189c chore(nixery): use nix-1p from within the depot
Since the source of nix-1p is checked in under //nix/nix-1p, we should
use it from there if Nixery is being built inside of depot.

Change-Id: Iddd54f7b93b398b2f909db6ee105366a9914a2ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5882
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2022-06-16 15:08:27 +00:00
Vincent Ambo
b9c31d6658 refactor(nix/buildkite): Optimise steps if builds are not enabled
Generating a release-only pipeline skips a bigger chunk of eval this
way (the step itself is never actually evaluated, which means we never
actually compute the drv), which can be quite beneficial in terms of
evaluation time.

Change-Id: I2739026ddd1c6a86f82627ac26a046c5fe7359ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5830
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-06-06 12:02:26 +00:00
Vincent Ambo
df83e3d556 fix(nix/buildkite): Disable depends_on if build phase is not run
Extra steps that use `depends_on` (e.g. if they need output from their
parent) should not actually depend on their parents build step if the
build phase is not active.

This is required to actually decouple the phases.

Change-Id: I398da9a8a53e97ca3c635342259fc722d54b8e4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5829
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-06-06 12:02:26 +00:00
Vincent Ambo
302f7e6623 feat(nix/buildkite): Allow toggling of individual phases
Using the `activePhases` attribute, the set of phases included in an
evaluation can be modified.

This lets users generate e.g. ONLY the release steps of a pipeline.

Change-Id: Ib0c38826dd69666094d619f5f324d1baafce8134
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5828
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-06-06 12:02:26 +00:00
Vincent Ambo
837560c846 fix(nix/emptyDerivation): Use system from readTree parameters
Change-Id: Ia7f9f4d0e7c06fa5433213a315c3354a83e94545
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5833
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-06-03 23:27:03 +00:00
Vincent Ambo
cf00365f84 feat(nix/buildkite): Validate available phases in extra steps
This will avoid things like extra steps being accidentally ignored
because of typos.

Change-Id: Ic4fa5925e42a7a449f89b4cde1510e216e91da6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5827
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
876b71f641 fix(nix/buildkite): Forbid 'prompt' in build phase steps
This would block CI on human-approval if people were allowed to do it,
so they're just not.

Change-Id: I8a9b657d5c91636a7b4de249b977e24fc0941a1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5826
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
56a97a0337 refactor(nix/buildkite): Explicit support for build phases
Previously the extra steps were roughly divided into steps that run
"at build time" (i.e. before we publish results to Gerrit), and
"post-build" (i.e. later on).

In practice, these are something like a build/release pairing, where
steps running after the build results are returned are mostly run for
side-effects (e.g. publishing git subtrees to external repos).

This refactoring makes this distinction explicit in //nix/buildkite
and changes the extraSteps API with an explicit `phases` attribute
instead of the previous `postStep` attribute.

In practice the previous API is still supported, but will throw
evaluation warnings until an arbitrarily chosen cutoff date of
2022-10-01 at which point we will change using it into a hard error.

This uncovered a few strange behaviours which we only accidentally
avoided, most of which I have left TODOs about and will clean up in
subsequent commits.

The purpose of this commit is to allow for separate evaluations of
only build or only release steps, for example if release steps are
evaluated in a slightly different context (e.g. with overridden
versioning that is not relevant to standard CI functionality).

Change-Id: I0b0186e3824273c15a774260708702d4a5974dac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5825
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
a027ee9f03 refactor(nix/buildkite): Rename "post" steps to "release" steps
This is in preparation for a subsequent CL that will do much more
significant changes in //nix/buildkite.

Change-Id: I80a8d67d3a7d593854c8d711572483c2581e7881
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5824
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-06-03 17:36:15 +00:00
Vincent Ambo
b8301ed64b docs(nix/buildkite): Fix an out-of-date comment about chunk size
Change-Id: Ic1f874f4ca83f9088355dc0512723ea962e0db52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5823
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-06-02 20:32:46 +00:00
sterni
3c5da97609 fix(nix/buildLisp): resolve eval problem in ccl code
The isPowerPC predicate has been [removed], since it was misleadingly
named (it just matches PowerPC, 32bit, little endian). This means the
64bit code path could now actually work.

Not sure about endianess, the CCL docs don't really say much regarding
that topic.

[removed]: https://github.com/NixOS/nixpkgs/pull/168113

Change-Id: Icf4a8c6b1df95fa597ed87508f57aaa73e6185ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5796
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-06-01 10:46:03 +00:00
Vincent Ambo
0d4cf119bc feat(nix-1p): Export subtree to GitHub
We needed a derivation for that, but this can also be used in the
Nixery docs building process (which includes nix-1p).

Change-Id: If97cf785a33d703af975da3b41de9b69566dfa81
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5789
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-29 12:30:49 +00:00
Vincent Ambo
9f6215f6e0 subtree(nix/nix-1p): Merge nix-1p into depot
It's time to stop maintaining this on GitHub, and use the
export-from-depot feature instead.

Change-Id: Ic6b840bb0e8580c7214113467b4995ea3d2fae02
2022-05-29 13:31:03 +02:00
sterni
6813598c17 feat(nix/utils): add onlyDrvPath to get the drvPath w/o the outputs
I want to use this utility in a deploy script where the .drv is
nix-copy-closure-d to a remote host and realized there. Consequently it
doesn't make sense that the local deploy script depends on the
derivation's outputs which drvPath does by default.

This also came up when working on //nix/buildkite, although we didn't
end up using it there.

Change-Id: I952bbfd4d7e9de212569d5ee12182eb50d360f53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5767
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-28 12:01:46 +00:00
sterni
a6367b4bdc fix(nix/utils): remove predicates based on symlink heuristic
Due to [nix#6579] the heuristic which allowed us to determine if a
symlink points to a directory is not reliable – if restrict-eval is
enabled it _will_ return wrong results. Until upstream resolves
this (and we backport the patch) it is probably best to not expose this
functionality at all.

[nix#6579]: https://github.com/NixOS/nix/issues/6579

Change-Id: Id847c794bb279be909c5426953c4fe13c2493343
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5761
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-28 11:35:42 +00:00
sterni
0e705fedb0 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: disable CCL once again due to
  The Mysterious Runtime Bug™.

* //users/tazjin/nixos: uninstall dmd which is broken in nixpkgs atm.

Change-Id: I8dd2220af48a7e087584b6f50529fb8477e6a2fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5699
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-27 21:15:10 +00:00
Vincent Ambo
c16a18a718 chore(nix/utils): Comment out tests for detection-ish of symlinks
This is broken for (as of yet unclear reasons) with restricted
evaluation mode.

Change-Id: Idbc16e7e21dfb113995c045659fefe2c1a535741
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5691
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-05-26 16:44:52 +00:00
sterni
5fae8b0826 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: ccl dumped images have fixed themselves… again

* //3p/git: rebase patch on 2.36.0

* //3p/overlays/haskell: remove upstreamed workarounds

* Disable everything depending on cgit temporarily, since it doesn't
  compile with git 2.36 yet.

Change-Id: I9dc11c0846641341adbdcc7162cbf149a15fe0cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5519
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-04 16:18:50 +00:00
Vincent Ambo
0ce396bdab feat(nix/lazy-deps): Add function to generate lazy binary dispatcher
There is a reoccuring problem in readTree-type repositories that use
nix-shell, where evaluation of the full set of dependencies that
should be made available to users takes a noticeable amount of time,
slowing down operations when `direnv` is involved.

In depot, we have so far fixed this by maintaining a manual `//bin`
directory which contains a set of symlinks to a central dispatch
script that can dispatch to various tools in depot lazily.

This script can instead be generated ad-hoc by Nix (pretty fast if we
can make assumptions like `git` and `nix-build` existing on user's
machines already) and added to $PATH.

The function introduced in this commit implements the logic for that.
The structure of the script is based on the existing
`//bin/__dispatch`.

This does not yet switch depot's envrc to use this new method of
installing dependencies lazily.

Change-Id: I92efcd9bb6aa51aa2709ad910a464e9dac97ee89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5512
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-05-02 23:34:55 +00:00
Vincent Ambo
c05c4995ab chore(3p/sources): Bump channels and overlays
Changes:

* updated keycloak configuration for new version
* migrate to emacs28 outside of //users, re-add emacs27 but with a
  warning attached urging people to migrate

Change-Id: I3e5765a63934541f72f6c4a8673d3b4671850c93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5501
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-04-21 16:54:07 +00:00
sterni
11f9306b9a chore(3p/sources): Bump channels & overlays
Use nixos-unstable-small which fixes CVE-2018-25032
(out of bounds write while compressing).

* //users/grfn/xanthous:

  - Supporting random-fu 0.3 requires considerable changes and patching
    random-extras (https://github.com/aristidb/random-extras/pull/5).

    For now we downgrade random-fu and its dependency rvar to 0.2.*,
    forcing us to build xanthous with GHC 8.10.7, due to random-fu 0.2.*
    not supporting that version.

    Nix expressions for the downgraded packages are checked in to avoid
    the potential need to compile Haskell at pipeline eval time.

  - generic-arbitrary exposes a GenericArbitrary newtype now.
    This means we no longer have to implement it in xanthous
    downstream and patch generic-arbitrary to expose the
    GArbitrary type class.

  - Minor adjustments for lens 5.0:

    Xanthous.Game.Memo: clear needs to use ASetter' instead of Lens'

    Xanthous.Data.EntityMap: TraversableWithIndex no longer has an
    itraversed function.

  - Xanthous.Orphans: adjust for aeson's KeyMap, use KM.size explicitly
    instead of relying on MonoTraversable's length

* //nix/buildLisp: the CCL issue has resurfaced, disabling the
  implementation once again.

* //3p/arion: remove, as depot uses the nixpkgs package of it anyways.

* //users/wpcarro: accomodate GHC 9.0.1's stricter parsing of operators.

* //users/tazjin: disable rustfmt as it stopped respecting settings

* //3p/overlays: upgrade home-manager until fix for serivce generation
  has landed upstream

* //users/grfn/system: remove rr override, as the pinned commit is part
  of the 5.5.0 release shipped by nixpkgs.

Change-Id: If229e7317ba48498f85170b57ee9053f6997ff8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5428
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-04-11 19:37:01 +00:00
Vincent Ambo
e70428e75b refactor(ops/pipelines): Configurable GraphQL token location
For external users of the pipeline construction, the token might be in
a different path than `/run/agenix/buildkite-graphql-token`.

It is made configurable through the BUILDKITE_TOKEN_PATH environment
variable. This should be configured on the pipeline level to apply to
all steps.

Change-Id: I23c52e2d705e4134b8b013f8603f92e5533a6e44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5424
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
2022-03-30 11:06:49 +00:00
sterni
508a62b603 chore(3p/sources): Bump channels & overlays
* Remove use of aliases that have been removed in nixpkgs commit
  a36f455905d55838a0d284656e096fbdb857cf3a:

  - ncat
  - x11
  - nologin
  - dbus_libs
  - emacsPackagesGen
  - man-pages
  - pulseaudioLight

Change-Id: Ide603bf48bc7f77e10e4aa715ba025aece3644fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5387
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-03-19 17:11:59 +00:00
Vincent Ambo
b8660386eb fix(nix/buildkite): Do not set 'branches' attribute in step groups
This is no longer accepted by the Buildkite API and causes build
failures.

Functionality is unchanged since we also set the property on the step
itself.

Change-Id: Ib0e0908e4093ca4522711170a7179ce4bacafdc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5324
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-02-24 16:50:32 +00:00
Vincent Ambo
c7bc8aac49 docs(yants): Update josh cloning instructions
Change-Id: I4e9e31488f91465df6b2c6800a3940118107524b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5315
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-19 11:37:00 +00:00
sterni
235eda6649 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: re-enable CCL, as the crash has been fixed upstream,
  although it is unclear what exactly caused / fixed it.

* //ops/whitby: the kitty build broke upstream, so we can't install the
  terminfo on whitby for a bit.

Change-Id: I5710acbe837fbc936e334b2e81f9cf00ed6ae280
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5274
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-13 09:15:09 +00:00