Commit graph

250 commits

Author SHA1 Message Date
Vincent Ambo
0ce396bdab feat(nix/lazy-deps): Add function to generate lazy binary dispatcher
There is a reoccuring problem in readTree-type repositories that use
nix-shell, where evaluation of the full set of dependencies that
should be made available to users takes a noticeable amount of time,
slowing down operations when `direnv` is involved.

In depot, we have so far fixed this by maintaining a manual `//bin`
directory which contains a set of symlinks to a central dispatch
script that can dispatch to various tools in depot lazily.

This script can instead be generated ad-hoc by Nix (pretty fast if we
can make assumptions like `git` and `nix-build` existing on user's
machines already) and added to $PATH.

The function introduced in this commit implements the logic for that.
The structure of the script is based on the existing
`//bin/__dispatch`.

This does not yet switch depot's envrc to use this new method of
installing dependencies lazily.

Change-Id: I92efcd9bb6aa51aa2709ad910a464e9dac97ee89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5512
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-05-02 23:34:55 +00:00
Vincent Ambo
c05c4995ab chore(3p/sources): Bump channels and overlays
Changes:

* updated keycloak configuration for new version
* migrate to emacs28 outside of //users, re-add emacs27 but with a
  warning attached urging people to migrate

Change-Id: I3e5765a63934541f72f6c4a8673d3b4671850c93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5501
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-04-21 16:54:07 +00:00
sterni
11f9306b9a chore(3p/sources): Bump channels & overlays
Use nixos-unstable-small which fixes CVE-2018-25032
(out of bounds write while compressing).

* //users/grfn/xanthous:

  - Supporting random-fu 0.3 requires considerable changes and patching
    random-extras (https://github.com/aristidb/random-extras/pull/5).

    For now we downgrade random-fu and its dependency rvar to 0.2.*,
    forcing us to build xanthous with GHC 8.10.7, due to random-fu 0.2.*
    not supporting that version.

    Nix expressions for the downgraded packages are checked in to avoid
    the potential need to compile Haskell at pipeline eval time.

  - generic-arbitrary exposes a GenericArbitrary newtype now.
    This means we no longer have to implement it in xanthous
    downstream and patch generic-arbitrary to expose the
    GArbitrary type class.

  - Minor adjustments for lens 5.0:

    Xanthous.Game.Memo: clear needs to use ASetter' instead of Lens'

    Xanthous.Data.EntityMap: TraversableWithIndex no longer has an
    itraversed function.

  - Xanthous.Orphans: adjust for aeson's KeyMap, use KM.size explicitly
    instead of relying on MonoTraversable's length

* //nix/buildLisp: the CCL issue has resurfaced, disabling the
  implementation once again.

* //3p/arion: remove, as depot uses the nixpkgs package of it anyways.

* //users/wpcarro: accomodate GHC 9.0.1's stricter parsing of operators.

* //users/tazjin: disable rustfmt as it stopped respecting settings

* //3p/overlays: upgrade home-manager until fix for serivce generation
  has landed upstream

* //users/grfn/system: remove rr override, as the pinned commit is part
  of the 5.5.0 release shipped by nixpkgs.

Change-Id: If229e7317ba48498f85170b57ee9053f6997ff8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5428
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-04-11 19:37:01 +00:00
Vincent Ambo
e70428e75b refactor(ops/pipelines): Configurable GraphQL token location
For external users of the pipeline construction, the token might be in
a different path than `/run/agenix/buildkite-graphql-token`.

It is made configurable through the BUILDKITE_TOKEN_PATH environment
variable. This should be configured on the pipeline level to apply to
all steps.

Change-Id: I23c52e2d705e4134b8b013f8603f92e5533a6e44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5424
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: asmundo <asmundo@gmail.com>
2022-03-30 11:06:49 +00:00
sterni
508a62b603 chore(3p/sources): Bump channels & overlays
* Remove use of aliases that have been removed in nixpkgs commit
  a36f455905d55838a0d284656e096fbdb857cf3a:

  - ncat
  - x11
  - nologin
  - dbus_libs
  - emacsPackagesGen
  - man-pages
  - pulseaudioLight

Change-Id: Ide603bf48bc7f77e10e4aa715ba025aece3644fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5387
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-03-19 17:11:59 +00:00
Vincent Ambo
b8660386eb fix(nix/buildkite): Do not set 'branches' attribute in step groups
This is no longer accepted by the Buildkite API and causes build
failures.

Functionality is unchanged since we also set the property on the step
itself.

Change-Id: Ib0e0908e4093ca4522711170a7179ce4bacafdc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5324
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2022-02-24 16:50:32 +00:00
Vincent Ambo
c7bc8aac49 docs(yants): Update josh cloning instructions
Change-Id: I4e9e31488f91465df6b2c6800a3940118107524b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5315
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-19 11:37:00 +00:00
sterni
235eda6649 chore(3p/sources): Bump channels & overlays
* //nix/buildLisp: re-enable CCL, as the crash has been fixed upstream,
  although it is unclear what exactly caused / fixed it.

* //ops/whitby: the kitty build broke upstream, so we can't install the
  terminfo on whitby for a bit.

Change-Id: I5710acbe837fbc936e334b2e81f9cf00ed6ae280
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5274
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-13 09:15:09 +00:00
Vincent Ambo
3d8ee62087 style(rust): Format all Rust code with rustfmt
Change-Id: Iab7e00cc26a4f9727d3ab98691ef379921a33052
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5240
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-08 12:06:39 +00:00
Vincent Ambo
a25675804c chore(nix/buildLisp): meta.targets -> meta.ci.targets
Change-Id: Ic8effcc94f3fcfa283436be3e5c0477ef89e9e94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5247
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-07 22:28:41 +00:00
Vincent Ambo
8b8c98380e refactor(readTree): deprecate meta.targets for meta.ci.targets
This means that we use the meta.ci attribute more consistently.

The meta.targets attribute is still read, but prints a big, red
warning telling people to migrate to the new one.

Fixes b/176

Change-Id: Ifb4452f529cfc6bbd5018ad7374cac1c83b10045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5238
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-07 15:13:52 +00:00
Vincent Ambo
3bde425860 refactor(nix): Move mkLabel from buildkite to readTree
This function is more generically useful than just for pipeline
construction.

A subsequent commit will use it inside of readTree itself.

Change-Id: I5eabd6f659726484667e060958865dddbc205762
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5237
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-07 15:13:52 +00:00
Vincent Ambo
b5150fbacf fix(nix/buildkite): read & print the store path of built derivations
This was useful to have in CI, e.g. when targeting a specific NixOS
system. The actual result symlink which is printed is not useful.

Alternative solution would be to change the wrapping of this so that
we conditionally create the symlink for extra steps, but I think it's
not worth the complexity of evaluating the step twice.

Change-Id: Id86eb5114bec935c63a2907ec5f169fc5d41a6cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5227
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-02-04 19:49:47 +00:00
sterni
0e45761853 chore(3p): Bump NixOS channels and emacs overlay to 2022-02-03
* //nix/buildLisp: This channel bump brought a bizarre regression
  in ccl, causing binaries to crash on thread clean up. This was
  likely caused by a glibc update in nixpkgs. We'll disable emitting CI
  targets for ccl until we can find out and fix what's going on.

Change-Id: I37629f384fa99ec4ef96ce7127fa7569adecb687
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5207
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
2022-02-04 14:39:41 +00:00
sterni
b57854e24d fix(nix/buildLisp): make uiop:argv0 work also for ccl and sbcl
Detection is broken there, too, as UIOP relies on setting the variable
before dumping the image in its portability wrapper dump-image which we
don't use at all.

Change-Id: If7bea5a8522a2e64707b1ee88d62d420bd00a952
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5112
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-02-02 18:21:06 +00:00
Vincent Ambo
bb140e16a2 feat(nix/buildkite): Add parentOverride argument to extra steps
This can be used to override the parent derivation if its output is
required, for example to inject versions which are only used during
releases to avoid cache-busting.

Change-Id: I2211496efa8f9bc98ea43b23e4f3f92c61a6da73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5184
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-02-02 13:36:58 +00:00
Vincent Ambo
ba7197c271 fix(nix/readTree): Use overrideAttrs with derivations
Derivations that support overrideAttrs now have their readTree
markers merged in using it, as passthru attributes.

This makes the significant difference that overriding readTree targets
using `overrideAttrs` keeps their readTree data intact.

Change-Id: Ieef635f048781bf4782c1a28532b89a66d9ca24d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5186
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-02 13:36:58 +00:00
Vincent Ambo
a49358d8aa fix(nix/buildGo): Use overrideAttrs to add metadata
Required for using overrideAttrs in readTree (cl/5186). Since this
uses pkgs.runCommand we know that overrideAttrs is available.

Change-Id: I18fdcc34cc79872834052caf4bf74555fdb766ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5187
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-02-02 13:36:58 +00:00
Vincent Ambo
aa122cbae7 style: format entire depot with nixpkgs-fmt
This CL can be used to compare the style of nixpkgs-fmt against other
formatters (nixpkgs, alejandra).

Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: cynthia <cynthia@tvl.fyi>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: grfn <grfn@gws.fyi>
2022-01-31 16:11:53 +00:00
Vincent Ambo
d7b2637f59 chore(nix/buildkite): reduce chunk size to 192
We've seen the famous 1 minute timeout on Buildkite again, probably
due to something (keys in targets?) increasing the overall payload
size of our chunks.

This reduces the chunk size by 25%. Lets keep an eye on it with this
value ...

Change-Id: I6bf0e9e4ab0d5b8de22773e6cd5da8d0959cc448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5105
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-29 09:17:07 +00:00
sterni
6908d960b2 feat(3p/overlays/ecl-static): 21.2.1 -> 1c98924
Seems like some issues to do with bytecode compilation have been fixed
at HEAD. closer-mop compiles again and an ironclad failure with the
next quicklisp/channel bump is avoided.

In this change pathname handling in ECL also changed somehow, causing it
to make the :directory part absolute by prefixing it with a slash which
made ld.bfd unhappy while linking an output path that began with a
double slash. This problem can be avoided by constructing the path as
ANSI Common Lisp intended. The truename on the out path is important to
make it recognize that it is indeed a directory.

Change-Id: I5e744022b92502f99ac0b33411a6be443707e200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5076
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-01-28 10:43:01 +00:00
Vincent Ambo
9e28029705 feat(nix/buildkite): Support un-gated post-build extra steps
Allows users to define steps with `postBuild = true` which always run
after 🦆, but do not require human approvals.

This can be useful for things like unconditional release steps.

Change-Id: Idbf6c48a9dedcfc6cc9b7f098423364e2fa72d2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5052
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
4d7dcf10ed refactor(nix/buildkite): Restrict step conditionals to refs only
The previous `condition` abstraction which allowed the full set of
Buildkite conditionals is way too leaky (it lets users to very
Buildkite-specific things which we may not want to allow, and which
are mostly not relevant to a pure evaluation).

Supporting only the `branches` condition (native to Buildkite) should
make it possible to port this to other future CI systems later.

Change-Id: Ib8adcc41db4f1a3566cbeecf13a4228403105c1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5051
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
1a1d706125 feat(nix/buildkite): Add support for gated steps
Adds support for extra build steps that specify a `prompt`. These
steps will be run at the end of the pipeline and will be gated by
human approval.

This mechanism can be used to, for example, stage releases of software
released from depot that are subject to approval.

Change-Id: I97bb505664a2ccf01142286f14e20a370afaa345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5033
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 14:57:06 +00:00
Vincent Ambo
5a88e47b71 refactor(ops/pipelines): Split build/post steps into separate chunks
This will create `build-chunk-$n.json` files for steps that should run
_before_ duck, and `post-chunk-$n.json` files for steps that should
run after duck.

The post steps are not yet uploaded to Buildkite, but we also don't
have any right now.

Change-Id: I7e1b59cf55a8bf1d97266f6e988aa496959077bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5047
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
2022-01-22 11:59:08 +00:00
Vincent Ambo
4e82a9dc86 feat(nix/buildkite): Add support for extra steps
This introduces a new feature to our CI system in which targets can
declare extra steps in `meta.ci.extraSteps`.

See the comment in //nix/buildkite/default.nix for an explanation of
how these extra steps are defined.

Change-Id: Icce2890c743286dd37f43024cd390dcebac8cdba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5008
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-21 11:49:03 +00:00
Vincent Ambo
16d72933fc chore(nix/buildkite): Drop --fork-point parameter
The --fork-point parameter is dependent on reflog data which may get
garbage collected. This can lead to flaky behaviour where it returns
no results and fails if `git gc` recently ran (Buildkite will do this
occasionally).

Though the parameter is semantically closer to what we're looking for,
the output is *usually* the same commit since we're not dealing with
more than one thing to compare.

Change-Id: Idc31e7a26fda2b7113edfa162d9d3811b1a01bf6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5032
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-20 19:00:28 +00:00
Vincent Ambo
d14c81007e fix(nix/buildkite): prevent swallowing errors from git-merge-base
The `local` usage we had before would silently swallow non-zero exit
statuses from the substituted git commands.

For some (as of yet unknown) reason, `git merge-base` seems to
sometimes silently fail and produce no output, which broke the rest of
the script logic.

This change will lead to an earlier error, but we don't know if it is
a fix for the actual cause of the git-merge-base problem because the
shape of that problem is unclear.

Change-Id: I4555c8638da450263fa2fd2c274dfdb69f65578e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5012
Tested-by: BuildkiteCI
Reviewed-by: kn <klemens@posteo.de>
2022-01-20 11:04:35 +00:00
Vincent Ambo
0652ac0ace refactor(nix/buildkite): Move fetch-parent-targets script here
This is no longer TVL-specific and should live here with the other
generalised stuff.

Change-Id: I95a1b4c0321f34812162d6fd40568269abf639dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5006
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-19 17:22:36 +00:00
sterni
307eea8e27 fix(nix/buildLisp): use filtered deps for native deps in testSuite
allDeps filters the lisp deps according to the given implementation,
processing any implementation conditional attribute sets. These are not
understood by allNative, so we need to pass it the already filtered
input or evaluation would fail.

Change-Id: I9eb2d0c3b2bf70d759d03490cf31fc585283ce7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5001
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-01-19 13:07:57 +00:00
Vincent Ambo
1c81ccb996 fix(3p): Move away from builtins.fetchGit
Moves to the derivation-based git fetchers everywhere in third-party.
This might help with forward-compatibility with newer Nix versions,
though that's not our primary concern right now.

Change-Id: I565bb72585b8639893e9ea3a9e233338aede63a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3903
Tested-by: BuildkiteCI
Reviewed-by: zseri <zseri.devel@ytrizja.de>
2022-01-17 14:18:37 +00:00
Vincent Ambo
0779f96687 feat(nix/buildkite): Check target map of parent to determine skips
This changes the logic for build pipeline generation to inspect
an (optional) parentTargetMap attribute which contains the derivation
map of a target commit.

Targets that existed in a parent commit with the same drv hash will be
skipped, as they are not considered to have changed.

This does not yet wire up any logic for retrieving the target map from
storage, meaning that at this commit all targets are always built.

The intention is that we will have logic to fetch the target
map (initially from Buildkite artefact storage), which we then pass to
the depot via externalArgs when actually generating the pipeline.

Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 11:49:01 +00:00
Vincent Ambo
0a21da2bb4 feat(ops/pipelines): Create drvmap structure for each commit
Always create a structure that maps all targets to derivations, and
persist it as a JSON file.

This relates to some of the ideas expressed in:

https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit

The file is always uploaded to Buildkite as an artifact. This allows
for retrieving it based on the commit ID in a Buildkite GraphQL query.

By default, Buildkite stores artefacts for 6 months. Storage location
can be overridden (with custom retention) through some environment
variables, but for now at TVL the Buildkite-managed storage is fine.
See also: https://buildkite.com/docs/pipelines/artifacts

In the subsequent filtering implementation, when diffing commits
across a time-range that exceeds artefact retention time, we should
simply default to building everything.

Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 10:26:08 +00:00
Vincent Ambo
b7ef2a579b refactor: Generalise pipeline generation in //nix/buildkite
Extracts the logic for generating our Buildkite pipeline (which has
been copy&pasted and slightly modified in some places outside of
depot) into a generic //nix/buildkite library.

This should cause no change in functionality.

Change-Id: Iad3201713945de41279b39e4f1b847f697c179f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4726
Autosubmit: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-12-28 15:34:39 +00:00
zseri
00adb6e8f3 feat(nix/yants): improve error message for errornous predicate
while trying to yantsify `mkSecrets` in https://cl.tvl.fyi/c/depot/+/4688,
I(zseri) needed to debug a failing evaluation which boiled down
to a result.ok containing something which wasn't boolean,
but the error message didn't indicate where that value came from.

I debugged yants and found that the only place which didn't
simply combine boolean values or use functions which always
return booleans, I managed to isolate the error to the
`pred v` expression. To avoid the necessity to debug yants
to find this, I improve the error message for this case
to mention that
- a restriction predicate is invalid
- what's the name of the failing restriction
- the unexpected predicate return value

Change-Id: I6c570a33ccc5afc445f208e2e8855c49fb37abaf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4698
Tested-by: BuildkiteCI
Reviewed-by: zseri <zseri.devel@ytrizja.de>
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: zseri <zseri.devel@ytrizja.de>
2021-12-27 08:20:58 +00:00
sterni
b9f510d989 fix(nix/buildLisp): don't emit a target for impl used at top level
These targets would be the same derivation, but named differently which
is noisy and causes a few, mostly subtle issues:

* Buildkite struggles with large pipelines a bit, we can save quite a
  few steps by removing these.

* Having two jobs for the same derivation sometimes causes the annoying
  situation that an agent would do nothing except waiting for a lock.

* Non-nix CI we add in the future may not be able to recognize that
  these targets are the same and do extra work unnecessarily.

Change-Id: I1103e719ade1d3859d222b713969ac34a8765cba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4515
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-20 12:46:13 +00:00
sterni
887ac4d3d4 feat(nix/utils): handle toString-able sets produced by builtins.path
When preparing cl/4381 I noticed that we actually handle this case
properly. depot.nix.utils.storePathName depot.path now works as
expected.

Change-Id: Ice9329c67b2e2210852012f5abe82fbbb13193de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4382
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: sterni <sternenseemann@systemli.org>
2021-12-17 10:47:34 +00:00
Griffin Smith
ca56cb2d9f docs(nix/readTree): Tweak grammar of docs for gather
This seemed to be missing a word previously.

Change-Id: Ifa860051d6b692a626dbaddbaee44b761f2274ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4386
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-16 21:14:15 +00:00
Vincent Ambo
2fa157ccd6 fix(readTree): Apply .skip-subtree to Nix-file children as well
This behaviour was previously confusing, since readTree's data
structure treats children from Nix files and directories as identical
but only one of them would be affected by .skip-subtree

The "subtree" to be skipped here refers to all children of the
structure.

Change-Id: Idf596c9823f09cc2acf49523916bde4b801b8519
2021-12-08 18:22:00 +00:00
sterni
8dc54f89cd fix(nix/runTestsuite): don't crash on big test suites
Having `prettyRes` in the execline script causes it to fail because of
the argv limit if your test suite is long enough. For the succeeding one
we can work around this by hashing it (since we only care that something
changes if the test suite changes), in the case of the failing one where
we want to print the results, we use runExecline's stdin mechanism.

Change-Id: I2489f76acfbe809351f51caefe2a477328a70ee3
2021-11-25 12:15:35 +01:00
Vincent Ambo
4f1249e46f refactor(readTree): Move 'drvTargets' into readTree
This function is also generally useful for readTree consumers that
have the concept of subtargets.

Change-Id: Ic7fc03380dec6953fb288763a28e50ab3624d233
2021-11-23 14:42:08 +00:00
Vincent Ambo
5cad3f7b81 refactor(readTree): Move 'gather' into readTree itself
Discovering CI targets is relevant to all readTree consumers and this
logic is not TVL-specific.

Change-Id: I81ed3d3f76a6c36119f04bee28ca995a013f0e35
2021-11-23 14:42:08 +00:00
Vincent Ambo
a2be05faa4 refactor(readTree): Move copy of 'fix' into readTree
This is often used when bootstrapping a repository with readTree,
before lib is available. Having this definition in readTree is more
convenient than copy&pasting it around to callsites.

Change-Id: I6d5d27ed142bea704843fe289ad2674be8c4d360
2021-11-23 12:02:53 +00:00
Vincent Ambo
bc51bd99d9 refactor(readTree): Move 'restrictFolder' function into readTree
This is generally useful for readTree users and should be part of
readTree itself.

This is a move towards exposing several readTree-related features from
the library itself, in the future also including logic like 'gather'.

Note that this has a small functional change: In error messages of the
function, the notation for accessing Nix attributes is now used rather
than the Perforce-style `//` notation common in TVL.

For example, an error at `//web/tvl/logo` will produce `web.tvl.logo`
in the error message (which corresponds to the readTree attribute
itself).

This makes more sense for non-TVL consumers of readTree, as the
Perforce-style notation is custom to us specifically.

Change-Id: I8e199e473843c40db40b404c20d2c71f48a0f658
2021-11-23 14:39:54 +03:00
Vincent Ambo
ac41d89ba5 refactor(readTree): Flip argument order of argFilter
Since the filters return 'args', this makes nesting of filters more
readable.

Change-Id: I775252460e3e077cc6db2fab6f3948414a95ecbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3873
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-11-15 12:54:32 +00:00
Vincent Ambo
a9a1a46458 Note recommended use of 'or' in override patterns
This fixes #6
2021-11-14 16:28:37 +03:00
Vincent Ambo
2a098e081b Document binding of attribute set arguments using '@'
First and foremost this is being added because it was lacking, and
nix-1p strives to have fairly complete coverage of all useful
features.

Additionally, as pointed out by @nixinator in #6 there is some
surprising behaviour around how default arguments work in combination
with '@' and I thought this was worth noting.
2021-11-14 16:20:19 +03:00
sterni
bef44b2042 refactor(nix/tag): use throw over lib.assertMsg
Slightly prettier error message etc.

Change-Id: I9af4ec6ce337c65b5ca0820dc15a4e084832c529
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3542
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-10-24 20:25:10 +00:00
Vincent Ambo
edd8680e87 chore(nix/renderMarkdown): Move out of //users/tazjin
Change-Id: Ia5e873420b57436abead8e02cef3e5b7149de123
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3731
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-10-19 11:47:38 +00:00
sterni
2397fd8d56 feat(nix/sparseTree): allow specifying subtrees as relative paths
Passed strings will be treated as a relative path below the given root,
which is quite convenient when using depot.path by eliminating a lot of
repetition.

Change-Id: I3da6058094484f4a6ffbb84f89ad4472b502a00c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3704
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-10-10 10:03:03 +00:00
sterni
9dae5acf64 test(nix/readTree): test marking behavior of readTree
This ensures in a simple example that __readTree and __readTreeChildren
are populated correctly.

Change-Id: I69a46b2ddde0d1f9bf0dff1c4780f033ac8fc27a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3655
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-10-05 13:25:13 +00:00
sterni
66fa718ceb feat(nix/utils): expose pathType of symlink target
In order to make readTree import symlinked directories I've been looking
into how to detect if a symlink points to a directory (since this would
allow us to use symlinks for //nix/sparseTree). I've found a hack for
this:

    symlinkPointsToDir = path: isSymlink path &&
      builtins.pathExists (toString path + "/.")

Unfortunately it doesn't seem to be possible to distinguish whether the
symlink target does not exist or is a regular file.

Since it's possible, I thought might as well add this to
`pathType`. To make returning the extra information workable, I've
elected to use the attribute set layout used by `//nix/tag`. This
doesn't require us to depend anything (as opposed to yants), but gives
us pattern matching (via `nix.tag.match`) and also quite idiomatic
checking of pathTypes:

    pathType ./foo ? file
    (pathType ./foo).symlink or null == "symlink-directory"

Nonexistent paths are encoded like this:

    pathType ./foo ? missing

Of course we can't use this in readTree (since it must be zero
dependency), but we can easily inline this hack at some point.

Change-Id: I15b64a1ea69953c95dc3239ef5860623652b3089
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3535
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: tazjin <mail@tazj.in>
2021-10-02 18:24:49 +00:00
sterni
975f23d11b fix(nix/readTree): generate the correct marker for nix file children
This was a regression introduced in cl/3554.

Change-Id: I0721693a6eb1b28976b28499875812b1c3d1c910
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3654
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-30 16:41:06 +00:00
sterni
852d059e2f feat(nix/nint): accept attribute set with stdout, stderr and exit
This extends the calling convention for nint in a non-breaking way: If
the called script returns an attribute set instead of a string the
following is done:

* If the attributes `stdout` and/or `stderr` exist, their content (which
  must be a string currently) is written to the respective output.

* If the attribute `exit` exists, nint will exit with the given exit
  code. Must be a number that can be converted to an `i32`. If it's
  missing, nint will exit without indicating an error.

Change-Id: I209cf178fee3d970fdea3b26e4049e944af47457
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3547
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-18 11:56:22 +00:00
sterni
5dec982334 refactor(nix/runTestsuite): clean up runTestsuite
* goodAss wasn't used before. Simplify it to just return a boolean, so
  we can use it for partitionTests later.

* goodIt also returns unnecessary extra meta information which is not
  used. Cleaning that up makes the condition extremely small, so we can
  inline it into (what was) goodIts.

* goodIts is just called in one place, so we can inline it into res.

Change-Id: I70cf4fa3f61ce1467a2ee5319f841cdd42db6a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3548
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-09-17 11:47:41 +00:00
sterni
bef796d1d9 test(nix/utils): unit test storePathName
Change-Id: I4208cf9e3c5e1d922ee5b5bffd034e4ac6d0e2c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3543
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-17 11:00:09 +00:00
sterni
d7f60bcb04 feat(nix/readTree): record list of children added by readTree
This change adds a new attribute to readTree nodes, `__readTreeChildren`
which is a list of attribute names added to this node by readTree.

This is then used by `gather` for `ci.targets` to avoid evaluating
attributes unnecessarily. Especially since Nix is not as lazy as we'd
like when determining types (i. e. child ? __readTree needs to force
`child` even when it's not an attribute set), evaluating attributes
unnecessarily is sometimes problematic.

Change-Id: I0a98691d41f987e23ee7e9ba21fbe465da5fe402
2021-09-15 22:37:25 +00:00
sterni
529ac7fd12 refactor(nix/nint): use stdout().write_all() over print!()
Change-Id: Ic372442c5156f1e13fea28c1b3e57d64e89ba49b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3546
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-15 12:28:27 +00:00
sterni
391a186d27 refactor(nix/readTree): remove unused assertMsg function
Change-Id: I4048a59ca3daeb541a7294ce72eebe32982d852c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3553
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-15 12:20:01 +00:00
sterni
4ebab80b5a test(nix/utils): unit test path predicates
Change-Id: I85f38ef8f284c4accd1ea744f61358622785358d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3534
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-09-14 14:17:44 +00:00
sterni
987b9b551c feat(nix/runTestsuite): rebuild if test case list changes
If the result of the assertions changes for a successful test
suite (this happens if tests are reworded, added or removed), this
makes sure the no-op derivation is rebuilt.

This makes sure that test suites show up in buildkite on ocassions other
than channel bumps, since they are only added to the job list if their
`outPath` is missing nowadays (see cl/3427).

Change-Id: Ia1050cca5eeed8b7da84c40f6154b40760a3047f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3536
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-09-14 14:10:46 +00:00
sterni
3a8e8120de feat(nix/tag): add functions for querying a tag's name and value
Both are just trivial wrappers around assertIsTag to make these lookups
more ergonomic. This also allows us to demote assertIsTag to an
implemtation detail.

Change-Id: Ib6ba2a858f4839354a57b660042b418976c4b1d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3541
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-09-14 14:02:03 +00:00
sterni
1883159f92 docs(nix/tag): correct comment for assertIsTag
Change-Id: Id54cc3a1d7c83c6a94b9446c76b00ca2ce1ab48e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3540
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-09-14 14:02:03 +00:00
sterni
62fa36c9c2 fix(nix/buildLisp): preserve argv[0] in buildLisp.program
I *thought* I was being clever with the (cdr (member …)) call, but
somehow completely forgot that *posix-argv* and
*command-line-argument-list* are equivalent to argv, so they also
contain the program name as the first element. Dropping that made
argument parsing completely break down, so let's revert back to the
older solution which works quite well.

Change-Id: If7d3321cda0ca512bc8c23b6541ce390b81a3e24
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3538
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-12 21:39:49 +00:00
sterni
596344faf4 docs(nix/tag): fix example for discrDef
Change-Id: I8b7accb399448f57ace502642bd91fcfae416f15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3507
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-09-10 11:08:03 +00:00
sterni
318d10e608 chore(nint): move from //users/sterni to //nix
Since //web/bubblegum depends on nint, we need to move it to a non user
directory to conform with the policy established via cl/3434.
Note that this likely doesn't mean greater stability (which isn't
really implied in depot anyways), since I still would like to use a more
elaborate calling convention to allow for additional useful features.

Change-Id: I616f905d8df13e3363674aab69a797b0d39fdd79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3506
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-10 11:08:03 +00:00
sterni
23dd8067c5 feat(nix/sparseTree): get a directory with only selected children
Given a path (which points to a directory and a list of paths which
are below that path, build a “sparse” version of that directory, so
that it only contains the listed paths (and their children):

    $ nix-build -E 'with import ./. {}; nix.sparseTree ./. [
        ./default.nix
        ./nix/readTree
        ./nix/buildLisp
        ./third_party/nixpkgs
        ./third_party/overlays
      ]'
    /nix/store/0ynj0gc613fs6mfp9snqcvdj5gfxbdzg-sparse-depot
    $ lr -t 'type == d' result/
    result/
    result/nix
    result/nix/buildLisp
    result/nix/buildLisp/example
    result/nix/readTree
    result/nix/readTree/tests
    […]
    result/third_party
    result/third_party/nixpkgs
    result/third_party/overlays
    result/third_party/overlays/haskell
    result/third_party/overlays/haskell/patches
    result/third_party/overlays/patches

This is useful if a derivation depends on depot.path (e. g. if it wants
to import depot at runtime). Usually this means that on every depot
commit (or even worse, every change of .git on a local machine), this
derivation has to be rebuild. By using sparseTree you can instead depend
on a stripped down version of depot which only contains the bits you
actually depend on, avoiding unrelated rebuilds.

Change-Id: I127b108c8b177c657fb46786d0a6256516fd2c52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3503
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-09 15:57:58 +00:00
sterni
5f9c85a1b5 refactor(nix/readTree): use throw instead of assertMsg
This gives a slightly prettier error message and won't leak the error
message when builtins.tryEval is used. Currently an error message from
the tests is always part of the pipeline evaluation log.

Change-Id: I9b488a440368091ed42d707ba4124f592a64bd86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3502
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-09 12:52:16 +00:00
Vincent Ambo
b1f4b530ec feat(readTree): Support scoped import arguments
This makes it possible to override Nix builtins within a readTree
structure. Why would you want to do that, you might ask? Well ...

Change-Id: Icc9cb32e5db4a2eba370cf81769c642d237d4937
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3499
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-09 11:37:06 +00:00
Vincent Ambo
aedde913d1 refactor(readTree): Pass all readTree parameters as function args
Instead of having a mix of depot-passed args (for the filter) and args
to the readTree function itself, make everything a single attribute
set of arguments passed to the function.

This also makes it a bit easier to extend this in the future.

Change-Id: I633c1fc96026d137b451bb604ef92be32571a0f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3498
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-09-08 17:58:53 +00:00
Vincent Ambo
d67b9e3192 refactor(bufCheck): Assume bufCheck is run in depot checkout
Rather than copying the depot path into the store on each commit,
assume bufCheck is run in the depot checkout (which it is, in
Buildkite land).

Change-Id: I4a4af2e5b45acad2d18218e503880ee63b20f078
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3462
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-29 12:41:07 +00:00
Vincent Ambo
8b851956ad feat(readTree): Add support for path-dependent args filtering
Adds another argument to readTree itself which can be passed when
importing readTree (e.g. in our default.nix) to filter the arguments
passed to a target based on that target's location in the tree.

This is intentionally not yet mentioned in the docs, and also
intentionally implemented in such a way that the API surface of
readTree doesn't change. The reason for this is that I want to figure
out whether these filter functions are actually useful, e.g. within
depot by filtering user-folder passing, and then refactor the readTree
API to find a public way of exposing this as part of the readTree
function itself (and not its import).

Relates to b/143.

Change-Id: I2cdf09f67916527d2337f4bfb578749aeac51a6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3433
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-26 20:33:52 +00:00
sterni
2f44743824 docs(nix/buildLisp): document NIX_BUILDLISP_LISP_ARGS
Change-Id: I54294da08ee08bcf6cba6c792a2a2235b988a778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3422
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-26 11:26:52 +00:00
sterni
ea6ee1e219 docs(nix/buildLisp): move example further up
Make "Example" the second section again since it got a bit buried under
a lot of detailed documentation you won't necessarily need right away.

Change-Id: I481354d1761c590e5872dfce8c3cf9934e278673
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3421
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-08-26 11:26:52 +00:00
sterni
2dd8bc7977 fix(nix/buildLisp): prevent image loader from parsing arguments
CCL and SBCL create executables by dumping their image. As a
consequence, some part of the respective compiler is embedded in the
resulting executable which is executed and loads the image. For CCL and
SBCL this piece of software seems to unconditionally parse arguments
which can't be prevented since it happens before any lisp is loaded.

Luckily in both cases the parsing stops at `--`, so we can just pass
this via the wrapper — we just need to work around the problem that this
will of course be left in argv and confuse any later code. This can be
rectified by deleting everything prior to the first `--` in the global
argument list on startup in both cases.

In cases we do want to pass arguments to the image loader, we can use
the special NIX_BUILDLISP_LISP_ARGS environment variable which is
understood by the wrapper.

Note: This fix doesn't interfere with ECL since it is not using the
wrapper script at the moment.

Fixes b/136.

Change-Id: I3f95aa61e945e51428021ca18232ff15c923f870
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3357
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-08-26 11:26:52 +00:00
sterni
2fa32b563f docs(nix/buildLisp): document recent changes to buildLisp
Doing this in a separate CL to avoid having to track the intermediate
changes no one will ever see in documentation as well which would be
unnecessary effort.

* Multi-implementation support introduced in cl/3292 and refined in
  cl/3368 in terms of the user interface.

* Implementation specific srcs and deps introduced in cl/3321

* Implementation passthru attrs and rename from .sbcl -> .repl was done
  in cl/3359

* ECL added in cl/3297, CCL in cl/3350

Change-Id: Ia13f2aea4e7e091c00991fcbfc601de364413979
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3380
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
7df7cd6257 feat(nix/buildLisp): pass implementation description instead of name
Instead of using a string to refer to an internal set defined in
buildLisp, we just expose the relevant sets (as nix.buildLisp.sbcl,
nix.buildLisp.ecl, …) and receive them as the `implementation`
argument directly. This has several advantages:

* It becomes easier to extend buildLisp, even for downstream users:
  Since you can just pass your own set, there's nothing stopping you
  from adding support for another implementation in a downstream
  derivation without having to edit the buildLisp file in any way which
  is great if you're using e. g. builtins.fetchGit to import it.

* Users can mess with the implementation set by changing out some parts
  of it for customization purposes. Note that currently the sets use a
  lot of self-references which aren't even bound by a fix-point, so to
  make this work smoothly, we'd need to add some overriding mechanism.

* The buildLisp code becomes quite a bit clearer. Since we're now always
  dealing with the implementation set, the confusing distinction between
  `impl`, `impl.name` and `implementation` no longer exists. `impl` is
  now exclusively an abbreviation of `implementation` (we could make
  this more consistent in the future even).

Change-Id: I36d68069dd1315610b2f7159941507b465469b7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3368
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2021-08-24 22:00:15 +00:00
sterni
d7e70b1d72 feat(nix/buildLisp): add ccl
This adds support for Clozure's CL implementation to buildLisp. This is
quite trivial in comparison to ECL since SBCL and CCL have very similar
in how they work (so much so that CCL also suffers from b/136).

Also the similarities in the code actually added here are striking, so
I'll try to make an effort to reduce the code duplication in the
future.

To fix builds with CCL the following changes were made:

* //3p/lisp/nibbles: The double inclusion of the types.lisp file was
  fixed. CCL doesn't like double definitions and refuses to compile
  otherwise.

* //3p/lisp/physical-quantities: Update to a new bug fix release which
  contains a compilation fix for CCL.

* //3p/lisp/routes: apply a patch fixing the build which was previously
  failing due to a double definition.

* //3p/lisp/usocket: only depend on sb-bsd-sockets for SBCL and ECL, the
  latter of which seems to have a SBCL compatible implementation of the
  package.

* Conditionally include a few CCL-specific source files and add
  `badImplementation` entries for the remaining failures which are
  //fun/gemma (to be expected) and //web/panettone which fails with an
  incredibly vague message.

Change-Id: I666efdc39a0f16ee1bb6e23225784c709b04e740
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3350
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
02566cdcfb feat(nix/buildLisp): add ecl
Adds ECL as a second supported implementation, specifically a statically
linked ECL. This is interesting because we can create statically linked
binaries, but has a few drawbacks which doesn't make it generally
useful:

* Loading things is very slow: The statically linked ECL only has byte
  compilation available, so when we do load things or use the REPL it is
  significantly worse than with e. g. SBCL.

* We can't load shared objects via the FFI since ECL's dffi is not
  available when linked statically. This means that as it stands, we
  can't build a statically linked //web/panettone for example.

Since ECL is quite slow anyways, I think these drawbacks are worth it
since the biggest reason for using ECL would be to get a statically
linked binary. If we change our minds, it shouldn't be too hard to
provide ecl-static and ecl-dynamic as separate implementations.

ECL is LGPL and some libraries it uses as part of its runtime are as
well. I've outlined in the ecl-static overlay why this should be of no
concern in the context of depot even though we are statically linking.

Currently everything is building except projects that are using cffi to
load shared libaries which have gotten an appropriate
`badImplementations` entry. To get the rest building the following
changes were made:

* Anywhere a dependency on UIOP is expressed as `bundled "uiop"` we now
  use `bundled "asdf"` for all implementations except SBCL. From my
  testing, SBCL seems to be the only implementation to support using
  `(require 'uiop)` to only load the UIOP package. Where both a
  dependency on ASDF and UIOP exists, we just delete the UIOP one.
  `(require 'asdf)` always causes UIOP to be available.

* Where appropriate only conditionally compile SBCL-specific code and
  if any build the corresponding files for ECL.

* //lisp/klatre: Use the standard condition parse-error for all
  implementations except SBCL in try-parse-integer.

* //3p/lisp/ironclad: disable SBCL assembly optimization hack for all
  other platforms as it may interfere with compilation.

* //3p/lisp/trivial-mimes: prevent call to asdf function by substituting
  it out of the source since it always errors out in ECL and we hardcode
  the correct path elsewhere anyways.

As it stands ECL still suffers from a very weird problem which happens
when compiling postmodern and moptilities:
https://gitlab.com/embeddable-common-lisp/ecl/-/issues/651

Change-Id: I0285924f92ac154126b4c42145073c3fb33702ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3297
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: eta <tvl@eta.st>
2021-08-24 22:00:15 +00:00
sterni
0285ea7eac feat(nix/buildLisp): expose drvs built w/ the other implementations
For every implementation we support an extra passthru attribute with the
name of the implementation is created which points to a version of the
derivation built with that implementation. E. g. if we support CCL, ECL
and SBCL, third_party.lisp.alexandria would have:

* third_party.lisp.alexandria.sbcl
* third_party.lisp.alexandria.ecl
* third_party.lisp.alexandria.ccl

To make this possible, the REPL derivation which was called `sbcl`
originally has been renamed to `repl`.

Since some things won't build with all implementations, we introduce a
brokenOn argument which influences the meta.targets list that is
created, but won't prevent the passthru attrs from being created to
ease debugging failures.

Change-Id: Icd6af345143593fac30ded10deabf31172e5d48a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3359
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
ee6b2003fc feat(nix/buildLisp): implementation specific deps and srcs
Both the deps and srcs arguments may now have special “filter sets” in
the lists they receive as arguments. When building, buildLisp checks if
such sets either have a attribute named like the current implementation
or a "default" attribute. If yes, the set is replaced by the respective
attribute's value. If no, the set is removed from the list without
replacement.

This can be used to add elements for (a) specific implementation(s):

  { sbcl = buildLisp.bundled "sb-posix"; }

  { sbcl = ./sbcl/optional-sbcl.lisp; }

or to switch between files for different implementations:

  # If a implementation case is missing and no default set present,
  # no file will be added. Compilation will likely fail as a result.
  {
    ecl = ./tf-ecl.lisp;
    ccl = ./tf-ccl.lisp;
    sbcl = ./tf-sbcl.lisp;
  }

or to account for special behavior for a certain implementation:

  {
    ccl = ./ccl-quirk-impl.lisp
    default = ./ansi-impl.lisp;
  }

Change-Id: I082c3701d1f5063b92100bf336a83425471c269d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3321
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
acb5994e87 feat(nix/buildLisp): allow implementation-specifc bundled functions
By implementing a bundled function for an implementation, we can use a
custom one for a specific implementation. This is useful for
implementations like ECL where a require will be compiled as an
instruction rather than importing all new symbols into a dump, so using
the underlying static or shared object directly would be beneficial.

overrideLisp for bundled libraries now only allows overriding the name
and implementation arguments.

Change-Id: I9036b29157e8daa4d86ff87d603b044373711dbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3301
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
d344637fe2 refactor(nix/buildLisp): prepare multi implementation support
Concept is roughly:

* receive extra argument `implementation` that refers to the name of an
  implementation or rather an attribute in an internal attribute set
  telling buildLisp how to do certain build steps.

* We assume an implementation can execute lisp files as scripts and that
  we can implement the following main tasks in lisp:

  - Building a library (`genCompileLisp`)

  - Building an executable (`genDumpLisp`)

  - Loading a library dynamically (`genLoadLisp`)

  Based on that we can implement:

  - Running a test suite (`genTestLisp`)

  - A REPL preloaded with a libraries and their dependencies (`lispWith`)

  Additional attributes for implementing these parts genericly  are
  added as needed (`faslExt` and `runScript`).

* `genCompileLisp` no longer prints a shell script which concatenates
  the individual FASLs. Instead it does the step previously done by the
  shell script itself. In essence `genCompileLisp` now writes a lisp
  script which compiles and installs the library to build.
  This will allow us extra freedom for different implementations, e. g.
  for ECL we'll want to build a object file archive additionally to fasl
  files in order to be able to link proper executables.

* `genLoadLisp` and `genTestLisp` are almost generic (the former just
  sometimes would need to use different file extensions), but we
  integrate them into the implementation “API” to facilitate minor
  tweaks we need to do like the `fasc` extension for ECL's native FASL
  files.

Change-Id: I1b8ccc0063159638ec7af534e9a6b5384e750193
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3292
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
32de4cbd93 refactor(users/grfn/gws.fyi): implement isDirectory in pure nix
Another day, another import from derivation avoided by
builtins.unsafeDiscardStringContext!

Change-Id: I67274b1ba13ba980bb3346b22f2955c702aa3151
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3372
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 12:13:06 +00:00
sterni
59195e76fa docs(nix/buildLisp): mention drawback of non-parallel compilation
To be fair this hardly matters since SBCL is quite fast, but compiling
ironclad with ECL is quite the experience…

Change-Id: Ib89cc50e5d557acec51fdb085bcbdfc99736221e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3342
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-12 15:08:15 +00:00
sterni
0298ec862e fix(nix/buildLisp): remove misplaced makeOverridables
`makeOverriddable` doesn't work for bundled sbclWith as is because it
uses the `//` operator internally which doesn't work with the types
`bundled` and `sbclWith` accept as arguments (string and list
respectively).

What's more, `bundled` already uses `makeOverridable` and allows to
override the internal call to `library` via `overrideLisp`. For
`sbclWith` no such mechanism exists, but this seems to be no concern for
now: Using `overrideLisp` for this hasn't worked so far (and failed with
a _hideous_ evaluation error), so there doesn't seem to be any real
demand for this feature. Maybe a feature for another CL.

Change-Id: I0b2f34c00a2143cd66dd43a6b1b2880af997ee50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3296
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-10 14:31:52 +00:00
sterni
c8f521e96a refactor(nix/buildLisp): use passthru for extra attributes
Using passthru and appending the attributes via `//` have the same
effect with a subtle difference: In the latter case re-evaluating
the derivation when using the underlying `mkDerivation`'s
`overrideAttrs` will delete all appended attributes. Using
passthru at least preserves the attributes although the self
reference to the derivation in `passthru.sbcl` will become
outdated (unless updated manually).

Change-Id: I8b85009f386b9375b86a23fd50c4ec8c6a9dea7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3257
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-08-01 23:37:03 +00:00
Profpatsch
72924facae fix(nix/writers/rust): remove args override
We can be closed world, so let’s restrict the arguments to the subset
we need for now.

The existing override was wrong, in that `// args` would use the
arguments we already added, again. So instead of deliberating about
how to make this work right in all cases, we don’t need it, we trim
it.

Change-Id: I6443a0808b8bfd5e4db939b669c6afc741954db8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3057
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-05-17 23:00:57 +00:00
Profpatsch
713f066297 feat(users/Profpatsch/blog): implement blog engine lol
My horrible blog engine v0.0.0.0.1. Don’t judge.

Change-Id: I427b4e84d67aa49793cb7625e4d8ca2824f00943
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2456
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-04-30 21:47:42 +00:00
Profpatsch
eb41eef612 chore(nix): move rustSimple from users.Profpatsch.writers
I think it’s solid enough to use in a wider context.

Change-Id: If53e8bbb6b90fa88d73fb42730db470e822ea182
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3055
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-04-24 10:23:55 +00:00
Profpatsch
7e888c3c7b feat(nix): add basic netstring nix generation functions
Moving to toplevel so I can use them with `runExecline`. They should
be pretty atomic, and are proven to work (tests are still in my user
dir, since they test the producers indirectly via the python parser
and I don’t want to pull it out right now).

Change-Id: Id0baa3adcb2ec646458a104c7868c2889b8c64f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3054
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2021-04-24 09:54:34 +00:00
Profpatsch
2224731e0d feat(nix/buildGo): add the import position to errmsg
When a foreign dep is missing a dependency, it is good to have a
context.

e.g. the `github.com/charmbracelet/bubblegum` package has a lot of
dependencies that are only used in its `examples/` dir; this is not
obvious, unless we also print where the imports come from.

New error message:

```
error: missing foreign dependency 'github.com/containerd/console' in 'github.com/charmbracelet/bubbletea, imported at /nix/store/0cry4sg6bzxqwk5zl2nxhas6k5663svg-source/tea.go:22:2'
```

Change-Id: If34a3c62b9d77d4aea108b5e011e16fbd03e8554
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-23 16:35:57 +00:00
sterni
f88ac5c0b5 feat(nix/utils): add storePathName, a more generic baseNameOf
This is a wrapper around baseNameOf which also can deal with
derivations. Added to //nix/utils since I've found myself introducing an
ad-hoc implementation of this for both //web/bubblegum and //nix/buildC.

Change-Id: I2fcd97a150d6eda21ab323fa0d881ff7442a892e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3049
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-20 11:00:52 +00:00
Vincent Ambo
f520bd40ca refactor: Replace 'depotPath' with 'depot.path'
Instead of having two ways of accessing the path to the depot (one of
which was stuttering, depot.depotPath) we settle on only one:
depot.path.

This was mostly used for NixOS module imports.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I2c0db23383fc34f6ca76baaad4cc4af2d9dfae15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2962
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-12 21:55:07 +00:00
Vincent Ambo
a559135970 refactor(readTree): Initialise repo roots without recursing
Plumbs an additional internal argument through readTree that indicates
whether the top-level of a tree is being read, and avoids recursing
into itself in that case. This changes the externally visible
behaviour of readTree (it is now expected to be called a level higher
than previously).

This allows us to reduce the amount of boilerplate needed to bootstrap
the TVL repository (by not having to specify the individual folders
that need to be read).

For reasons related to an infinite recursion we could not (be bothered
to) debug, the top-level `config` key (which held the attribute set
passed on by readTree) has been removed. This is not needed, as it is
already passed on by readTree ...

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Id6e39b57b2f5b3473c4b695a72dd1d01fcfb7a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2961
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-04-12 21:55:07 +00:00
Vincent Ambo
f39b5a8a0d docs(readTree): Add SPDX license header on readTree
Change-Id: I9a6eafa7d4bb3d590dfa35d368adfd25aeed7f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2936
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2021-04-12 13:56:44 +00:00
sterni
8d4b2f3d54 refactor(sterni): use pkgs over third_party to import from nixpkgs
This should ease migrating to a distinction between depot.third_party
and pkgs (as in nixpkgs) in the future.

Ref cl/2910, b/108.

Change-Id: I53a854071fddd7c0d0526cc4c5b16998202082c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2913
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-10 11:40:18 +00:00
sterni
811fff2d8b feat(nix/buildManPages): infra for generating man page dirs from nix
Very simple builder which builds (optionally) gzipped man pages from a
list of attrsets and links them into a common man directory with the
correct layout, so it should be installable immediately.

Additionally runs mandoc -T lint, but by default only for informational
purposes as it is very strict and some things are almost never true (for
example all Xrs being present in the respective directory).

buildManPages.single exposes the internal builder for a single,
optionally gzipped man page from a nix attrset.

Change-Id: I43fce011716f4a7cc80521f222800ca99ba54060
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2654
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-03-31 23:06:10 +00:00
Profpatsch
d0b635f6f2 fix(nix/runTestsuite): use s6-portable-utils
Same as 221698c603dcb318c609b4d21cb2a9fada44a14c

We had a bunch of instances of
https://github.com/NixOS/nix/issues/2176,
where nix would exit with a “killed by signal 9” error.

According to Eelco in that issue, this is perfectly normal behaviour
of course, and appears if the last command in a loop closes `stdout`
or `stdin`, then the builder will SIGKILL it immediately. This is of
course also a perfectly fine error message for that case.

It turns out that mainly GNU coreutils exhibit this behaviour …

Let’s see if using a more sane tool suite fixes that.

Change-Id: Iaf9e542952ca36c02208a3f067f575ba978272b4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2663
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2021-03-26 11:01:20 +00:00
Vincent Ambo
d34c527372 refactor: Replace some uses of builtins.toFile with pkgs.writeText
I'm looking at removing some of these because they can cause
unnecessary build steps during CI pipeline generation.

Change-Id: I84742968918090c050d2eedab8a1b42692632a42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2655
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-03-25 19:14:36 +00:00