Commit graph

93 commits

Author SHA1 Message Date
Aspen Smith
bab3db2fa4 fix(aspen/system): Fix and reenable system build
Change-Id: I36f7a772e70be8711c03150b4b6dfcd6d4e299f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11371
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-04-07 14:19:01 +00:00
Florian Klink
f7cdbbef45 chore(users/flokli/nixos): restore archeology-ec2 build
Temporarily remove parquet-tools, so this builds again.

Can be re-added once https://github.com/NixOS/nixpkgs/pull/301032 landed
upstream.

Change-Id: Ie74f014eb8158d5f529a5f1c55788a4edc5c805d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11347
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-04-03 11:19:34 +00:00
sterni
73b1f0407b chore(3p/sources): bump channels & overlays – xz edition
Update all 3p/sources as we do normally except

- agenix which is still pinned to 0.15.0

- nixpkgs (unstable) which we bump to the HEAD of the staging-next
  branch. This branch includes the downgrade of xz from 5.6.1 to
  5.4.6 (https://github.com/nixos/nixpkgs/commit/d6dc19adbd). It
  also includes the second haskell-updates rotation with GHC 9.6.4
  which contains a few build fixes that seem to be required to get
  our Haskell targets to work.

Note that this only reverts xz to a version that doesn't contain the now
known backdoor (CVE-2024-3094) which may or may not actually affect
NixOS. Additionally reverting to a version before the malicious
contributor's involvement may be difficult, but prudent:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024

Changes required by the updates:

- //3p/overlays/haskell:

  - Update ihp-hsx to latest master to fix build with Stackage LTS 22.
  - Update tmp-postgres to latest master to work around failure with
    ansi-wl-pprint >= 1.
  - Patch punycode for mtl >= 2.3.

- //users/Profpatsch:

  - Clean up some warnings, mostly about unused dependencies
  - my-prelude: Fix build with ghc-boot-9.6.4
  - cas-serve: Use crypton over unmaintained cryptonite
  - ical-smolify: skip in ci, iCalendar would require heavy patching to
    work with Stackage LTS 22.

- //users/{wpcarro,aspen,flokli}:

  Disable home-manager / nixos configuration builds that seem to have
  transient failures that should disappear as we move away from
  staging-next and closer to an actual channel release.

Change-Id: I5cca48e101041c3aedc1d9932dbca2cac885fcc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11289
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-31 22:27:04 +00:00
Aspen Smith
82ecd61f5c chore(users): grfn -> aspen
Change-Id: I6c6847fac56f0a9a1a2209792e00a3aec5e672b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10809
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2024-02-14 19:37:41 +00:00
Florian Klink
18d582f96b feat(third_party/nixpkgs): allow setting crossSystem
This allows cross-compiling depot targets to other architectures, so
`nix-build --argstr crossSystem aarch64-linux -A tvix.nar-bridge`
will emit a cross-compiled aarch64-linux binary.

There's still some other cross-related issues in crate2nix to sort out
for crate2nix builds, but this CL can already land.

Co-Authored-By: raitobezarius <tvl@lahfa.xyz>
Change-Id: I467d49d125dd707a4142bfde18eea4a1c3afaf70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9755
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-30 10:43:52 +00:00
Vincent Ambo
110328232d chore(3p/sources): bump channels & overlays (2023-08-21)
Included changes:

* tvix/eval: enable some lang tests on nix_latest

  Nix 2.16 contains some breaking language changes which Tvix does
  not yet implement, but the existing tests for them are now passed by
  Nix 2.16 (but not yet by Tvix).

* tvix/eval: disable a lang test on nix_latest

  In Nix 2.17, the identifier formatting test fails because some
  behaviour changed. We have not investigated further yet.

* 3p/overlays: use version of ihp-hsx that works with GHC 9.4

  Originally from the separate cl/9185.

* top-level: introduce a mechanism to exclude build targets from CI in
  the top level. This fixes b/296.

* users/grfn: disable builds of xanthous (and dependents) until the
  CLs fixing its build are submitted

* 3p/overlays: build nixos-option against Nix 2.15, the only version
  with which it builds

* 3p/overlays: bump tdlib to 1.8.16

Change-Id: Ia377f39dbdb08ac45ff830a615e64babc091e5ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9125
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-09-06 13:38:15 +00:00
Vincent Ambo
1aa8b62ae3 chore(depot): skip builds for broken packages
Considers `note.meta.broken` in addition to `node.meta.ci.skip`,
because people might be inheriting this from something, or using an
extremely complicated package mechanism in which setting their own
meta attributes is ugly/non-trivial.

Change-Id: Ia93cfbba6af545e370a7c7dbd51f24e1df68bbf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9270
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-09-06 08:40:44 +00:00
Vincent Ambo
e43569d31d fix: make depot-gcroot derivation a plain list of out paths
This just creates a text file (similar to the propagated deps file)
which lists the outpaths of all targets.

It's unclear why the previous fix didn't work, but it was done solely
based on docs not on reading code. In general we don't really need a
setup hook though and nobody remembers why that was the solution we
used anyways, so lets remove it.

Change-Id: I2a945925f4fa56b272aa44c29229b6a1aefa80e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8228
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-08 16:59:08 +00:00
Vincent Ambo
252d8b10db fix: use depsTargetTargetPropagated for depot's gcroot
We've been seeing CI failures after a recent nixpkgs bump which only
occur on canon *after* a CL has been merged. In these CI failures, the
`ci.gcroot` attribute is built to "anchor" the latest canon build, but
fails because it tries to execute random non-executable files (e.g.
the sqlite database produced by //corp/russian/data-import).

From what I can tell, there has been a recent change in nixpkgs where
makeSetupHooks' arguments were restructured. What was previously
`deps` is now `propagatedBuildInputs`, and
`depsTargetTargetPropagated`.

The latter is supposed to be used for non-executable targets,
according to the docs.

This commit changes the entire set to use that flag, as we don't
actually want any executable setup hook - just a gcroot.

Change-Id: Ib258c4f3b09d746c3d000d1c7480d2b4101ab75e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8227
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-03-08 10:28:04 +00:00
Vincent Ambo
53d59ab1f6 feat(3p/overlays): Build overlaid packages in CI explicitly
Change-Id: I2f9e4f6f5f0193456f773e62ce5b8163b253de0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5489
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-26 15:43:45 +00:00
sterni
28dacca91d feat: make builtins.currentSystem illegal in depot
Since cl/7260 has eliminated all uses of builtins.currentSystem
in the main evaluation path of depot, we can ensure that we use
localSystem consistently in the future by making it impossible
to access the value of builtins.currentSystem in readTree nodes.

Change-Id: I6ded54021c42fcf31a80268149179f95f22bad88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7261
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-11-12 21:54:45 +00:00
Vincent Ambo
6cfa6bb59e feat: Add top-level system parameter
This has come up a couple of times. This way system is passed to all
derivations. Maybe we can do something useful with it.

Change-Id: Ia7dfcffbc82abbd3128342a8971a3861865be713
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5832
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-06-03 23:27:03 +00:00
Vincent Ambo
aa122cbae7 style: format entire depot with nixpkgs-fmt
This CL can be used to compare the style of nixpkgs-fmt against other
formatters (nixpkgs, alejandra).

Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: cynthia <cynthia@tvl.fyi>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: grfn <grfn@gws.fyi>
2022-01-31 16:11:53 +00:00
Vincent Ambo
a874f5df78 fix: Fix build inclusion logic
The `or` statement is a *single* expression, leading to an incorrect
negation.

Change-Id: I46fa85c22cf145b9204bf26b85b963494b069d06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5013
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-01-20 10:13:52 +00:00
Vincent Ambo
effcd31651 refactor: Replace 'meta.ci' -> 'meta.ci.skip'
We need 'meta.ci' to be an attribute set for new CI features.

Change-Id: I83d04e2d74e42e49fe739b049ee4ba799f6d5d86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5007
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-01-20 07:40:18 +00:00
Vincent Ambo
0779f96687 feat(nix/buildkite): Check target map of parent to determine skips
This changes the logic for build pipeline generation to inspect
an (optional) parentTargetMap attribute which contains the derivation
map of a target commit.

Targets that existed in a parent commit with the same drv hash will be
skipped, as they are not considered to have changed.

This does not yet wire up any logic for retrieving the target map from
storage, meaning that at this commit all targets are always built.

The intention is that we will have logic to fetch the target
map (initially from Buildkite artefact storage), which we then pass to
the depot via externalArgs when actually generating the pipeline.

Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 11:49:01 +00:00
Vincent Ambo
4ab061ed98 fix(ops/pipelines): Realise anchor derivation for rooting
Turns the anchor derivation into something that can actually be
built (a call creating a propagated build inputs file), and builds it.

This should fix the anchoring logic we have on canon.

Change-Id: If6a7662b82e2e396388980f65e332cf67a45b46e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4763
Tested-by: BuildkiteCI
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-02 22:25:42 +00:00
zseri
f791539987 feat(3p/nixpkgs): allow injecting user nixpkgs config
use case: system-wide 'testing' usage of content-addressed derivations

Change-Id: I1f63ddf679da7d53ff0d8a851642dd081a70fe55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4551
Tested-by: BuildkiteCI
Reviewed-by: zseri <zseri.devel@ytrizja.de>
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: zseri <zseri.devel@ytrizja.de>
2021-12-28 09:34:34 +00:00
sterni
faca687dda fix: make sure depot.path is named "depot" (again)
It is a reasonable enough assumption that depot.path will have the store
path name "depot". In the past this used to be the case, since most
people would checkout the repository as "depot" (funnily enough in the
past, you wouldn't have been able to substitute some things if you used
a different name). When we started to use cleanSource for depot.path the
default name "source" would be assigned, breaking e. g. the assumptions
of //web/todolist. We now set to "depot" statically.

Fixes: b/162

Change-Id: I72fdb488e045bdaee80d1df0334b026060dcd116
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4381
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-16 18:11:02 +00:00
Vincent Ambo
5cad3f7b81 refactor(readTree): Move 'gather' into readTree itself
Discovering CI targets is relevant to all readTree consumers and this
logic is not TVL-specific.

Change-Id: I81ed3d3f76a6c36119f04bee28ca995a013f0e35
2021-11-23 14:42:08 +00:00
Vincent Ambo
a2be05faa4 refactor(readTree): Move copy of 'fix' into readTree
This is often used when bootstrapping a repository with readTree,
before lib is available. Having this definition in readTree is more
convenient than copy&pasting it around to callsites.

Change-Id: I6d5d27ed142bea704843fe289ad2674be8c4d360
2021-11-23 12:02:53 +00:00
Vincent Ambo
bc51bd99d9 refactor(readTree): Move 'restrictFolder' function into readTree
This is generally useful for readTree users and should be part of
readTree itself.

This is a move towards exposing several readTree-related features from
the library itself, in the future also including logic like 'gather'.

Note that this has a small functional change: In error messages of the
function, the notation for accessing Nix attributes is now used rather
than the Perforce-style `//` notation common in TVL.

For example, an error at `//web/tvl/logo` will produce `web.tvl.logo`
in the error message (which corresponds to the readTree attribute
itself).

This makes more sense for non-TVL consumers of readTree, as the
Perforce-style notation is custom to us specifically.

Change-Id: I8e199e473843c40db40b404c20d2c71f48a0f658
2021-11-23 14:39:54 +03:00
Vincent Ambo
d295ba2a4e chore(depot): Restrict access to targets from //corp
This folder is used for some TVL corp stuff, like the website and
maybe some documents and future projects, that are not under the same
license as the rest of depot.

To avoid accidental licensing issues, access to it is restricted to
other stuff within //corp.

In general, TVL corp projects *should* also be free software and live
outside of //corp - the folder is primarily intended for stuff that is
relevant to the company operations (also for the sake of
transparency).

Change-Id: I15e7e72e82d8ac1c875899f16becd731f64f6b3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3875
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-11-15 13:28:42 +00:00
Vincent Ambo
ac41d89ba5 refactor(readTree): Flip argument order of argFilter
Since the filters return 'args', this makes nesting of filters more
readable.

Change-Id: I775252460e3e077cc6db2fab6f3948414a95ecbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3873
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-11-15 12:54:32 +00:00
Vincent Ambo
dc8d3e869d refactor(depot): Generalise folder restriction readTree filter
This refactors the readTree filter which disallows access to //users
from outside of //users into a reusable function.

The only change in functionality is that the error message has changed
slightly. I thought it is useful to keep the message consistent (i.e.
always including a path), thus only a part of the error is templated
in (describing the reason for why a specific sub path is unavailable).

Change-Id: I30ad38b2677be5aa502c753c8c71e7ba3efc87be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3872
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-11-15 12:26:41 +00:00
Vincent Ambo
0939d4bf05 refactor(depot): Import depot.path using lib.cleanSource
This imports a copy of the depot as `depot.path` without including the
.git directory and other stuff that isn't tracked in the tree.

Significantly reduces thie amount of data copied into the store.

Change-Id: I567c0f969d1cea81d121588548f6db627c8f3432
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3601
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-09-20 08:30:04 +00:00
sterni
d7f60bcb04 feat(nix/readTree): record list of children added by readTree
This change adds a new attribute to readTree nodes, `__readTreeChildren`
which is a list of attribute names added to this node by readTree.

This is then used by `gather` for `ci.targets` to avoid evaluating
attributes unnecessarily. Especially since Nix is not as lazy as we'd
like when determining types (i. e. child ? __readTree needs to force
`child` even when it's not an attribute set), evaluating attributes
unnecessarily is sometimes problematic.

Change-Id: I0a98691d41f987e23ee7e9ba21fbe465da5fe402
2021-09-15 22:37:25 +00:00
sterni
318d10e608 chore(nint): move from //users/sterni to //nix
Since //web/bubblegum depends on nint, we need to move it to a non user
directory to conform with the policy established via cl/3434.
Note that this likely doesn't mean greater stability (which isn't
really implied in depot anyways), since I still would like to use a more
elaborate calling convention to allow for additional useful features.

Change-Id: I616f905d8df13e3363674aab69a797b0d39fdd79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3506
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-10 11:08:03 +00:00
Vincent Ambo
8b4e0dbf2e feat(depot): Prevent imports from NIX_PATH within the depot
Provided without further comment.

Change-Id: I19d14530243c72d49b14526f769b964b06959db8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3500
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-09-09 11:37:06 +00:00
Vincent Ambo
aedde913d1 refactor(readTree): Pass all readTree parameters as function args
Instead of having a mix of depot-passed args (for the filter) and args
to the readTree function itself, make everything a single attribute
set of arguments passed to the function.

This also makes it a bit easier to extend this in the future.

Change-Id: I633c1fc96026d137b451bb604ef92be32571a0f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3498
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-09-08 17:58:53 +00:00
Vincent Ambo
1af8bf2b02 fix(ops): Add another target that needs a //users exception
Change-Id: I3edede16890893e14727283a12c7fa3310cb4dc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3441
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-08-26 21:04:34 +00:00
Vincent Ambo
df566e9b4a style(depot): Use depot path format for args filter errors
As pointed out by sterni, we don't need lib for this.

Change-Id: Ide9719641098f770a098a938d047afa0dcb5dc6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3439
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-08-26 20:54:20 +00:00
Vincent Ambo
e1f83cc086 feat(depot): Disallow access to //users from outside of it
Code under this depot path is essentially unstable and potentially
unreviewed - this is a good thing (people can play around with cursed
stuff all they want), but we should not make the rest of the
repository depend on any of it.

Any cursed things that are required outside of users can be moved to a
different depot path if people agree with that.

Change-Id: I46a34a0e9662069c01b43d9a653e5545e325e587
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3434
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-08-26 20:39:07 +00:00
Vincent Ambo
f520bd40ca refactor: Replace 'depotPath' with 'depot.path'
Instead of having two ways of accessing the path to the depot (one of
which was stuttering, depot.depotPath) we settle on only one:
depot.path.

This was mostly used for NixOS module imports.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I2c0db23383fc34f6ca76baaad4cc4af2d9dfae15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2962
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-12 21:55:07 +00:00
Vincent Ambo
a559135970 refactor(readTree): Initialise repo roots without recursing
Plumbs an additional internal argument through readTree that indicates
whether the top-level of a tree is being read, and avoids recursing
into itself in that case. This changes the externally visible
behaviour of readTree (it is now expected to be called a level higher
than previously).

This allows us to reduce the amount of boilerplate needed to bootstrap
the TVL repository (by not having to specify the individual folders
that need to be read).

For reasons related to an infinite recursion we could not (be bothered
to) debug, the top-level `config` key (which held the attribute set
passed on by readTree) has been removed. This is not needed, as it is
already passed on by readTree ...

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Id6e39b57b2f5b3473c4b695a72dd1d01fcfb7a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2961
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-04-12 21:55:07 +00:00
Vincent Ambo
d7b89df748 feat(ops/pipelines): Add gcroots for depot builds on canon
Adds a conditional build step that only runs on the canon branch, and
only if 🦆 (the status reporting step) succeeds, which creates a
new Nix GC root for all depot targets named `depot-canon`.

In practice this might be a bit racey, as canon builds are not
guaranteed to succeed in order (though it is likely). This shouldn't
matter much in practice: We only want to prevent rebuilds of the whole
world.

This fixes b/102

Change-Id: Id3d0bf4158bffcb1ed6929888a29d31609b6ece1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2904
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-11 20:09:53 +00:00
Vincent Ambo
6c4ece917d feat(3p/nixpkgs): Plumb through externalArgs.nixpkgsBisectPath
This lets the import of the depot root accept an additional argument
called `externalArgs`, which can be used to pass additional arguments
into a depot package set.

This is used in //third_party/nixpkgs for replacing the source of the
nixos-unstable channel with a path. With this we can bisect the
nixpkgs used in third_party easily.

Change-Id: I4f65eb3d6b521ed9f437649b7b068f1e6ab8210f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2925
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-11 11:08:49 +00:00
Vincent Ambo
473604f567 refactor: Move nixpkgs attribute to third_party.nixpkgs
Please read b/108 to make sense of this.

This gets rid of the explicit list of exposed packages from nixpkgs,
and instead makes the entire package set available at
`third_party.nixpkgs`.

To accommodate this, a LOT of things have to be very slightly shuffled
around. Some of this was done in already submitted CLs, but this
change is unfortunately still quite noisy.

Pay extra attention to:

* overlay-like functionality that was partially moved to actual
  overlays (partially as in, the minimum required to get a green
  build)

* modified uses of the package set path, esp. in NixOS systems

Special notes:

* xanthous has been disabled in CI because of issues with the Haskell
  overlay
* //third_party/nix has been disabled because of other unclear
  dependency issues

Both of these will be tackled in a followup CL.

Change-Id: I2f9c60a4d275fdb5209264be0addfd7e06c53118
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2910
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-04-10 21:18:55 +00:00
Vincent Ambo
d02eb5f8bf fix(tvix): Add //tvix to readTree roots
Change-Id: I7d0e81d6a88f351c7451454f5d506d708e7ed9ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2710
Tested-by: BuildkiteCI
Reviewed-by: adisbladis <adisbladis@gmail.com>
2021-03-31 19:23:39 +00:00
Vincent Ambo
aaaadbbcdc chore: Remove //overrides from depot
This mechanism wasn't actually used for anything other than a minor
external compatibility thing for buildGo.nix, which can also be solved
with a function parameter.

This breaks //fun/gemma because it means that the elmPackages used to
build it are no longer reachable from depot. We'll sort this out later.

Change-Id: I1bf2240435e869cdc4e99bdd1a138fdd2e76f96e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2646
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-23 23:55:49 +00:00
sterni
e91d5e4e61 fix(config): remove ciBuilds inherit
The ciBuilds attribute seems to no longer exist and it breaks the
evaluation of the config attribute. It's only appearance was in
besadii which doesn't actually use the attribute.

Removing the ciBuilds inherit fixes these issues.

Change-Id: Ibbf3413ba6efe10ad868cf57cf0711d574860f97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2487
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-02-06 17:54:18 +00:00
Vincent Ambo
9c482d6238 feat(ci): Add subtarget support for builds
We have naturally evolved a distinction between logical and physical
targets.

Physical targets are those which correspond directly to a tree
location on disk and can be built with `-A path.to.files`, while
logical targets are those that are exported from within an expression
but do not have a corresponding file on disk.

This change adds support for exporting logical targets from any tree
location by adding a `meta.targets` attribute containing keys into
itself, which will be consumed by the CI target gathering logic and
included in the generated pipeline.

Note that the labels for subtargets are syntactically different to
emphasise that they do not correspond to a file location. For example,
this change enables 'ops.nixos.whitbySystem' as a subtarget, which is
labeled in CI as `ops/nixos:whitbySystem`.

Change-Id: Ied09647a62c2ba98e3914548e3742ad422c63ecf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1893
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-31 23:14:11 +00:00
Vincent Ambo
61d2d2d503 feat(ops/pipelines): Dynamically generate CI pipeline from targets
Create the pipeline by outputting a file that contains nix-build
invocations for each target's *derivation path*.

Each invocation has a generated Nix expression passed to it with `-E`
which fetches the correct target from the tree while correctly
handling targets with strange characters (such as in Go-packages).

This makes it possible to run target-level granular pipelines. We're
getting somewhere!

Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-31 23:14:11 +00:00
Vincent Ambo
4ff9d5dee8 feat: Implement automatic CI target detection for the depot
Automatically walk the entire depot tree and pick out things that are
"buildable", then include them in the attribute `ci.targets` (which is
now also the target for CI builds).

A long time ago, in a land far away, we (well, I, at the time) had a
prototype of this which ran into constant issues with infinite
recursions while trying to walk the tree. In fact, this is why
readTree originally gained the `__readTree`-attribute which marks
things that were imported automatically.

Based on some code edef whipped up earlier (with the breakthrough
being that we also add the attribute to top-level folders, which
suddenly resolves a whole bunch of problems), I've now implemented
this actually working version.

At the moment all builds still happen as one big bag of builds, but at
some point we will granularise this.

Change-Id: I86f12ce7f63dae98e7e5c6646a4e9d220de783f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1854
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-26 23:49:32 +00:00
Vincent Ambo
bc5d4672aa fix: Remove ./presentations from top-level
This folder doesn't exist, it's part of my user folder now. We didn't
notice because nothing is walking the tree.

Change-Id: Idc6f20a8e4806a158c598fd63d381ab07934be1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1843
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
dc07977866 chore(ops): Clean up old GCP infrastructure files
This removes almost all of the GCP-infrastructure leftovers from my
previous setup.

The DNS configuration is retained, but moves to my user folder
instead.

Change-Id: I1867acd379443882f11a3c645846c9902eadd5b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/782
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-06-29 21:24:49 +00:00
Vincent Ambo
1640d9d145 refactor(ci-builds): Split up CI builds into multiple buckets
These categories separate CI targets, which hopefully avoids the
out-of-space errors we have been seeing on Sourcehut.

The sets of CI build targets are made available in the depot itself so
that besadii can be updated to create a new build for each target
group.

For convenience, 'ciBuilds' contains an '__allTargets' attribute which
combines the contents of each target batch - this makes it possible to
still invoke a build for everything by using:

  nix-build -A ciBuilds.__allTargets

Note: Some targets that were previously built in CI aren't anymore,
most importantly my NixOS systems which don't fit on Sourcehut.

Change-Id: Ia15ed7b743c8add51ae08ce0827a0ddfacd637e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/570
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-24 02:38:21 +00:00
Vincent Ambo
268729083e refactor(ops/nixos): Move my NixOS configurations to //users/tazjin
NixOS modules move one level up because it's unlikely that //ops/nixos
will contain actual systems at this point (they're user-specific).

This is the first users folder, so it is also added to the root
readTree invocation for the repository.

Change-Id: I546c701145fa204b7ba7518a8a56a783588629e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/244
Reviewed-by: tazjin <mail@tazj.in>
2020-06-13 23:52:35 +00:00
Vincent Ambo
9658e96a87 feat: Add 'depotPath' to depot root import
This is useful for things like including NixOS modules in
configurations without creating long and error-prone relative paths.

Change-Id: I4a5ebb1a0e5adf90b6bc50e884db453e12461001
Reviewed-on: https://cl.tvl.fyi/c/depot/+/243
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-13 23:52:35 +00:00
Vincent Ambo
a97094cc3e chore: Add //net to local package trees 2020-06-08 00:31:52 +01:00