Commit graph

66 commits

Author SHA1 Message Date
sterni
318d10e608 chore(nint): move from //users/sterni to //nix
Since //web/bubblegum depends on nint, we need to move it to a non user
directory to conform with the policy established via cl/3434.
Note that this likely doesn't mean greater stability (which isn't
really implied in depot anyways), since I still would like to use a more
elaborate calling convention to allow for additional useful features.

Change-Id: I616f905d8df13e3363674aab69a797b0d39fdd79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3506
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-10 11:08:03 +00:00
Vincent Ambo
8b4e0dbf2e feat(depot): Prevent imports from NIX_PATH within the depot
Provided without further comment.

Change-Id: I19d14530243c72d49b14526f769b964b06959db8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3500
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-09-09 11:37:06 +00:00
Vincent Ambo
aedde913d1 refactor(readTree): Pass all readTree parameters as function args
Instead of having a mix of depot-passed args (for the filter) and args
to the readTree function itself, make everything a single attribute
set of arguments passed to the function.

This also makes it a bit easier to extend this in the future.

Change-Id: I633c1fc96026d137b451bb604ef92be32571a0f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3498
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-09-08 17:58:53 +00:00
Vincent Ambo
1af8bf2b02 fix(ops): Add another target that needs a //users exception
Change-Id: I3edede16890893e14727283a12c7fa3310cb4dc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3441
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-08-26 21:04:34 +00:00
Vincent Ambo
df566e9b4a style(depot): Use depot path format for args filter errors
As pointed out by sterni, we don't need lib for this.

Change-Id: Ide9719641098f770a098a938d047afa0dcb5dc6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3439
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-08-26 20:54:20 +00:00
Vincent Ambo
e1f83cc086 feat(depot): Disallow access to //users from outside of it
Code under this depot path is essentially unstable and potentially
unreviewed - this is a good thing (people can play around with cursed
stuff all they want), but we should not make the rest of the
repository depend on any of it.

Any cursed things that are required outside of users can be moved to a
different depot path if people agree with that.

Change-Id: I46a34a0e9662069c01b43d9a653e5545e325e587
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3434
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-08-26 20:39:07 +00:00
Vincent Ambo
f520bd40ca refactor: Replace 'depotPath' with 'depot.path'
Instead of having two ways of accessing the path to the depot (one of
which was stuttering, depot.depotPath) we settle on only one:
depot.path.

This was mostly used for NixOS module imports.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I2c0db23383fc34f6ca76baaad4cc4af2d9dfae15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2962
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-12 21:55:07 +00:00
Vincent Ambo
a559135970 refactor(readTree): Initialise repo roots without recursing
Plumbs an additional internal argument through readTree that indicates
whether the top-level of a tree is being read, and avoids recursing
into itself in that case. This changes the externally visible
behaviour of readTree (it is now expected to be called a level higher
than previously).

This allows us to reduce the amount of boilerplate needed to bootstrap
the TVL repository (by not having to specify the individual folders
that need to be read).

For reasons related to an infinite recursion we could not (be bothered
to) debug, the top-level `config` key (which held the attribute set
passed on by readTree) has been removed. This is not needed, as it is
already passed on by readTree ...

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Id6e39b57b2f5b3473c4b695a72dd1d01fcfb7a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2961
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-04-12 21:55:07 +00:00
Vincent Ambo
d7b89df748 feat(ops/pipelines): Add gcroots for depot builds on canon
Adds a conditional build step that only runs on the canon branch, and
only if 🦆 (the status reporting step) succeeds, which creates a
new Nix GC root for all depot targets named `depot-canon`.

In practice this might be a bit racey, as canon builds are not
guaranteed to succeed in order (though it is likely). This shouldn't
matter much in practice: We only want to prevent rebuilds of the whole
world.

This fixes b/102

Change-Id: Id3d0bf4158bffcb1ed6929888a29d31609b6ece1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2904
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-11 20:09:53 +00:00
Vincent Ambo
6c4ece917d feat(3p/nixpkgs): Plumb through externalArgs.nixpkgsBisectPath
This lets the import of the depot root accept an additional argument
called `externalArgs`, which can be used to pass additional arguments
into a depot package set.

This is used in //third_party/nixpkgs for replacing the source of the
nixos-unstable channel with a path. With this we can bisect the
nixpkgs used in third_party easily.

Change-Id: I4f65eb3d6b521ed9f437649b7b068f1e6ab8210f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2925
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-11 11:08:49 +00:00
Vincent Ambo
473604f567 refactor: Move nixpkgs attribute to third_party.nixpkgs
Please read b/108 to make sense of this.

This gets rid of the explicit list of exposed packages from nixpkgs,
and instead makes the entire package set available at
`third_party.nixpkgs`.

To accommodate this, a LOT of things have to be very slightly shuffled
around. Some of this was done in already submitted CLs, but this
change is unfortunately still quite noisy.

Pay extra attention to:

* overlay-like functionality that was partially moved to actual
  overlays (partially as in, the minimum required to get a green
  build)

* modified uses of the package set path, esp. in NixOS systems

Special notes:

* xanthous has been disabled in CI because of issues with the Haskell
  overlay
* //third_party/nix has been disabled because of other unclear
  dependency issues

Both of these will be tackled in a followup CL.

Change-Id: I2f9c60a4d275fdb5209264be0addfd7e06c53118
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2910
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-04-10 21:18:55 +00:00
Vincent Ambo
d02eb5f8bf fix(tvix): Add //tvix to readTree roots
Change-Id: I7d0e81d6a88f351c7451454f5d506d708e7ed9ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2710
Tested-by: BuildkiteCI
Reviewed-by: adisbladis <adisbladis@gmail.com>
2021-03-31 19:23:39 +00:00
Vincent Ambo
aaaadbbcdc chore: Remove //overrides from depot
This mechanism wasn't actually used for anything other than a minor
external compatibility thing for buildGo.nix, which can also be solved
with a function parameter.

This breaks //fun/gemma because it means that the elmPackages used to
build it are no longer reachable from depot. We'll sort this out later.

Change-Id: I1bf2240435e869cdc4e99bdd1a138fdd2e76f96e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2646
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-23 23:55:49 +00:00
sterni
e91d5e4e61 fix(config): remove ciBuilds inherit
The ciBuilds attribute seems to no longer exist and it breaks the
evaluation of the config attribute. It's only appearance was in
besadii which doesn't actually use the attribute.

Removing the ciBuilds inherit fixes these issues.

Change-Id: Ibbf3413ba6efe10ad868cf57cf0711d574860f97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2487
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-02-06 17:54:18 +00:00
Vincent Ambo
9c482d6238 feat(ci): Add subtarget support for builds
We have naturally evolved a distinction between logical and physical
targets.

Physical targets are those which correspond directly to a tree
location on disk and can be built with `-A path.to.files`, while
logical targets are those that are exported from within an expression
but do not have a corresponding file on disk.

This change adds support for exporting logical targets from any tree
location by adding a `meta.targets` attribute containing keys into
itself, which will be consumed by the CI target gathering logic and
included in the generated pipeline.

Note that the labels for subtargets are syntactically different to
emphasise that they do not correspond to a file location. For example,
this change enables 'ops.nixos.whitbySystem' as a subtarget, which is
labeled in CI as `ops/nixos:whitbySystem`.

Change-Id: Ied09647a62c2ba98e3914548e3742ad422c63ecf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1893
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-31 23:14:11 +00:00
Vincent Ambo
61d2d2d503 feat(ops/pipelines): Dynamically generate CI pipeline from targets
Create the pipeline by outputting a file that contains nix-build
invocations for each target's *derivation path*.

Each invocation has a generated Nix expression passed to it with `-E`
which fetches the correct target from the tree while correctly
handling targets with strange characters (such as in Go-packages).

This makes it possible to run target-level granular pipelines. We're
getting somewhere!

Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-31 23:14:11 +00:00
Vincent Ambo
4ff9d5dee8 feat: Implement automatic CI target detection for the depot
Automatically walk the entire depot tree and pick out things that are
"buildable", then include them in the attribute `ci.targets` (which is
now also the target for CI builds).

A long time ago, in a land far away, we (well, I, at the time) had a
prototype of this which ran into constant issues with infinite
recursions while trying to walk the tree. In fact, this is why
readTree originally gained the `__readTree`-attribute which marks
things that were imported automatically.

Based on some code edef whipped up earlier (with the breakthrough
being that we also add the attribute to top-level folders, which
suddenly resolves a whole bunch of problems), I've now implemented
this actually working version.

At the moment all builds still happen as one big bag of builds, but at
some point we will granularise this.

Change-Id: I86f12ce7f63dae98e7e5c6646a4e9d220de783f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1854
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-26 23:49:32 +00:00
Vincent Ambo
bc5d4672aa fix: Remove ./presentations from top-level
This folder doesn't exist, it's part of my user folder now. We didn't
notice because nothing is walking the tree.

Change-Id: Idc6f20a8e4806a158c598fd63d381ab07934be1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1843
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
dc07977866 chore(ops): Clean up old GCP infrastructure files
This removes almost all of the GCP-infrastructure leftovers from my
previous setup.

The DNS configuration is retained, but moves to my user folder
instead.

Change-Id: I1867acd379443882f11a3c645846c9902eadd5b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/782
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-06-29 21:24:49 +00:00
Vincent Ambo
1640d9d145 refactor(ci-builds): Split up CI builds into multiple buckets
These categories separate CI targets, which hopefully avoids the
out-of-space errors we have been seeing on Sourcehut.

The sets of CI build targets are made available in the depot itself so
that besadii can be updated to create a new build for each target
group.

For convenience, 'ciBuilds' contains an '__allTargets' attribute which
combines the contents of each target batch - this makes it possible to
still invoke a build for everything by using:

  nix-build -A ciBuilds.__allTargets

Note: Some targets that were previously built in CI aren't anymore,
most importantly my NixOS systems which don't fit on Sourcehut.

Change-Id: Ia15ed7b743c8add51ae08ce0827a0ddfacd637e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/570
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-24 02:38:21 +00:00
Vincent Ambo
268729083e refactor(ops/nixos): Move my NixOS configurations to //users/tazjin
NixOS modules move one level up because it's unlikely that //ops/nixos
will contain actual systems at this point (they're user-specific).

This is the first users folder, so it is also added to the root
readTree invocation for the repository.

Change-Id: I546c701145fa204b7ba7518a8a56a783588629e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/244
Reviewed-by: tazjin <mail@tazj.in>
2020-06-13 23:52:35 +00:00
Vincent Ambo
9658e96a87 feat: Add 'depotPath' to depot root import
This is useful for things like including NixOS modules in
configurations without creating long and error-prone relative paths.

Change-Id: I4a5ebb1a0e5adf90b6bc50e884db453e12461001
Reviewed-on: https://cl.tvl.fyi/c/depot/+/243
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-13 23:52:35 +00:00
Vincent Ambo
a97094cc3e chore: Add //net to local package trees 2020-06-08 00:31:52 +01:00
Vincent Ambo
5d9d84f4cf refactor: Pass the depot as an argument named 'depot'
This change, which I've been meaning to do for a while, renames the
attributes passed by readTree to things in the tree so that:

* the depot root is now 'depot'
* depot.third_party is additionally passed as 'pkgs' (for
  compatibility with exported subtrees)
2020-02-21 12:45:43 +00:00
Vincent Ambo
e50c362244 feat(lisp/dns): Check in very early DNS-over-HTTPS client
This includes very barebones support for querying TXT and MX records
right now. The returned structure is not turned into a more convenient
format and error handling is, well, NIL.
2020-01-22 18:04:26 +00:00
Vincent Ambo
700a4c2e05 fix(build): Pass 'lib' as readTree argument to packages 2020-01-11 22:23:20 +00:00
Vincent Ambo
4a9c6ab6a2 refactor(nix/readTree): Move readTree to its own subfolder 2019-12-21 05:42:49 +00:00
Vincent Ambo
58f2f6f12d feat(bootstrapping-2018): Fix build and check in built presentation
This lets people browse the folder more easily, should they be so
inclined for whatever reason.
2019-12-21 01:07:29 +00:00
Vincent Ambo
a65cb50252 fix: Various minor fixes for new repository layout 2019-12-20 20:44:00 +00:00
Vincent Ambo
8de5d093d8 refactor: Fix a variety of filepaths for repo relayouting
This fixes readTree and the various project builds, as well
as (hopefully) most documentation links inside of the projects.
2019-12-20 20:37:02 +00:00
Vincent Ambo
124d26210b chore(build): Add package required for buildGo at the top-level
This is required to maintain buildGo compatibility with non-depot
setups.
2019-12-19 15:38:53 +00:00
Vincent Ambo
51ad58f8de fix(build): Add missing packages for Nixery images 2019-12-18 19:08:10 +00:00
Vincent Ambo
c776bd383d fix(build): Fix compatibility with Nixery instance 2019-12-18 17:39:00 +00:00
Vincent Ambo
0e5da6c34c feat: Expose readTree for downstream consumers
Exposes readTree from the package set but with a twist: It's exposed
as a functor that references the `.config` field from itself to get at
the configuration to be passed to packages.

This makes it possible for downstream users to make use of `readTree`
but with their own configuration.
2019-12-16 13:32:10 +00:00
Vincent Ambo
00129e06b2 fix(build): Make repository callable in Nix
This is required for some things that expect package sets to be
callable, e.g. Nixery.
2019-12-13 21:32:35 +00:00
Vincent Ambo
e9431682b2 refactor: Move CI setup to separate Nix file
Instead of polluting the repository namespace with the list of CI
projects, move that to a separate file.

Currently the list of projects to be built by CI is still hardcoded,
but this will be fixed soon.
2019-12-09 10:58:09 +00:00
Vincent Ambo
77c64ed8f6 fix(build): Temporarily hardcode list of CI projects
Due to a missing feature in readTree I can't currently generate this
list automatically.
2019-12-09 02:52:11 +00:00
Vincent Ambo
01bad09eed refactor: Introduce new layout with nixpkgs in third_party
This is not the final layout yet, but makes it so that my top-level
attribute set is no longer overlaid into nixpkgs itself.

This is useful for other people who are importing my monorepo.
2019-12-09 02:40:48 +00:00
Vincent Ambo
45d63bce17 feat(nix): Filter projects that should be built by CI
Instead of specifying CI projects manually, this filters them to move
the CI configuration into the derivations `meta` attributes.
2019-11-15 23:25:41 +00:00
Vincent Ambo
c1c379848a chore(nix): Move files around to conform to new read-tree layout
Broadly speaking, the following things are included:

* there is now a uniform `args` struct that is passed to all
  derivations, package headers have been changed appropriately
* overrides are now loaded from a separate `override` folder just
  using read-tree.nix
* third-party packages have moved into the `third_party` attribute set
2019-11-15 15:26:08 +00:00
Vincent Ambo
8e53dec88c chore: Remove unstable channel (no longer in use) 2019-11-14 13:44:03 +00:00
Vincent Ambo
8ffe811d46 feat(services): Add nixcon-demo service with simple web server 2019-10-26 13:27:56 +02:00
Vincent Ambo
2be1fff251 feat(third_party): Add naersk to package set 2019-10-26 13:27:41 +02:00
Vincent Ambo
9eabf6236f fix: Enable building of broken packages
Happstack is currently erroneously (afaict) marked as broken.
2019-10-26 12:10:21 +02:00
Vincent Ambo
9081e4aae4 chore: Bump nixpkgs and use Terraform from stable 2019-10-25 10:03:52 +02:00
Vincent Ambo
3a79e8b932 refactor(build): Add ciProjects attribute for CI builds
This attribute contains a list of all derivations that should be built
by the CI for this repository.

This includes all of my own packages that are not marked as broken, as
well as select third-party packages.
2019-09-22 17:19:13 +01:00
Vincent Ambo
415dc01e5e chore: Bump Nix channel to a more recent version
This version has Nixery popularity data available.
2019-09-21 15:41:19 +01:00
Vincent Ambo
16b317fa5a chore: Update kontemplate to v1.8.0
This version is agnostic of the working directory even if
insertFile/insertTemplate are used, which makes it a lot nicer to work
with in this repository structure.
2019-09-04 12:58:04 +01:00
Vincent Ambo
bcd7710be5 feat(tools): Introduce pass-compatible wrapper using Cloud KMS
Adds a shell script that supports a subset of the 'pass' interface for
compatibility with kontemplate, and wraps kontemplate in a script that
places this version on the PATH.

This makes it possible to use Cloud KMS encrypted secrets with kontemplate.
2019-09-03 16:12:30 +01:00
Vincent Ambo
f2e0f3ee27 chore(third_party): Remove git-appraise
Not actually in use here ...
2019-09-02 20:01:25 +01:00