Following the advice of Domen's nix.dev anti-patterns, I'm preferring something
like...
```nix
builtins.path { path = /path/to/some.where; name = "some.where"; }
```
...to
```nix
/path/to/some/where
```
While the former is more verbose, it will fail to build when the path doesn't
exist, which I prefer.
Automatically walk the entire depot tree and pick out things that are
"buildable", then include them in the attribute `ci.targets` (which is
now also the target for CI builds).
A long time ago, in a land far away, we (well, I, at the time) had a
prototype of this which ran into constant issues with infinite
recursions while trying to walk the tree. In fact, this is why
readTree originally gained the `__readTree`-attribute which marks
things that were imported automatically.
Based on some code edef whipped up earlier (with the breakthrough
being that we also add the attribute to top-level folders, which
suddenly resolves a whole bunch of problems), I've now implemented
this actually working version.
At the moment all builds still happen as one big bag of builds, but at
some point we will granularise this.
Change-Id: I86f12ce7f63dae98e7e5c6646a4e9d220de783f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1854
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
This was used previously when build granularity was besadii's task,
which it no longer is.
Change-Id: I6df2db1ed4730a7953199b7b48aa9ad916418b22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1853
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This is a temporary state (TODO added) to be picked up by the new CI
logic.
Change-Id: Id4702740ffd18325088e2a8a0c6157a8cee7ccf7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
These projects, which are not currently included in CI runs, don't
build at the moment.
Upcoming logic changes would mean that we would start including them
in CI, which is undesirable until they're fixed - but I'm not going to
be doing that now.
Change-Id: I7c337e098be8bff00db6d99fc7236a695f5a85f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1850
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This file is not a readTree-compatible Nix file, but rather a NixOS
module. At some point it should be moved elsewhere and .skip-subtree'd
to avoid this issue.
Change-Id: If1b3f7cc80084af1f44036b8b9272f7b76438c2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1849
Tested-by: BuildkiteCI
Reviewed-by: multi <depot@in-addr.xyz>
This is required to automatically walk the tree (see subsequent
commits).
Note: Lisp packages are removed from the CI builds in this commit
because the attrValues of third_party.lisp will contain an element
that is simply `true`, which causes a type error.
These packages are re-added when CI refactoring is complete.
Change-Id: I21e2b719e6c7161c23d2867a216f4daa1c6c8394
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1848
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This should actually just be an attribute set.
Change-Id: Idea1a9f7cfbb2eecd7e6342c6b5aeb66d3f3441a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1845
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This folder doesn't exist, it's part of my user folder now. We didn't
notice because nothing is walking the tree.
Change-Id: Idc6f20a8e4806a158c598fd63d381ab07934be1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1843
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
We don't need to build this anymore.
Change-Id: I0ddd4ec3db9eb4774553003e18c5503b0f431810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1842
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
This used to be part of the public interface, but was removed and
replaced with a (less useful) format string.
Change-Id: I387557c20c2eddde16974c3fcad1712569db5325
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1841
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
You can now provide a list of Nix derivations to tvlc to get a git worktree + sparse-checkout containing only the paths needed to build the specified derivations.
Known bugs: even though //third_party is only passed to readdir(), git doesn't know this and includes all of //third_party/*.
Change-Id: I9dccebd3fbff4bb04ebd568175cf0a7e37d71ab3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1826
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
I would prefer to define constants/briefcase in terms of `(getenv "BRIEFCASE")`
and assert that `(f-exists? (getenv "BRIEFCASE"))`, in one location:
constants.el
TL;DR:
- Prefer `(getenv "BRIEFCASE")` to `(f-expand "~/briefcase")`. I should audit my
Emacs for references to ~/briefcase and replace those calls with `getenv`.
- Remove calls setting <nixpkgs> and <depot> and rely exclusively on <briefcase>
- Prefer ~/nixpkgs-channels to ~/nixpkgs.
Notes:
- I need a better way of calling `home-manager switch` that resides within my
briefcase
This will look up a file in the current worktree of the git repository
enclosing `default-directory'.
In combination with project-find-file this lets me toggle between
switching to a file within a project, and within the whole depot.
Change-Id: Ie1011f10051fc2c4bd4279b0944a79c7edf92f3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1838
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
If this ends up working well I'll extract it to tvl.el
Change-Id: I83722abf33a3346ccc7957c8d64d6381b15c6ee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1837
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Expose an `sbcl` attribute on packages and programs, to allow for easier
development either with SLY or on a REPL.
Change-Id: Ide4d087a5223561e1fe192ef32dc593c54b5a20e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1834
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is the clang-tidy lint 'google-explicit-constructor'.
There's a whole bunch of breakage that was introduced by this, and we
had to opt out a few types of this (esp. the string formatting crap).
In some cases minor other changes have been done to keep the code
working, instead of converting between types (e.g. an explicit
comparison operator implementation for nix::Pid).
Change-Id: I12e1ca51a6bc2c882dba81a2526b9729d26988e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1832
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
I wanted Gitea to call Buildkite's pre-receive pipeline and either accept or
reject the incoming code depending on the outcome. The problem is that I can
only *create* builds from Gitea's pre-receive hook.
Now I'm left with two options:
1. run the lint-secrets step in post-receive
2. run `/nix/store/<hash>/git-secrets --scan-history $REPO_PATH` in Gitea
As far as I can tell, I cannot define Gitea hooks in Nix, which is unfortunate;
otherwise, option 2 would appeal more.
I'm doing option one for now.
So it turns out that I was wrong and that .git/config is stateful. Multiple
calls to --add-provider will append the same provider each time...
Instead I'm defining secret-patterns.txt and version-controlling it.
Then:
- dev-side: I'm adding `providers = cat ci/secret-patterns.txt` to .git/config
- ci-side: I'm adding `providers = cat ci/secret-patterns.txt` to .git/config
Unfortunately this is ad-hoc configuration ci-side, which I would like to
avoid. The good news is that my pre-commit hooks and failures from git-secrets
should now align with my CI, since they're both reading from
secret-patterns.txt. One step backwards... two steps forwards?
I'm also `cat .git/config` because I think the Buildkite destroys the
.git/config file for each build, but I want to verify that. If it does, I prefer
that because it seems to share the spirit of the "Destroy Your Darlings" essay.
Problem: my dev machine returns a different value for `git config --get-all
secrets.patterns` than my CI machine... I ran `git-secrets --register-aws` to
get additional coverage, but it's still not the same. I created an issue on the
git-secrets GH repo to get better troubleshooting advice, but I don't need the
logging info. anymore, so I'm removing it.
Somehow `git-secrets --scan-history` is exiting non-zero, when I don't think it
should. Logging some environment information to get a better idea of what's
going on.
After a handful of failed attempts to run lint-secrets.sh due to a missing
`git-secrets` executable on my git server, I decided that now was a good time to
use Nix to define my BuildKite pipelines.
TL;DR:
- Delete ci/scripts directory
- Define ci/pipelines/{briefcase,socrates}.nix
Outside of this repository:
- I logged into my admin account at git.wpcarro.dev and changed my Gitea
post-receive hook to trigger the briefcase pipeline
- I logged into my BuildKite account, deleted my build-briefcase pipeline,
created a new briefcase pipeline that called:
```shell
nix-build -A ci.pipelines.briefcase -o briefcase.yaml
buildkite-agent pipeline upload briefcase.yaml
```
One day I will audit all of my ad-hoc, non-mono-repo activity (like the steps I
listed above) and attempt to fit everything herein... one step at a time,
though!
The previous clang-tidy invocation missed some header files, which has
now been rectified.
Change-Id: I31547754fbf52f439dc7aeefb08ab90bd50c4156
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1831
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
This reverts commit 8fa3087067.
Reason for revert:
This almost worked. We discovered two important issues:
- The daemon startup does not correctly handle the socket passed in by systemd.
- There is some issue with chunking of large calls, running a build for ci-builds resulted in:
tazjin@whitby /depot (canon)> nix-build -A ciBuilds.__allTargets
E20200821 01:42:22.846053 12601 shared.cc:306] error: Rpc call addTextToStore to unix:///nix/var/nix/daemon-socket/socket failed (RESOURCE_EXHAUSTED): Received message larger than max (10889961 vs. 4194304)
Change-Id: Ic5ba4ef06a4953cf71a36b139fe25ea673cb6fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1802
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
... this is going to break so much stuff. Lets have some fun.
Change-Id: If0185e0323391c7055d47b797083bb5afde57cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1829
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
.. turns out producing an entire build log on a single line is not
useful.
This does not use `<< std::endl` because we have run into buffering
issues with the implementation of the logs->gRPC sink, but intend to
replace this in the future using a structured sink for BuildEvent
protos rather than a raw stream.
Change-Id: Ia9b05fa804391d389e2ef53ab4436c0ec5cc452e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1828
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
