I wanted Gitea to call Buildkite's pre-receive pipeline and either accept or
reject the incoming code depending on the outcome. The problem is that I can
only *create* builds from Gitea's pre-receive hook.
Now I'm left with two options:
1. run the lint-secrets step in post-receive
2. run `/nix/store/<hash>/git-secrets --scan-history $REPO_PATH` in Gitea
As far as I can tell, I cannot define Gitea hooks in Nix, which is unfortunate;
otherwise, option 2 would appeal more.
I'm doing option one for now.
So it turns out that I was wrong and that .git/config is stateful. Multiple
calls to --add-provider will append the same provider each time...
Instead I'm defining secret-patterns.txt and version-controlling it.
Then:
- dev-side: I'm adding `providers = cat ci/secret-patterns.txt` to .git/config
- ci-side: I'm adding `providers = cat ci/secret-patterns.txt` to .git/config
Unfortunately this is ad-hoc configuration ci-side, which I would like to
avoid. The good news is that my pre-commit hooks and failures from git-secrets
should now align with my CI, since they're both reading from
secret-patterns.txt. One step backwards... two steps forwards?
I'm also `cat .git/config` because I think the Buildkite destroys the
.git/config file for each build, but I want to verify that. If it does, I prefer
that because it seems to share the spirit of the "Destroy Your Darlings" essay.
Problem: my dev machine returns a different value for `git config --get-all
secrets.patterns` than my CI machine... I ran `git-secrets --register-aws` to
get additional coverage, but it's still not the same. I created an issue on the
git-secrets GH repo to get better troubleshooting advice, but I don't need the
logging info. anymore, so I'm removing it.
Somehow `git-secrets --scan-history` is exiting non-zero, when I don't think it
should. Logging some environment information to get a better idea of what's
going on.
After a handful of failed attempts to run lint-secrets.sh due to a missing
`git-secrets` executable on my git server, I decided that now was a good time to
use Nix to define my BuildKite pipelines.
TL;DR:
- Delete ci/scripts directory
- Define ci/pipelines/{briefcase,socrates}.nix
Outside of this repository:
- I logged into my admin account at git.wpcarro.dev and changed my Gitea
post-receive hook to trigger the briefcase pipeline
- I logged into my BuildKite account, deleted my build-briefcase pipeline,
created a new briefcase pipeline that called:
```shell
nix-build -A ci.pipelines.briefcase -o briefcase.yaml
buildkite-agent pipeline upload briefcase.yaml
```
One day I will audit all of my ad-hoc, non-mono-repo activity (like the steps I
listed above) and attempt to fit everything herein... one step at a time,
though!
Adding a fake secret to test to the new CI build step. I'm not sure I expect
this to fail the step because it relies on a pattern that I defined in
.git/config... let's see!
Having `git secret hide` as a pre-commit hook doesn't make much sense to me. I
will detail why when/if I write a blog post on briefcase's secret mgt setup.
The problem is, if I change secrets.json and then run `git status`, I won't see
any pending changes. This is because secrets.json is gitignore'd. If I run `git
secret hide` everytime I save secrets.json, I can rest assured that my `git
status` will be consistent with any updates to secrets.json.
I'm attempting to maintain a top-level secrets.json that defines all of the
sensitive data that I'd like to version-control without exposing everything in
cleartext to the world. To that end, I'm using `git secret`, which will use
`gpg` to encrypt secrets.json everytime I call `git secret hide` and decrypt
everytime I call `git secret reveal`.
I'm going to try this until I don't like it anymore... if that day comes...
I should write a blog post about my setup to solicit useful feedback and share
my ideas with others.
Adding a dummy, top-level secrets.json file using `git-secret`. It might be nice
to have a mono-secrets file in json because then I can use it with `jq` like:
```shell
$ jq '.secret' --join-output < ~/briefcase/secrets.json
```
This morning I'm attempting to secure my monorepo. How?
- `git secret`: DONE: To version-control sensitive data
- `git secrets`: TODO: Lint code for sensitive data
I will probably update the CI to call `git secrets --scan` or some similar
command to fail when that exists non-zero. I have much to learn, but doing is
the best way to learn it.
Anytime something before or during window-manager.el fails to evaluate,
I lose the ability to type, but I *can* still click.
@tazjin recommended that I use the mouse to cycle to the *Warnings*
buffer, which led me to another bug in a series of bugs that I'm
uncovering: ~/briefcase/org didn't exist.
A simple mistake like this should break my WM startup, so I decided to
remove most of my init hook logic.
Problem: dependency loading order
I originally assumed that keybindings.el was the first module to `require
'evil` because init.el shows:
```elisp
(require 'keybindings)
(require 'window-manager)
```
The problem is that keybindings.el calls `require 'window-manager` and
window-manager.el requires evil! I admit, I've created a bit of a birds nest for
myself.
A few thoughts:
- keybindings.el doesn't need to `require 'window-manager`. Fixed!
- window-manager.el shouldn't need to `require 'evil`. TODO...
While debugging some broken Emacs config, I ran Emacs in X, where for some
reason my PATH doesn't have my nix-env dependencies... because of this, when I
call `~/.nix-profile/bin/wpcarros-emacs` to start my Emacs, I saw warnings about
missing packages that I hadn't seen before. Nice!
In favor of keybindings.el!
Now I have:
- kbd.el: There are no keybindings in this file. It's just a library for working
with keybindings in Emacs.
- keybindings.el: (hopefully) all of my keybindings for EXWM, evil, etc.
In the spirit of Marie Kondo, I'm tidying up!
TL;DR:
- Prefer .envrc `use_nix` and delete all dir-locals.nix files
- Remove ~all references to <nixpkgs>, <unstable>, <depot> and prefer
referencing each with briefcase.third_party.{pkgs,unstable,depot}
- Delete nixBufferFromShell function since I was only using that in
dir-locals.nix files
My builds are still failing. This time with...
```
error: getting status of /home/wpcarro/nixpkgs-channels: Permission denied
```
...what confused me was the following:
```shell
$ sudo -u buildkite-agent-socrates stat /home/wpcarro/nixpkgs-channels
permission denied
```
But `ls -al /home/wpcarro | grep nixpkgs-channels` showed `r-w` for all users...
Thankfully @riking on ##tvl told me that I should check the permissions for
/home/wpcarro and /home...
After running `ls -al /home`, I saw `---` for all user... I then reproduced the
error by running:
```shell
$ sudo -u buildkite-agent-socrates stat /home
permission denied
```
Great!
So then I moved nixpkgs-channels to /var/lib/buildkite-agent-socrates. @edef
recommended that I read more about DynamicUser= setting for systemd, which looks
relevant after I took a cursory glance.
I'll also want a more declarative way to manager this, but I'm making small
improvements every day.
After enabling buildkite-agent using NixOS, it runs as its own user,
buildkite-agent-socrates, which does not have its own home directory. I moved
the SSH key that I made when running buildkite-agent as wpcarro into /etc/ssh
and `chown`'d it for buildkite-agent-socrates.
For now, I'm supporting two CI pipelines:
- build-socrates
- build-briefcase
Conceptually, build-briefcase should cover what build-socrates does now, but
eventually I would like build-socrates to call `switch-to-configuration` so that
all of my websites, etc. stay fresh.
Unforeseen problem: `buildkite-agent` runs its builds in a separate directory,
so if I want the `nix-build` command to build the newly checked out code, I need
to set <briefcase> to the CWD.
I've encountered a few problems with attempting to support nixos-rebuild:
- the activation step requires `sudo` privileges
- the `buildkite-agent` runs on the same machine, socrates, that is rebuilding
itself. This means that when the activation step runs, it will attempt to
restart `buildkite-agent` when the agent is still working
I'm slowly removing places in my nix code that rely on '<bracket>' notation, so
that I no longer depend on NIX_PATH being set. I still have more work to do.
{covid-uk,sandbox}/default.nix are breaking when I attempt to run my
build-socrates.sh script locally, so I'm temporarily disabling them until I can
get CI working as I expect.
I would like to setup a polkit rule to allow `buildkite-agent` (i.e. a
forthcoming user) to call `nixos-rebuild`. I need to know the `action.id` before
I can write a reliable rule.
- using `set -euo pipefail` for setting recommended failure-modes
- using `set -x` and `echo "$PATH"` to debug my failing build
Sidenote: I find BuildKite's documentation quite helpful!
I've been doing a few take-home assessment recently, all of which I've attempted
to solve using Haskell. I'm having a good time, and I'm noticing strong and weak
points with my Haskell programming. I always attempt to apply any feedback a
reviewer gives me, and I'm storing my first drafts, second attempts, and
feedback here for now.
This recently attempt was for a role at Jane Street.
I believe `use_nix` looks for shell.nix and then default.nix files, so I was
wrong earlier when I said that I could get rid of shell.nix files altogether.
I need to learn more about `use_nix` and getting environments from default.nix
files.
