Commit graph

582 commits

Author SHA1 Message Date
Griffin Smith
04ae293360 feat(tools): Add script to flamegraph a command
Add tools.perf-flamegraph, which collects the base case execution of
perf piped through stackcollapse-perf and flamegraph to flamegraph the
execution of an external command via perf.

Change-Id: I671fe254dc374b6cd7deca2d3bdea266164de025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1176
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-15 20:21:34 +00:00
Griffin Smith
a0f9ae996d refactor(tvl.el): Don't submit on rubberstamp
Now that we're requiring CI to verify all changes (yay) it's not
actually possible to push and submit in one go - this repurposes the
rubberstamp command to push and self-+2, which is still useful for
self-owned things like system config etc.

Change-Id: Ie064adb6b7b0c1fd6bffdf4583a201ea02d4f8ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/939
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-06 17:44:18 +00:00
Vincent Ambo
29984bb6f9 feat(tools/hash-password): Add a wrapper script for slappasswd
This wrapper script correctly invokes slappasswd for generating ARGON2
hashes.

People without Nix will need to figure this out on their own.

Change-Id: I264715ba4932855b868bef70cc9102515b3d129e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/833
Tested-by: BuildkiteCI
Reviewed-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-01 20:39:07 +00:00
Vincent Ambo
cdc687d35b feat(tools/nsfv-setup): Move NSFV setup script out of //users
This program can be used for noise-cancelling in arbitrary audio
streams. See CL/576 and CL/577 for more details.

This moves the script out of my users folder and to //tools, and adds
two changes:

* Existing sinks will be removed & reinitialised
* The sink is changed from stereo->mono (which seems to make little
  difference but is more reliable)
* The parameter has changed from 50 to 42 because I felt like it

Change-Id: Id9ff285fefd682ccc4d9f18e46b5fde9cd056aa7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/578
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-26 13:55:00 +00:00
Griffin Smith
c96f8327f6 feat(tvl.el): Add push-wip command
P W, to push as a work-in-progress CL

Change-Id: Ibc6973512e8dcd3ce77b2f1064906a98e9a3a182
Reviewed-on: https://cl.tvl.fyi/c/depot/+/575
Reviewed-by: tazjin <mail@tazj.in>
2020-06-25 12:06:51 +00:00
Vincent Ambo
b3a0a96953 fix(tvl.el): Use 'canon' branch instead of 'master'
Change-Id: I779c3c979cc21a428188d31b399fd8ba97017f8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/567
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-24 01:34:02 +00:00
Luke Granger-Brown
68779da596 fix(gerrit-cli): don't output the path to SSH if it's on the path
Change-Id: Ic5ef3e9ae7984cc6c84ef99553bc94a7d44c3929
Reviewed-on: https://cl.tvl.fyi/c/depot/+/551
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:30 +00:00
Vincent Ambo
082ed2a74f chore(cheddar): Clean up all clippy lints
Change-Id: I5544c10e964b57891fce6ea01f09eeec406e9bef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/510
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-20 03:00:46 +00:00
Vincent Ambo
e27b0a3013 feat(cheddar): Add Sourcegraph-compatible highlighting server
Sourcegraph uses a component called syntect_server to syntax-highlight
source files.

Since we already have custom syntaxes, overrides and configuration we
might as well use them with Sourcegraph!

This implements the syntect_server "protocol" (it's just a single
endpoint) so that we can swap out the syntect_server component with
cheddar.

Note: There's a few clippy lints here, but they're being solved in a
followup commit because I wanted to take care of all of them at
once (not just the ones introduced in this change).

Change-Id: Ib518a2fa1b9fee299fe599482403599583cac479
Reviewed-on: https://cl.tvl.fyi/c/depot/+/509
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-20 02:59:20 +00:00
Vincent Ambo
9266b3947d feat(emacs-pkgs/tvl): Add #'tvl-depot-status function
This function opens the depot in magit. By default the depot path is
assumed to be `/depot`, but that is made available as a customise
option that users can override.

Change-Id: Iefc127cad515dbae330d7af3aa69a0c53aa02015
Reviewed-on: https://cl.tvl.fyi/c/depot/+/504
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-19 17:54:46 +00:00
Vincent Ambo
e7f8bdd548 feat(emacs-pkgs/tvl): Add magit shortcuts for reviews & co
Introduces two new shortcuts in the magit popup for pushing:

* R: Push for review
* S: Push to submit

The existing "P" command remains, but has been renamed to
"rubberstamp" and documented with a warning.

To make this work across people's different setups, customisation
options for TVL-specific code are being added.

Change-Id: I48e75db55b5f793342fc898093629e437f58dcb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/503
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-19 17:54:46 +00:00
Griffin Smith
5b833a1375 feat(emacs-packages/tvl): Init with magit gerrit suffix
Add a simple tvl.el, with only one magit transient command to push and
submit in one go to Gerrit

Change-Id: If02065797a8db39669b85acf87a1c4b43d1482ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/494
Reviewed-by: tazjin <mail@tazj.in>
2020-06-19 03:35:27 +00:00
Vincent Ambo
dbb4a57bbf style(cheddar): Reformat with rustfmt
Change-Id: I9a0764f23c1269447a8b18f3128074fc58f70456
Reviewed-on: https://cl.tvl.fyi/c/depot/+/492
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-19 02:49:41 +00:00
Vincent Ambo
d1524a5a6f refactor(cheddar): Pass references readers & writers into renderers
This paves the way for using other things than stdin/stdout as
sources/sinks, which is required for example for implementing a
syntect_server replacement based on cheddar.

Change-Id: I5779db8dbf7b7ced109c26b940f721d237d60785
Reviewed-on: https://cl.tvl.fyi/c/depot/+/491
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-19 02:49:41 +00:00
Vincent Ambo
0b0e8c2da7 chore(tazjin/emacs): Move //tools/emacs to //users/tazjin/emacs
Change-Id: I9dfbd0bb1fd3e215bb91c5734fb0934ee3faeae6
2020-06-18 20:19:50 +01:00
Vincent Ambo
5290c9f59c feat(tools/emacs): Enable go-mode in .go2-files
Change-Id: I03cf0641b22828eb400b9220e0756d9b7aca01cd
2020-06-18 02:42:19 +01:00
Vincent Ambo
c224a738e7 chore(tools/emacs): Remove hardcoded TVL meeting code
Change-Id: I76d24c20dc41969efe242008a2dec4a2a7944c5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/450
Reviewed-by: tazjin <mail@tazj.in>
2020-06-17 13:50:35 +00:00
Vincent Ambo
5e5e02d6a7 refactor(cheddar): Switch to clap-rs for command line arguments
The complexity of the arg parsing is increasing somewhat because we're
adding more features to cheddar, so to set us up for that this
switches the arg parsing to the somewhat more flexible clap.

Change-Id: I187bc0c1b6c6bd596fa0f6bb494b04e335262ba9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/445
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-06-17 01:18:16 +00:00
Vincent Ambo
51594b0594 chore(cheddar): Bump all dependencies
Manual updates:
  comrak 0.6 -> 0.7
  syntect 3.3 -> 4.2.0

Leading to these automatic updates:
    Updating crates.io index
    Updating adler32 v1.0.4 -> v1.1.0
      Adding autocfg v1.0.0
    Updating base64 v0.10.1 -> v0.12.2
    Removing bindgen v0.53.3
    Removing cexpr v0.4.0
      Adding chrono v0.4.11
    Removing clang-sys v0.29.3
    Updating comrak v0.6.2 -> v0.7.0
    Removing env_logger v0.7.1
    Removing glob v0.3.0
    Updating hermit-abi v0.1.13 -> v0.1.14
    Removing humantime v1.3.0
      Adding indexmap v1.4.0
    Updating itoa v0.4.5 -> v0.4.6
    Updating libc v0.2.70 -> v0.2.71
    Removing libloading v0.5.2
    Removing log v0.4.8
    Updating miniz_oxide v0.3.6 -> v0.3.7
    Removing nom v5.1.1
      Adding num-integer v0.1.43
      Adding num-traits v0.2.12
    Updating onig v5.0.0 -> v6.0.0
    Removing peeking_take_while v0.1.2
    Updating plist v0.4.2 -> v1.0.0
    Updating proc-macro2 v1.0.17 -> v1.0.18
    Removing quick-error v1.2.3
    Updating quote v1.0.6 -> v1.0.7
    Updating regex v1.3.7 -> v1.3.9
    Updating regex-syntax v0.6.17 -> v0.6.18
    Removing rustc-hash v1.1.0
    Updating ryu v1.0.4 -> v1.0.5
    Updating serde v1.0.110 -> v1.0.112
    Updating serde_derive v1.0.110 -> v1.0.112
    Updating serde_json v1.0.53 -> v1.0.55
    Removing shlex v0.1.1
    Updating syn v1.0.25 -> v1.0.31
    Updating syntect v3.3.0 -> v4.2.0
    Removing termcolor v1.1.0
    Removing version_check v0.9.2
    Removing which v3.1.1
    Updating yaml-rust v0.4.3 -> v0.4.4

Change-Id: I2c2ee6a45bf7a5286fe30d61de52d7993871e7e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/444
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-16 23:59:48 +00:00
Vincent Ambo
9a7a0aa597 chore: Remove traces of Hound
We can always revert this if we want it back.

Change-Id: I1332b6dd541199584b7b5b94a8651172d79e53a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/442
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-16 20:32:18 +00:00
Griffin Smith
18994b9c58 fix(tools/gerrit-cli): Don't wrap ssh sniffing in a subshell
This was actually executing ssh, which was kinda silly - we really just
want to check the return code of the `which` invocation

Change-Id: I8a4b277a2be3b0c6a43d2da830cb82a32f9ee51e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/428
Reviewed-by: tazjin <mail@tazj.in>
2020-06-16 15:43:28 +00:00
Griffin Smith
406a66bed1 feat(gerrit-cli): Add gerrit CLI script
Add a script to run gerrit commands on the depot host, which reads the
gerrit username from the TVL_USERNAME env var or $(whoami) by default.

At some point this may want to solve the configuration problem in a more
general fashion, but for now this seems relatively obvious.

Change-Id: Ied91c1d26daf4770aef74b2e755d1760c486bb7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/396
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-16 01:54:39 +00:00
Vincent Ambo
29bab159ab feat(depot-build): Add a tool to build at arbitrary tree locations
Assuming you're in `/depot/fun/clbot`, you can do this:

    tazjin@frog /d/f/clbot (master)> depot-build
    Building //fun/clbot
    /nix/store/i1zc6g58wa4819kyhaqi12zsh3hr31ph-clbot

It is automatically added to the $PATH using direnv, too.

Change-Id: Ia3341704e6317c2b8de40a3fa1be3b680d21a42d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/388
Reviewed-by: eta <eta@theta.eu.org>
2020-06-15 19:08:53 +00:00
Vincent Ambo
9885b70b7a feat(cheddar): Override syntax highlighting for 'rules.pl' to Prolog
Adds a mechanism for per-filename overrides of the chosen language
syntax and configures it for Gerrit's submit rule file.

This also switches the syntax set used to the one from
//third_party/bat_syntaxes, which contains custom additions such as
Prolog support.

Change-Id: I2023dbad5b326305ef2ef0ecf34ef66a3f7575ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/349
Reviewed-by: riking <rikingcoding@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-15 16:52:27 +00:00
Vincent Ambo
d570fbbad3 chore(tools/emacs): Update window management configuration for frog
Change-Id: I848d4c1d58a81a66800f25459fee8f85d3f269f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/64
Reviewed-by: tazjin <mail@tazj.in>
2020-06-11 23:20:46 +00:00
Vincent Ambo
524b621231 feat(tools/emacs): Configurable depot path, defaulting to /depot 2020-06-11 18:33:02 +01:00
Vincent Ambo
820f591cbb chore(tools/emacs): Remove unused #'notmuch-depot-apply-patch 2020-06-11 18:32:38 +01:00
Vincent Ambo
3407baa756 fix(tools/emacs): Fix path to explain-pause-mode 2020-06-11 12:20:37 +01:00
Vincent Ambo
cc24097029 feat(3p/emacs): Check in derivation for explain-pause-mode
This mode makes it possible to track down slowness in Emacs functions
across the board.

https://github.com/lastquestion/explain-pause-mode

Very useful for some things I need to debug ...
2020-06-03 02:31:31 +01:00
Vincent Ambo
991cf97df2 feat(tools/emacs): Install and configure hound package at cs.tazj.in 2020-05-27 00:06:59 +01:00
Vincent Ambo
74a70896f2 feat(tools/emacs): Add vterm-send-ctrl-x function
This is useful if something accidentally opens nano.
2020-05-26 01:21:42 +01:00
Vincent Ambo
6f8fbf4aa4 chore(tools/cheddar): Bump dependencies & upgrade to LLVM 10 2020-05-25 23:37:39 +01:00
Vincent Ambo
c08886ab82 fix(tools/emacs): Use explicit includes 2020-05-22 19:05:39 +01:00
Vincent Ambo
1086072d3d feat(tools/emacs): Configure eglot to shut down automatically
When all buffers related to a specific eglot instance are gone, eglot
can shut down.
2020-05-19 14:23:59 +01:00
Vincent Ambo
a3894c2bf1 feat(tools/emacs): Install meson-mode 2020-05-17 23:59:00 +01:00
Vincent Ambo
d97cf260a2 refactor(tools/emacs): Move third-party overrides to //third_party
EXWM and vterm are overridden, but this shouldn't be happening outside
of //third_party
2020-05-16 04:08:28 +01:00
Vincent Ambo
a078cabc71 fix(tools/emacs): Configure xkb switching shortcuts on EXWM launch
This enables half of the shortcuts for switching keyboard languages
when EXWM launches.

The cyrillic ones are commented out because Emacs doesn't start
otherwise, I've no idea why and don't care at the moment.
2020-05-16 03:55:31 +01:00
Raphael Borun Das Gupta
bdda24a772 chore(nix): update channel 19.03 -> 20.03
Use a NixOS / NixPkgs release that's actually being supported
and regularly updated.
2020-05-01 11:58:31 +01:00
Vincent Ambo
5cc37a15a5 feat(defzone): Add plist argument format for SOA records
These records have so many fields that it's difficult to track what's
what in a long list. For convenience they're now specified in plist
format (see the example).

There isn't really a point to this because the SOA record is the one I
care the *least* about practically as Cloud DNS sets it for me, but
whatever.
2020-04-20 22:39:58 +01:00
Vincent Ambo
8c86b9b5f6 feat(defzone): Add an Emacs package for generating zone files
This currently only supports a small subset of available records, but
I actually kind of like the nesting.
2020-04-20 22:39:58 +01:00
Vincent Ambo
590e1e1048 feat(tools/emacs): Add screen layouts for nugget to EXWM config 2020-04-18 15:27:56 +01:00
Vincent Ambo
820b1bb0c9 feat(tools/emacs): Refactor monitor layout setup in EXWM
vauxhall (my laptop) now has an additional screen connected at home,
but sometimes I use that screen for my desktop computer (nugget).

This refactors the randr configuration for EXWM to support somewhat
more dynamic, multi-monitor layouts and adds key bindings to toggle
between some of the different configurations I want.
2020-04-17 13:13:59 +01:00
Vincent Ambo
afc729c419 feat(tools/emacs): Install polymode & protobuf-mode
... how did I not have protobuf-mode installed? What?
2020-04-06 23:19:52 +01:00
Vincent Ambo
e3864f3b85 feat(tools/emacs): Title the TVL window correctly 2020-04-04 22:27:06 +01:00
Vincent Ambo
e9aa5ae463 fix(tools/emacs): Include date in scrot screenshot names 2020-04-04 20:05:32 +01:00
Vincent Ambo
f99fc009fc fix(tools/emacs): Fix notmuch-show-open-or-close-subthread
Accidentally dropped a key parameter, also restored interactive
functionality via prefix arguments..
2020-04-02 21:59:18 +01:00
Vincent Ambo
40d13cc87f feat(tools/emacs): Bindings for (un)collapsing notmuch subthreads
Modifies notmuch-show-open-or-close-subthread to take a parameter
instead of using prefix to toggle the argument, and binds that
function to C-, and C-. in notmuch-show-mode-map to enable convenient
collapsing/uncollapsing of subthreads from point.
2020-04-02 01:35:16 +01:00
Vincent Ambo
5aaeaa406a feat(tools/emacs): Implement 'notmuch-show-open-or-close-subthread'
Does what it says on the tin.
2020-04-01 22:28:23 +01:00
Vincent Ambo
049a13562c feat(tools/emacs): Install eglot & dependencies 2020-03-29 22:31:32 +01:00
Vincent Ambo
8df1371b9d chore(tools/emacs): Remove unimportant 'important' label from graph 2020-03-11 05:55:22 +00:00
Vincent Ambo
049709a7f3 fix(tools/emacs): Set correct file name format for org-journal 2020-03-09 01:07:32 +00:00
Vincent Ambo
9372f57659 feat(tools/emacs): Configure org-journal
Configures org-journal to store journal files on camden and encrypt
them to my GPG key.

Journal entries are weekly, with weeks starting Saturday (yes, there's
a reason for this).
2020-03-09 01:00:11 +00:00
Vincent Ambo
15ae7fadb8 fix(tools/emacs): Set scratch buffer to Emacs Lisp mode
This used to be the case, but it changed to lisp-interaction-mode at
some point which is incredibly annoying.
2020-03-09 00:16:18 +00:00
Vincent Ambo
067acdbcae chore(tools/emacs): Configure all-tags list in notmuch-hello
I've been setting these options manually in my sessions for some
reason. Let's make it official!
2020-03-09 00:08:33 +00:00
Vincent Ambo
28ec16e618 feat(tools/emacs): Add a function to graph unread emails per label
This uses the built-in chart.el library to create a quick graph of the
number of unread emails in each notmuch tag. Some generic tags are
excluded from the overview.
2020-03-08 23:58:01 +00:00
Vincent Ambo
65e533431a refactor: emacsPackagesNg -> emacsPackages
The former is now just an alias for the latter anyways.
2020-03-08 23:20:03 +00:00
Vincent Ambo
6cbcadc2e4 feat(tools/emacs): Install org-journal & org-ql 2020-03-08 23:16:44 +00:00
Vincent Ambo
4c0e565fe1 feat(tools/emacs): Install cmake-mode 2020-02-26 15:20:41 +00:00
Florian Klink
215df37187 fix(popcount): Fix nix-build -A nixery-popcount
Previously, this was failing as follows:

```
these derivations will be built:
  /nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv
building '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv'...
building
warning: GOPATH set to GOROOT (/nix/store/4859cp1v7zqcqh43jkqsayl4wrz3g6hp-go-1.13.4/share/go) has no effect
failed to initialize build cache at /homeless-shelter/.cache/go-build: mkdir /homeless-shelter: permission denied
builder for '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed with exit code 1
error: build of '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed
```
2020-02-26 08:31:46 +00:00
Vincent Ambo
bbab6cd1c1 feat(tools/emacs): Use patched rcirc 2020-02-24 16:40:43 +00:00
Vincent Ambo
4bbbb58cb5 chore: Rename pkgs->depot in all Nix file headers 2020-02-21 13:54:53 +00:00
Vincent Ambo
14b5447aac Merge branch 'feat/cheddar-extensions' 2020-02-09 12:07:10 +00:00
Vincent Ambo
31b3e533c6 chore(tools): Remove //tools/blog_cli
No longer needed with the deprecation of tazblog.
2020-02-09 01:30:25 +00:00
Vincent Ambo
ec42a67569 feat(tools/emacs): Bump EXWM to recent master
This version of EXWM fixes some issues with binding keys in EXWM as
well as other minor annoyances.
2020-02-07 12:41:43 +00:00
Vincent Ambo
c56968a767 feat(tools/emacs): Install ace-link
It now has vterm-mode support, which has finally convinced me to try it.
2020-02-05 11:58:57 +00:00
Vincent Ambo
dba2e5426e fix(emacs-pkgs/nix-util): Don't leave sly build out-links around 2020-01-27 02:11:04 +00:00
Vincent Ambo
a2d0b7f399 feat(emacs.d): Add interactive 'scrot-select' screenshot function 2020-01-26 19:13:59 +00:00
Vincent Ambo
fb5f26e548 feat(emacs.d): Add notmuch-depot-apply-patch helper function
This function lets me interactively apply a patch from the currently
opened notmuch message to the depot.
2020-01-24 18:51:13 +00:00
Vincent Ambo
98cc5f9fac feat(emacs.d): Add key binding for counsel-notmuch 2020-01-22 17:26:11 +00:00
Vincent Ambo
8f68497269 fix(emacs-pkgs/nix-util): Ensure compatibility with non-native JSON 2020-01-20 00:58:46 +00:00
Vincent Ambo
d752cbecc0 feat(cheddar): Add toggle flag for about-filter behaviour
Cheddar now needs to be passed the --about-filter flag to toggle the
behaviour for rendering Markdown into HTML.

By default Markdown will be highlighted like normal source code (i.e.
cgit source-filtering is the default behaviour).
2020-01-19 23:54:30 +00:00
Vincent Ambo
0b3b8a07c4 feat(emacs): Install geiser for Guile 2020-01-19 19:35:27 +00:00
Vincent Ambo
9879703155 fix(emacs-pkgs/dottime): Do not call telega macros in advice setup
This advice is potentially defined before the autoloads for telega
have run, which means that the macro-expansion fails and
`telega-ins-fmt` is looked up as a function.

With this setup the initialisation works as expected.
2020-01-19 16:48:11 +00:00
Vincent Ambo
f09c6a8ecd feat(emacs.d): Switch font to JetBrains Mono
Lets evaluate this for a bit. With the current settings it even seems
to render _okay_ on nugget.
2020-01-19 16:39:21 +00:00
Vincent Ambo
1031d890ec fix(builder): Fix minor logging switcharoo 2020-01-19 07:44:36 +00:00
Vincent Ambo
3567f8bd0a fix(emacs.d): Use "light" Input font on nugget 2020-01-19 02:27:54 +00:00
Vincent Ambo
bdd488ce27 fix(emacs-pkgs/nix-util): Use lexical-let to allow variable capture
The lambda that acts as the sentinel for building SBCL with packages
needs to be able to capture variables if lexical binding is enabled,
which is made possible by the lexical-let form.
2020-01-17 19:09:46 +00:00
Vincent Ambo
815a224744 feat(emacs): Install lispy for evaluation
Not configuring this automatically yet, I mostly want to try it out.
2020-01-17 17:48:13 +00:00
Vincent Ambo
86c8275a53 fix(emacs.d): Load relevant modes for Lisp development in IELM 2020-01-17 16:52:17 +00:00
Vincent Ambo
381a859b3b feat(emacs-pkgs/nix-util): Add nix/sly-from-depot function
Adds a function that can launch Sly with a pre-configured SBCL for a
Lisp derivation in the depot.

This makes it convenient to spin up development environments for Lisp
libraries and programs by simply calling `M-x nix/sly-from-depot RET
tools.something`.

This relies on `nix-depot-path` being configured currently as I have
not yet reliably added the depot to my NIX_PATH on all machines.
2020-01-17 16:43:49 +00:00
Vincent Ambo
bfd0bc2a79 feat(emacs.d): Configure Sly for Common Lisp
Sets up Lisp modes in Sly REPL and points at the local hyperspec
checkout.

In fact the Hyperspec bit should probably be managed by Nix, but one
step at a time.
2020-01-15 11:42:19 +01:00
Vincent Ambo
829de752da fix(emacs.d): Trim superfluous whitespace from TODO comments 2020-01-11 23:23:46 +00:00
Vincent Ambo
2236d43ff7 feat(cheddar): Implement callout paragraphs
Implements support for tagging paragraphs that begin with a callout
word (TODO, WARNING, QUESTION, TIP) with an additional `cheddar-*`
class that makes it possible to render these callouts specially.

This is currently not the nicest implementation, but it works.
2020-01-11 05:06:36 +00:00
Vincent Ambo
d4e469508f refactor(cheddar): Extract code block highlighting into function
Since I am going down the path of adding additional Markdown
extensions it makes sense to avoid letting `format_markdown` turn into
a giant beast of a function.

Therefore this commit extracts the logic for rendering code blocks via
syntect and changes the innards of `format_markdown` to instead
provide arbitrary AST value replacements.
2020-01-11 02:17:19 +00:00
Vincent Ambo
cd0093d46e fix(emacs): Minor typo fix 2020-01-08 00:32:56 +00:00
Vincent Ambo
9e2f86d65d chore(emacs): Bump emacs-libvterm to a more recent version 2020-01-07 22:28:36 +00:00
Vincent Ambo
efc782fcdc fix(emacs): Disable linking of imagemagick due to vterm crashes
See explanatory comment.
2020-01-07 22:28:17 +00:00
Vincent Ambo
1e71b80f6a fix(emacs): Build emacs with imagemagick support 2020-01-05 17:37:01 +00:00
Vincent Ambo
b2cd30bfd4 chore(emacs.d): Remove edwina configuration
Sounds good, doesn't work.

(Okay, it does - but not like I want it to and with too many caveats
at the moment - maybe later)
2020-01-05 01:13:00 +00:00
Vincent Ambo
3638048c9b fix(emacs.d): Use 'fish' from $PATH when launching vterms
This is required because the configuration is used on machines where
fish comes from Nix, and on ones where it does not.
2020-01-04 22:50:19 +00:00
Vincent Ambo
41232a1c78 feat(emacs.d): Toggle email signature manually with C-c C-w
The signature itself is read from ~/.signature
2020-01-01 16:28:47 +01:00
Vincent Ambo
39131d9e0f fix(emacs.d): Disable notmuch FCC directories 2019-12-30 22:19:56 +01:00
Vincent Ambo
08dd267c19 chore(dottime.el): Remove advice for notmuch date rendering
This code has moved into a patch for notmuch itself.
2019-12-27 19:18:00 +01:00
Vincent Ambo
100f451b0a fix(dottime.el): Defer package-specific setup 2019-12-27 13:04:49 +01:00
Vincent Ambo
a54cc040ad fix(emacs.d): Load dottime.el after libraries that it modifies 2019-12-27 12:54:12 +01:00
Vincent Ambo
309d1ac50f feat(dottime.el): Support dottime display in notmuch thread view
Advises notmuch to display `Date` headers using dottime.
2019-12-27 12:54:12 +01:00
Vincent Ambo
29e71053be feat(dottime.el): Add arbitrary offsets & prefixes to dottime-format
This gives users from Elisp slightly more flexibility about the
display of dottime.
2019-12-27 12:54:12 +01:00
Vincent Ambo
5fd7de921b feat(dottime.el): Use dottime in telega.el, too 2019-12-27 12:54:12 +01:00
Vincent Ambo
b48797e4ce chore(emacs.d): Remove explicit FCC dir for private account
No longer required because Gmail does this automatically.
2019-12-26 15:14:37 +01:00
Vincent Ambo
cf9dd380f3 chore(emacs.d): Remove GPG agent warmup from mail sending
Fetching credentials is no longer handled by msmtp itself.
2019-12-26 14:13:40 +01:00
Vincent Ambo
29af390e8c fix(emacs.d): Slightly more reliable way of setting edwina prefix 2019-12-25 14:10:00 +01:00
Vincent Ambo
b59788779a fix(emacs.d): Display newest emails first in notmuch 2019-12-25 14:09:42 +01:00
Vincent Ambo
4a0cf0de98 fix(emacs.d): Avoid advising split-window-* functions
Advising these functions apparently breaks things internally.
2019-12-24 18:43:52 +01:00
Vincent Ambo
637cb0bd2a fix(emacs.d): Pick new buffers *into* the active window
Instead of splitting below and moving the target buffer into the new
split, split and move the buffer into the active window.

The other way around does (for some reason I don't fully understand)
not work because `split-window-below` may return invalid windows.
2019-12-24 18:15:30 +01:00
Vincent Ambo
ceaa0ff0ba feat(emacs.d): Advise split-window-below to call edwina-arrange 2019-12-24 15:43:24 +01:00
Vincent Ambo
86c2ebc096 feat(emacs.d): Enable edwina package for window management
Configures edwina using `s-w` as the key prefix (in line with my other
EXWM-related commands).

An additional function is added that switches to a buffer (borrowing
the implementation from `ivy-switch-buffer`) but splitting it into a
new window instead.
2019-12-24 15:30:25 +01:00
Vincent Ambo
ccd31b8898 feat(emacs.d): Bind 's-g m' to notmuch UI 2019-12-23 16:05:05 +01:00
Vincent Ambo
a36e9f817d fix(cheddar): Fix errors if filename does not have an extension 2019-12-22 15:21:23 +00:00
Vincent Ambo
f1c018af18 feat(emacs): Install edwina package
No configuration for this yet, I just want to try it out. It seems
like an interesting way of managing Emacs windows!
2019-12-22 14:30:28 +00:00
Vincent Ambo
03acae2f85 feat(cheddar): Style pre-elements for better readability
Uses GitHub-like styling for <pre> elements, i.e. slight padding and
background colour highlighting.
2019-12-21 16:18:26 +00:00
Vincent Ambo
f904951384 feat(cheddar): Render code blocks in Markdown via syntect
Implements fully static (i.e. no JavaScript!) highlighting of code
blocks when rendering Markdown.

This works by walking through the Comrak AST and replacing any code
blocks with pre-rendered HTML blocks.

Syntaxes are chosen based on the "block info", which is the string
users put after the block's opening fence. This can either be
a (case-insensitive) name of a syntax, or alternatively a file
extension associated with the desired syntax.

The theme is set to one that imitates GitHub.
2019-12-21 15:57:02 +00:00
Vincent Ambo
671dbc7f72 docs(cheddar): Mention cgit filter compatibility in README 2019-12-21 15:56:44 +00:00
Vincent Ambo
d6c255a35d feat(cheddar): Render Markdown via Comrak
Renders any ".md" file by pushing it through the Comrak rendering
pipeline.

This does not yet implement syntax highlighting of fenced blocks, but
we're getting there.
2019-12-21 14:49:35 +00:00
Vincent Ambo
d35aa4ae46 refactor(cheddar): Set up scaffolding for Markdown rendering
Generalises the two bits of the program that will be required either
way (extension parsing and syntax loading).

A dependency on Comrak is introduced as I think GitHub-flavoured
Markdown (with all its fancy extensions) is desirable!
2019-12-21 14:09:12 +00:00
Vincent Ambo
81d9b81b06 feat(cheddar): Use syntax highlighting assets from bat
This uses Nix to inject the path to the syntax highlighting assets
that ship with the bat source code into the cheddar build at compile
time, where the Rust compiler then inserts it into the binary via
macros.

bat has a lot of custom syntax highlighting definitions that they
collected from all over the place (including for languages like Nix!)
and this makes them accessible to cheddar.

Also if you're reading this, can you just take a moment to appreciate
how incredible it is that Nix just lets us do something like this?!
2019-12-21 04:55:10 +00:00
Vincent Ambo
4681c07bde feat(tools/cheddar): Check in new syntax highlighting tool
The first step with this tool will be to use it as a source-filter for
cgit. The second step is to use it as the Markdown renderer by
depending on one of the Markdown libraries, with integration for
rendering code snippets directly.
2019-12-21 04:54:57 +00:00
Vincent Ambo
03bfe08e1d chore: Significantly restructure folder layout
This moves the various projects from "type-based" folders (such as
"services" or "tools") into more appropriate semantic folders (such as
"nix", "ops" or "web").

Deprecated projects (nixcon-demo & gotest) which only existed for
testing/demonstration purposes have been removed.

(Note: *all* builds are broken with this commit)
2019-12-20 20:18:41 +00:00
Vincent Ambo
e900c4279b feat(journaldriver): Add Naersk-based build configuration 2019-12-20 20:03:36 +00:00
Vincent Ambo
384ade5e39 merge(journaldriver): Merge journaldriver into tools/journaldriver 2019-12-20 19:44:19 +00:00
Vincent Ambo
5c0f874229 chore(tools/journaldriver): Prepare tree for depot merge 2019-12-20 19:43:50 +00:00
Vincent Ambo
7158d56499 fix(tools): Correctly resolve kms_pass->pass
The binary name doesn't match the one inside of the derivation, but
this is intentional.
2019-12-20 13:13:06 +00:00
Vincent Ambo
da9dd45ebf fix(emacs.d): Correctly restore default font size in randr-single 2019-12-20 12:37:16 +00:00
Vincent Ambo
90ebd154ca feat(emacs.d): Add key binding for opening telega 2019-12-19 15:19:24 +00:00
Vincent Ambo
c1a5d7d62a feat(emacs): Add telega.el support
Packages the telega-server binary and adds the required mode into
Emacs.

Unread message count is displayed in the modeline, which is neat.

Probably need to figure out some key bindings for this.
2019-12-19 14:48:22 +00:00
Vincent Ambo
166aff23bc fix(nix-util): Correctly enable lexical scope in this library 2019-12-19 14:23:14 +00:00
Vincent Ambo
c2f97577f7 feat(emacs-pkgs): Introduce nix-util helper library
Adds an Emacs library with so far a grand total of one helper function
that can prefetch and insert a git repository at point.

This is very useful for the various Go repo imports I am doing at the
moment.
2019-12-19 01:40:27 +00:00
Vincent Ambo
a40c2474fe feat(emacs.d): Set screen-specific default text scales 2019-12-17 17:48:13 +00:00
Vincent Ambo
be28071e56 feat(emacs.d): Add functions & bindings to manage global font size
This makes it possible to quickly adjust the size of text in all
frames using one keyboard shortcut. Each of these functions
understands a prefix argument to mean "please only operate on the
current buffer", hence the following bindings and effects:

Global:
* `C-=`: Increase the global font size (chosen because `+` lies on the
  `=` key)
* `C--`: Decrease the global font size
* `C-x C-0`: Restore the global default font size

Local:
* `C-u C-=`: Increase the local font size
* `C-u C--`: Decrease the local font size
* `C-u C-x C-0`: Restore the local default font size
2019-12-17 17:48:13 +00:00
Vincent Ambo
caf23a9e2a chore(emacs.d): Clean up unused functions 2019-12-17 12:33:15 +00:00
Vincent Ambo
940a9c580c feat(emacs.d): Add function to insert TODO comments 2019-12-17 12:09:44 +00:00
Vincent Ambo
cc68e28514 chore(emacs.d): Include all git repos under ~/ in magit 2019-12-17 11:47:30 +00:00
Vincent Ambo
ef3e5072ef fix(emacs.d): Fix regex passing in counsel-rg 2019-12-17 11:46:58 +00:00
Vincent Ambo
5727b6d24f refactor(emacs.d): Move ivy configuration to use-package 2019-12-17 11:46:41 +00:00
Vincent Ambo
7965f3ba3a fix(emacs.d): Disable C-x C-c 2019-12-17 00:45:30 +00:00
Vincent Ambo
3a719993fc fix(emacs.d): Change binding for ivy-browse-repositories
This avoids a conflict with some work tooling.
2019-12-17 00:38:18 +00:00
Vincent Ambo
0b6365efe5 fix(emacs): withLocalEmacs now loads packages via $EMACSLOADPATH
For some reason setting the initial load-path via `--directory` does
not work and leads to an Emacs without any packages.
2019-12-17 00:37:49 +00:00
Vincent Ambo
05168b9a3c feat(emacs): Add 'withLocalEmacs' override function
This function makes it possible to build an Emacs instance that,
instead of launching an Emacs built by Nix, configures an Emacs
already present on the system to use the packages built by Nix.

This **requires** that the versions of the two Emacsen (i.e. the one
used by Nix to build and the one used to run the packages) are kept in
sync, otherwise byte-code incompatibilities may lead to undefined
behaviour.
2019-12-16 23:57:32 +00:00
Vincent Ambo
43f07099a7 feat(emacs): Install 'request.el', an HTTP client 2019-12-16 23:57:17 +00:00
Vincent Ambo
99909ddd04 feat(emacs): Add withLocalConfig override function
This function makes it possible to pass in a folder with additional
local configuration from somewhere downstream.
2019-12-16 13:33:29 +00:00
Vincent Ambo
6ae799e8c8 feat(emacs): Make list of packages overridable
Exposes an `overrideEmacs` which can take a package function that
receives the current package list and can make arbitrary modifications
to it.

This makes it possible for me to maintain a private overlay for e.g.
work purposes with packages that should not be visible in my public
repos.
2019-12-16 11:59:07 +00:00
Vincent Ambo
67a9b3cc52 feat(emacs.d): Install refine 2019-12-16 11:38:22 +00:00
Vincent Ambo
e5839306f8 fix(emacs.d): Set correct magit-repository-directories path
This was set to my old home directory name from a different machine
and I had low-key been wondering why it didn't work, but not enough to
go do something about it.
2019-12-16 10:04:46 +00:00
Vincent Ambo
6108759688 fix(term-switcher): Add dependency on vterm 2019-12-16 03:57:03 +00:00
Vincent Ambo
20ef6710e0 fix(emacs.d): Only highlight current line in prog-modes
This is extremely annoying in vterms, so now it's gone!
2019-12-16 03:56:12 +00:00
Vincent Ambo
d63996c3cb feat(term-switcher): Use emacs-libvterm instead of launching X-terms 2019-12-16 03:51:28 +00:00
Vincent Ambo
e991834275 feat(emacs.d): Install and configure emacs-libvterm
This incredible package provides a fully functional, libvterm based
terminal emulator inside of Emacs.

Killer feature: It's possible to switch the buffer into a full Emacs
text mode (read-only) for selections and such.
2019-12-16 03:51:28 +00:00
Vincent Ambo
6b72c45ad3 feat(aoc2019): Add solutions for day4
Finally an excuse to use some cl-defstruct magic in Elisp!
2019-12-16 01:42:29 +00:00
Vincent Ambo
cc51fb6ce7 feat(aoc2019): Add solution for day3/2 2019-12-16 00:45:25 +00:00
Vincent Ambo
458163910b chore(emacs.d): Use local term-switcher package 2019-12-15 22:55:43 +00:00
Vincent Ambo
f34c82f765 refactor(emacs-pkgs): Extract term-switcher into separate emacs pkg 2019-12-15 22:54:58 +00:00
Vincent Ambo
9f8542fe75 chore(emacs.d): Remove smartparens
I actually find a lot of its behaviour very annoying, rather than
smart. Good old electric-pair-mode usually does what I want.
2019-12-15 22:54:58 +00:00
Vincent Ambo
97555884e2 fix(emacs.d): Fix uses use-package's :config instead of :init
In several places I used `:init` instead of `:config` and things
worked due to lazy loading anyways.

That's no excuse not to fix it though ...
2019-12-15 22:54:58 +00:00
Vincent Ambo
6ee061e205 refactor(emacs.d): Use dottime from my dottime package
Instead of locally advicing format-time-string, use my dottime
package.
2019-12-15 22:54:58 +00:00
Vincent Ambo
2bf2567284 feat(emacs-pkgs): Check in separate dottime package
This package adds a toggle for displaying modeline time in dottime, as
well as a function for formatting time as dottime generically.
2019-12-15 22:54:58 +00:00
Vincent Ambo
02e2877193 chore(emacs): Move carp-mode to third-party 2019-12-15 20:55:46 +00:00
Vincent Ambo
00c9060c2f feat(emacs.d): Introduce custom mc/mark-dwim cursor marker
Adds a "do what I mean" multiple-cursor selection with the logic that
I find most useful:

* If there is no active region, mark the next line (or lines, based on
  prefix argument)
* If there is an active region that spans multiple lines, call
  `mc/edit-lines`
* If there is an active region on a single line, trigger a custom
  selection hydra with functionality equivalent to
  `mc/mark-more-like-this-extended` but a slightly improved user
  experience

Hopefully this will make it easier to get into the habit of actually
using multiple-cursors without calling the mc commands via M-x
2019-12-15 17:13:17 +00:00
Vincent Ambo
895deabb3a chore(emacs.d): Comment out not-currently-used email bindings 2019-12-15 17:09:39 +00:00
Vincent Ambo
c16d9bb9b8 chore(tools): Add 'rink' 2019-12-14 17:53:33 +00:00
Vincent Ambo
cbe7b47a96 refactor(tools): __dispatch.sh now uses relative paths
This enables usage of __dispatch.sh from anywhere, even outside of the
depot.

Specifically this means I can add `~/depot/bin` to my $PATH and all
the registered tools work from anywhere.
2019-12-14 17:40:21 +00:00
Vincent Ambo
f69dfbc40e fix(emacs.d): Fix loading of custom.el 2019-12-14 17:15:47 +00:00
Vincent Ambo
c8bf0f9008 feat(emacs.d): Add function to open depot in magit 2019-12-14 17:07:25 +00:00
Vincent Ambo
7315ca331e chore(emacs.d): Clean up <tab> bindings
One simple rule to rule them all: In prog-mode, <tab> tries to invoke
company.
2019-12-14 15:59:20 +00:00
Vincent Ambo
9df9fafd82 fix(emacs.d): Let emacs write to customize configuration
Configures Emacs' `customize` to write directly to my Emacs
configuration. This comes with the caveat that the new config will
only be loaded if my Emacs is rebuilt.
2019-12-14 15:38:03 +00:00
Vincent Ambo
fadf0619eb feat(tools/emacs): Build fully pre-configured Emacs
Builds an Emacs that is not only configured with the required packages
but with the entire Emacs configuration for my personal setup.

This means that `nix-env -iA tools.emacs` will install a
fully-configured Emacs that can be launched as the window manager from
my ~/.xsession.
2019-12-14 15:25:04 +00:00
Vincent Ambo
7d03ab7140 chore(emacs.d): Move init/* to config/* 2019-12-14 15:24:53 +00:00
Vincent Ambo
1584607fb9 chore: Remove some configuration from a previous life 2019-12-14 14:56:13 +00:00
Vincent Ambo
bd7903815d feat(emacs.d): Load local Emacs configuration if present
This allows me to have local Emacs configuration that is tracked
separately from the rest.
2019-12-14 13:35:03 +00:00
Vincent Ambo
c490bbf4b3 chore(emacs.d): Import several months of privately tracked changes
These changes were made in a fork of my configuration that is not
publicly available and detailed history will not be included in this
public commit.
2019-12-14 13:23:20 +00:00
Vincent Ambo
1f204f2891 feat(emacs.d): Display modeline time in dottime
See dotti.me for more information about dottime.

<3 edef
2019-12-14 13:22:28 +00:00
Vincent Ambo
c1ba41a62d feat(emacs.d): Generalise counsel terminal selector
This makes it possible to use my terminal selector with any X11
terminal, configurable via two simple variables.
2019-12-14 13:21:12 +00:00
Vincent Ambo
2b80265a1f refactor(emacs): Rewrite derivation to match depot layout 2019-12-14 12:47:07 +00:00
Vincent Ambo
863f5b146d chore(emacs): Move Nix derivation for building Emacs over from infra 2019-12-14 11:57:11 +00:00
Vincent Ambo
a80ebfb067 docs(emacs): Update README file 2019-12-14 11:32:54 +00:00
Vincent Ambo
b938d716ff merge(emacs): Merge (unrelated) emacs.d history into tools/emacs 2019-12-14 11:31:25 +00:00
Vincent Ambo
15c61c0bee chore(emacs): Move emacs config to tools/emacs 2019-12-14 11:30:56 +00:00
Vincent Ambo
fe01b82c5d refactor(tools/blog_cli): Build with buildGo.nix
This is the first Go package properly built with external dependencies
using buildGo.nix
2019-12-13 21:31:01 +00:00
Vincent Ambo
c93130c8c3 feat(aoc2019): Solve day 3 puzzles
Now this was quite something.
2019-12-10 14:45:28 +00:00
Vincent Ambo
c81be2f836 feat(aoc2019): Solve day 2 puzzles 2019-12-09 14:43:09 +00:00
Vincent Ambo
85fa12da33 feat(aoc2019): Solve day 1 puzzles 2019-12-09 14:43:01 +00:00
Vincent Ambo
ded10fbd83 feat(tools): Add auto-dispatch for Advent of Code solutions 2019-12-09 12:27:07 +00:00
Vincent Ambo
514c48fdde feat(tools): Set up Advent of Code 2019 scaffolding 2019-12-09 11:44:21 +00:00
Vincent Ambo
46e5a23937 fix: Amend import paths to use whitelisted nixpkgs 2019-12-09 02:55:22 +00:00
Vincent Ambo
01bad09eed refactor: Introduce new layout with nixpkgs in third_party
This is not the final layout yet, but makes it so that my top-level
attribute set is no longer overlaid into nixpkgs itself.

This is useful for other people who are importing my monorepo.
2019-12-09 02:40:48 +00:00
Vincent Ambo
2b82f1b71a refactor: Reshuffle file structure for better code layout
This gets rid of the package called "server" and instead moves
everything into the project root, such that Go actually builds us a
binary called `nixery`.

This is the first step towards factoring out CLI-based functionality
for Nixery.
2019-11-27 14:12:38 +00:00
Vincent Ambo
df88da126a fix(builder): Ensure "solo-metapackages" do not break builds
The previous logic failed because single meta-packages such as
"nixery.dev/shell" would not end up removing the meta-package itself
from the list of packages passed to Nix, causing a build failure.

This was a regression introduced in 827468a.
2019-11-27 13:28:13 +00:00
Vincent Ambo
a924093d09 test(builder): Add test coverage for name->image conversion
Adds tests to cover that packages & metapackages are parsed into image
names correctly.
2019-11-27 13:28:13 +00:00
Vincent Ambo
f1655c5cbb fix(tools): Enable CI runs for tools.gotest 2019-11-22 16:29:43 +00:00
Vincent Ambo
1619f58d78 feat(tools): Add 'gotest' program to demonstrate pkgs.buildGo
This is a tiny program that does nothing but exists to demonstrate
pkgs.buildGo by building a program that depends on a local library as
well as a protobuf definition.
2019-11-22 16:28:22 +00:00
Vincent Ambo
45d63bce17 feat(nix): Filter projects that should be built by CI
Instead of specifying CI projects manually, this filters them to move
the CI configuration into the derivations `meta` attributes.
2019-11-15 23:25:41 +00:00
Vincent Ambo
c1c379848a chore(nix): Move files around to conform to new read-tree layout
Broadly speaking, the following things are included:

* there is now a uniform `args` struct that is passed to all
  derivations, package headers have been changed appropriately
* overrides are now loaded from a separate `override` folder just
  using read-tree.nix
* third-party packages have moved into the `third_party` attribute set
2019-11-15 15:26:08 +00:00
Vincent Ambo
104c930040 chore(build): Use significantly fewer layers for Nixery itself
Nixery itself is built with the buildLayeredImage system, which takes
some time to create large numbers of layers.

This adjusts the default number of image layers from 96 to 20.

Additionally Nixery's image is often loaded with `docker load -i`,
which ignores layer cache hits anyways.

Additionaly the CI build is configured to use only 1, which speeds up
CI runs.
2019-11-09 15:35:58 +00:00
Vincent Ambo
9a8abeff97 feat(build): Integration test on both CPU architectures 2019-11-09 15:35:58 +00:00
Vincent Ambo
1d6898a7cc feat(build): Include arm64 in build matrix 2019-11-09 15:35:58 +00:00
Vincent Ambo
145b7f4289 fix(build-image): Allow "cross-builds" of images for different arch
Imports the package set twice in the builder expression: Once
configured for the target system, once configured for the native
system.

This makes it possible to fetch the actual image contents for the
required architecture, but use local tools to assemble the symlink
layer and metadata.
2019-11-09 15:35:58 +00:00
Vincent Ambo
d7ccf35149 feat(builder): Support 'arm64' meta-package
Specifying this meta-package toggles support for ARM64 images, for
example:

    # Pull a default x86_64 image
    docker pull nixery.dev/hello

    # Pull an ARM64 image
    docker pull nixery.dev/arm64/hello
2019-11-09 15:35:58 +00:00
Vincent Ambo
3c2de4c037 refactor(builder): Parameterise CPU architecture to use for images
Adds the CPU architecture to the image configuration. This will make
it possible to let users toggle architecture via meta-packages.

Relates to #13
2019-11-09 15:35:58 +00:00
Vincent Ambo
7afbc912ce chore(build): Add nixery-popcount to top-level package set 2019-11-03 01:33:36 +00:00
Vincent Ambo
05b5b1718a feat(popcount): Cache seen narinfos on disk 2019-11-03 01:33:36 +00:00
Vincent Ambo
6a2fb092a7 chore: Add missing copyright headers to source files 2019-11-03 01:33:36 +00:00
Vincent Ambo
b03f7a1b4d feat(popcount): Add new narinfo-based popcount implementation
Adds an implementation of popcount that, instead of realising
derivations locally, just queries the cache's narinfo files.

The downside of this is that calculating popularity for arbitrary Nix
package sets is not possible with this implementation. The upside is
that calculating the popularity for an entire Nix channel can now be
done in ~10 seconds[0].

This fixes #65.

[0]: Assuming a /fast/ internet connection.
2019-11-03 01:33:36 +00:00
Vincent Ambo
2d4a3ea307 chore(server): Remove outdated TODO
Real-life experience has shown that the weighting of the metric
produced here is appropriate.
2019-10-29 19:16:24 +01:00
Vincent Ambo
904c3dade0 refactor(server): Handle non-error errors safely
This case should not be possible unless something manually constructs
a logrus entry with a non-error value in the log.ErrorKey field, but
it's better to be safe than sorry.
2019-10-29 19:16:24 +01:00
Vincent Ambo
3a7c964a22 chore(build): Configure build to use new GCS configuration options 2019-10-28 22:31:44 +01:00
Vincent Ambo
ab190256ab fix(server): Use correct scope for GCS tokens 2019-10-28 22:31:44 +01:00
Vincent Ambo
3611baf040 docs(under-the-hood): Update builder & storage backend information
Both of these no longer matched the reality of what was actually going
on in Nixery.
2019-10-28 22:31:44 +01:00
Vincent Ambo
b736f5580d docs: Add storage configuration options to README 2019-10-28 22:31:44 +01:00
Vincent Ambo
d8fba23365 fix(server): Thread request context to all relevant places
Previously background contexts where created where necessary (e.g. in
GCS interactions). Should I begin to use request timeouts or other
context-dependent things in the future, it's useful to have the actual
HTTP request context around.

This threads the request context through the application to all places
that need it.
2019-10-28 22:31:44 +01:00
Vincent Ambo
30e618b65b chore(server): Move cache miss log statement to debug level
This is very annoying otherwise.
2019-10-28 22:31:44 +01:00
Vincent Ambo
4332d38f4f fix(server): Correctly construct filesystem paths for layer serving 2019-10-28 22:31:44 +01:00
Vincent Ambo
b60a8d007b fix(server): Ensure paths exist when renaming in filesystem storage
The point at which files are moved happens to also (initially) be the
point where the `layers` directory is created. For this reason
renaming must ensure that all path components exist, which this commit
takes care of.
2019-10-28 22:31:44 +01:00
Vincent Ambo
c08aa52558 fix(server): Ensure error messages are correctly printed in logs
I assumed (incorrectly) that logrus would already take care of
surfacing error messages in human-readable form.
2019-10-28 22:31:44 +01:00
Vincent Ambo
790bce219c feat(server): Add filesystem storage backend config options
The filesystem storage backend can be enabled by setting
`NIXERY_STORAGE_BACKEND` to `filesystem` and `STORAGE_PATH` to a disk
location from which Nixery can serve files.
2019-10-28 22:31:44 +01:00
Vincent Ambo
167a0b3263 refactor(server): Pass HTTP request to storage.ServeLayer
The request object is required for some serving methods (e.g. the
filesystem one).
2019-10-28 22:31:44 +01:00
Vincent Ambo
e5bb2fc887 feat(server): Implement initial filesystem storage backend
This allows users to store and serve layers from a local filesystem
path.
2019-10-28 22:31:44 +01:00
Vincent Ambo
e8fd6b6734 refactor(server): Change setup to create new storage backends 2019-10-28 22:31:44 +01:00
Vincent Ambo
20e0ca53cb feat(server): Implement GCS storage backend with new interface
Logical implementation is mostly identical to the previous one, but
adhering to the new storage.Backend interface.
2019-10-28 22:31:44 +01:00
Vincent Ambo
f7d16c5d45 refactor(server): Introduce pluggable interface for storage backends
This abstracts over the functionality of Google Cloud Storage and
other potential underlying storage backends to make it possible to
replace these in Nixery.

The GCS backend is not yet reimplemented.
2019-10-28 22:31:44 +01:00
Vincent Ambo
ffe58d6cb5 refactor(build): Do not expose nixery-server attribute
In most cases this is not useful for users without the wrapper script,
so users should always build nixery-bin anyways.
2019-10-28 22:31:44 +01:00
Vincent Ambo
7b7d21205f docs: Update GCS signing key documentation
This key is now taken straight from the configured service account
key.
2019-10-27 13:58:04 +01:00
Vincent Ambo
3a5db4f9f1 refactor(server): Load GCS signing key from service account key
The JSON file generated for service account keys already contains the
required information for signing URLs in GCS, thus the environment
variables for toggling signing behaviour have been removed.

Signing is now enabled automatically in the presence of service
account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
2019-10-27 13:58:04 +01:00
Vincent Ambo
cca835ae37 fix(build): Only take the first matching hash for source hashing
Some Nix download mechanisms will add a second hash in the store path,
which had been added to the source hash output (breaking argument
interpolation).
2019-10-11 14:23:41 +01:00
Vincent Ambo
1853c74998 refactor(server): Only compress symlink forest layer once
Instead of compressing & decompressing again to get the underlying tar
hash, use a similar mechanism as for store path layers for the symlink
layer and only compress it once while uploading.
2019-10-11 12:37:00 +01:00
Vincent Ambo
e22ff5d176 fix(server): Use uncompressed tarball hashes in image config
Docker expects hashes of compressed tarballs in the manifest (as these
are used to fetch from the content-addressable layer store), but for
some reason it expects hashes in the configuration layer to be of
uncompressed tarballs.

To achieve this an additional SHA256 hash is calculcated while
creating the layer tarballs, but before passing them to the gzip
writer.

In the current constellation the symlink layer is first compressed and
then decompressed again to calculate its hash. This can be refactored
in a future change.
2019-10-11 12:37:00 +01:00
Vincent Ambo
0693e371d6 feat(server): Apply GZIP compression to all image layers
This fixes #62
2019-10-11 12:37:00 +01:00
Vincent Ambo
bf2718cebb chore(build): Use separate GCS bucket for CI runs
This has become an issue recently with changes such as GZIP
compression, where CI runs no longer work because they conflict with
the production bucket for the public instance.
2019-10-11 12:37:00 +01:00
Vincent Ambo
d7ffbbdea4 refactor(server): Use logrus convenience functions for logs
Makes use of the `.WithError` and `.WithField` convenience functions
in logrus to simplify log statement construction.

This has the added benefit of making it easier to correctly log
errors.
2019-10-06 23:05:23 +01:00
Vincent Ambo
c1020754a2 fix(build-image): Import 'match' from builtins 2019-10-06 23:05:23 +01:00
Vincent Ambo
6f148f789f refactor(server): Convert existing log entries to structured format
This rewrites all existing log statements into the structured logrus
format. For consistency, all errors are always logged separately from
the primary message in a field called `error`.

Only the "info", "error" and "warn" severities are used.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f77c93b6ae feat(server): Add log level to severity mapping
The output format now writes a `severity` field that follows that
format that should be recognised by Stackdriver Logging.
2019-10-06 23:05:23 +01:00
Vincent Ambo
6912658c72 feat(server): Use hash of Nixery source as version
Uses a hash of Nixery's sources as the version displayed when Nixery
launches or logs an error. This makes it possible to distinguish
between errors logged from different versions.

The source hashes should be reproducible between different checkouts
of the same source tree.
2019-10-06 23:05:23 +01:00
Vincent Ambo
95abb1bcde feat(server): Initial Stackdriver-compatible log formatter
This formatter has basic support for the Stackdriver Error Reporting
format, but several things are still lacking:

* the service version (preferably git commit?) needs to be included in
  the server somehow
* log streams should be split between stdout/stderr as that is how
  AppEngine (and several other GCP services?) seemingly differentiate
  between info/error logs
2019-10-06 23:05:23 +01:00
Vincent Ambo
0642f7044d fix(server): Amend package path for Go tooling compatibility
With these changes it is possible to keep Nixery in $GOPATH and build
the server in there, while still having things work correctly via Nix.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f4bf3518f6 refactor(server): Replace log calls with logrus
This introduces a structured logging library that can be used (next
step) to attach additional metadata to log entries.
2019-10-06 23:05:23 +01:00
Vincent Ambo
d9b329ef59 refactor(server): Always include 'cacert' & 'iana-etc'
These two packages almost always end up being required by programs,
but people don't necessarily consider them.

They will now always be added and their popularity is artificially
inflated to ensure they end up at the top of the layer list.
2019-10-03 22:50:02 +01:00
Vincent Ambo
9bb6d0ae25 fix(server): Ensure build cache objects are written to GCS
Cache writes might not be flushed without this call.
2019-10-03 22:50:02 +01:00
Vincent Ambo
48a5ecda97 feat(server): Order layers in image manifest based on merge rating
Image layers in manifests are now sorted in a stable (descending)
order based on their merge rating, meaning that layers more likely to
be shared between images come first.

The reason for this change is Docker's handling of image layers on
overlayfs2: Images are condensed into a single representation on disk
after downloading.

Due to this Docker will constantly redownload all layers that are
applied in a different order in different images (layer order matters
in imperatively created images), based on something it calls the
'ChainID'.

Sorting the layers this way raises the likelihood of a long chain of
matching layers at the beginning of an image.

This relates to #39.
2019-10-03 22:50:02 +01:00
Vincent Ambo
0d820423e9 chore(build-image): Remove nixery-build-layers
This functionality has been rolled into the server component and is no
longer required.
2019-10-03 22:29:50 +01:00
Vincent Ambo
6b06fe27be feat(server): Implement creation of layer tarballs in the server
This will create, upload and hash the layer tarballs in one disk read.
2019-10-03 22:29:50 +01:00
Vincent Ambo
1124b8c236 fix(server): Do not invoke layer build if no layers are missing
This previously invoked a Nix derivation that spent a few seconds on
making an empty object in JSON ...
2019-10-03 13:21:04 +01:00
Vincent Ambo
feba42e409 feat(server): Fetch popularity data on launch
The last missing puzzle piece for #50!
2019-10-03 13:21:04 +01:00
Vincent Ambo
43a642435b feat(server): Reimplement local manifest cache backed by files
Implements a local manifest cache that uses the temporary directory to
cache manifest builds.

This is necessary due to the size of manifests: Keeping them entirely
in-memory would quickly balloon the memory usage of Nixery, unless
some mechanism for cache eviction is implemented.
2019-10-03 13:21:04 +01:00
Vincent Ambo
313e5d08f1 refactor(builder): Streamline layer creation & reintroduce caching
The functions used for layer creation are now easier to follow and
have clear points at which the layer cache is checked and populated.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
53906024ff refactor: Remove remaining MD5-hash mentions and computations 2019-10-03 13:21:04 +01:00
Vincent Ambo
f6b40ed6c7 refactor(server): Cache manifest entries for layer builds
MD5 hash checking is no longer performed by Nixery (it does not seem
to be necessary), hence the layer cache now only keeps the SHA256 hash
and size in the form of the manifest entry.

This makes it possible to restructure the builder code to perform
cache-fetching and cache-populating for layers in the same place.
2019-10-03 13:21:04 +01:00
Vincent Ambo
355fe3f5ec feat(server): Reintroduce manifest caching to GCS
The new builder now caches and reads cached manifests to/from GCS. The
in-memory cache is disabled, as manifests are no longer written to
local file and the caching of file paths does not work (unless we
reintroduce reading/writing from temp files as part of the local
cache).
2019-10-03 13:21:04 +01:00
Vincent Ambo
1308a6e1fd refactor(server): Clean up cache implementation
A couple of minor fixes and improvements to the cache implementation.
2019-10-03 13:21:04 +01:00
Vincent Ambo
0698d7f2aa chore(server): Remove "layer seen" cache
This cache is no longer required as it is implicit because the layer
cache (mapping store path hashes to layer hashes) implies that a layer
has been seen.
2019-10-03 13:21:04 +01:00
Vincent Ambo
64fca61ea1 fix(server): Upload symlink layer created by first Nix build
This layer is needed in addition to those that are built in the second
Nix build.
2019-10-03 13:21:04 +01:00
Vincent Ambo
f4f2909573 fix(server): Specify correct authentication scope for GCS
When retrieving tokens for service service accounts, some methods of
retrieval require a scope to be specified.
2019-10-03 13:21:04 +01:00
Vincent Ambo
aa02ae1421 feat(server): Implement new build process core
Implements the new build process to the point where it can actually
construct and serve image manifests.

It is worth noting that this build process works even if the Nix
sandbox is enabled!

It is also worth nothing that none of the caching functionality that
the new build process enables (such as per-layer build caching) is
actually in use yet, hence running Nixery at this commit is prone to
doing more work than previously.

This relates to #50.
2019-10-03 13:21:04 +01:00