Commit graph

573 commits

Author SHA1 Message Date
Vincent Ambo
bbab6cd1c1 feat(tools/emacs): Use patched rcirc 2020-02-24 16:40:43 +00:00
Vincent Ambo
4bbbb58cb5 chore: Rename pkgs->depot in all Nix file headers 2020-02-21 13:54:53 +00:00
Vincent Ambo
14b5447aac Merge branch 'feat/cheddar-extensions' 2020-02-09 12:07:10 +00:00
Vincent Ambo
31b3e533c6 chore(tools): Remove //tools/blog_cli
No longer needed with the deprecation of tazblog.
2020-02-09 01:30:25 +00:00
Vincent Ambo
ec42a67569 feat(tools/emacs): Bump EXWM to recent master
This version of EXWM fixes some issues with binding keys in EXWM as
well as other minor annoyances.
2020-02-07 12:41:43 +00:00
Vincent Ambo
c56968a767 feat(tools/emacs): Install ace-link
It now has vterm-mode support, which has finally convinced me to try it.
2020-02-05 11:58:57 +00:00
Vincent Ambo
dba2e5426e fix(emacs-pkgs/nix-util): Don't leave sly build out-links around 2020-01-27 02:11:04 +00:00
Vincent Ambo
a2d0b7f399 feat(emacs.d): Add interactive 'scrot-select' screenshot function 2020-01-26 19:13:59 +00:00
Vincent Ambo
fb5f26e548 feat(emacs.d): Add notmuch-depot-apply-patch helper function
This function lets me interactively apply a patch from the currently
opened notmuch message to the depot.
2020-01-24 18:51:13 +00:00
Vincent Ambo
98cc5f9fac feat(emacs.d): Add key binding for counsel-notmuch 2020-01-22 17:26:11 +00:00
Vincent Ambo
8f68497269 fix(emacs-pkgs/nix-util): Ensure compatibility with non-native JSON 2020-01-20 00:58:46 +00:00
Vincent Ambo
d752cbecc0 feat(cheddar): Add toggle flag for about-filter behaviour
Cheddar now needs to be passed the --about-filter flag to toggle the
behaviour for rendering Markdown into HTML.

By default Markdown will be highlighted like normal source code (i.e.
cgit source-filtering is the default behaviour).
2020-01-19 23:54:30 +00:00
Vincent Ambo
0b3b8a07c4 feat(emacs): Install geiser for Guile 2020-01-19 19:35:27 +00:00
Vincent Ambo
9879703155 fix(emacs-pkgs/dottime): Do not call telega macros in advice setup
This advice is potentially defined before the autoloads for telega
have run, which means that the macro-expansion fails and
`telega-ins-fmt` is looked up as a function.

With this setup the initialisation works as expected.
2020-01-19 16:48:11 +00:00
Vincent Ambo
f09c6a8ecd feat(emacs.d): Switch font to JetBrains Mono
Lets evaluate this for a bit. With the current settings it even seems
to render _okay_ on nugget.
2020-01-19 16:39:21 +00:00
Vincent Ambo
1031d890ec fix(builder): Fix minor logging switcharoo 2020-01-19 07:44:36 +00:00
Vincent Ambo
3567f8bd0a fix(emacs.d): Use "light" Input font on nugget 2020-01-19 02:27:54 +00:00
Vincent Ambo
bdd488ce27 fix(emacs-pkgs/nix-util): Use lexical-let to allow variable capture
The lambda that acts as the sentinel for building SBCL with packages
needs to be able to capture variables if lexical binding is enabled,
which is made possible by the lexical-let form.
2020-01-17 19:09:46 +00:00
Vincent Ambo
815a224744 feat(emacs): Install lispy for evaluation
Not configuring this automatically yet, I mostly want to try it out.
2020-01-17 17:48:13 +00:00
Vincent Ambo
86c8275a53 fix(emacs.d): Load relevant modes for Lisp development in IELM 2020-01-17 16:52:17 +00:00
Vincent Ambo
381a859b3b feat(emacs-pkgs/nix-util): Add nix/sly-from-depot function
Adds a function that can launch Sly with a pre-configured SBCL for a
Lisp derivation in the depot.

This makes it convenient to spin up development environments for Lisp
libraries and programs by simply calling `M-x nix/sly-from-depot RET
tools.something`.

This relies on `nix-depot-path` being configured currently as I have
not yet reliably added the depot to my NIX_PATH on all machines.
2020-01-17 16:43:49 +00:00
Vincent Ambo
bfd0bc2a79 feat(emacs.d): Configure Sly for Common Lisp
Sets up Lisp modes in Sly REPL and points at the local hyperspec
checkout.

In fact the Hyperspec bit should probably be managed by Nix, but one
step at a time.
2020-01-15 11:42:19 +01:00
Vincent Ambo
829de752da fix(emacs.d): Trim superfluous whitespace from TODO comments 2020-01-11 23:23:46 +00:00
Vincent Ambo
2236d43ff7 feat(cheddar): Implement callout paragraphs
Implements support for tagging paragraphs that begin with a callout
word (TODO, WARNING, QUESTION, TIP) with an additional `cheddar-*`
class that makes it possible to render these callouts specially.

This is currently not the nicest implementation, but it works.
2020-01-11 05:06:36 +00:00
Vincent Ambo
d4e469508f refactor(cheddar): Extract code block highlighting into function
Since I am going down the path of adding additional Markdown
extensions it makes sense to avoid letting `format_markdown` turn into
a giant beast of a function.

Therefore this commit extracts the logic for rendering code blocks via
syntect and changes the innards of `format_markdown` to instead
provide arbitrary AST value replacements.
2020-01-11 02:17:19 +00:00
Vincent Ambo
cd0093d46e fix(emacs): Minor typo fix 2020-01-08 00:32:56 +00:00
Vincent Ambo
9e2f86d65d chore(emacs): Bump emacs-libvterm to a more recent version 2020-01-07 22:28:36 +00:00
Vincent Ambo
efc782fcdc fix(emacs): Disable linking of imagemagick due to vterm crashes
See explanatory comment.
2020-01-07 22:28:17 +00:00
Vincent Ambo
1e71b80f6a fix(emacs): Build emacs with imagemagick support 2020-01-05 17:37:01 +00:00
Vincent Ambo
b2cd30bfd4 chore(emacs.d): Remove edwina configuration
Sounds good, doesn't work.

(Okay, it does - but not like I want it to and with too many caveats
at the moment - maybe later)
2020-01-05 01:13:00 +00:00
Vincent Ambo
3638048c9b fix(emacs.d): Use 'fish' from $PATH when launching vterms
This is required because the configuration is used on machines where
fish comes from Nix, and on ones where it does not.
2020-01-04 22:50:19 +00:00
Vincent Ambo
41232a1c78 feat(emacs.d): Toggle email signature manually with C-c C-w
The signature itself is read from ~/.signature
2020-01-01 16:28:47 +01:00
Vincent Ambo
39131d9e0f fix(emacs.d): Disable notmuch FCC directories 2019-12-30 22:19:56 +01:00
Vincent Ambo
08dd267c19 chore(dottime.el): Remove advice for notmuch date rendering
This code has moved into a patch for notmuch itself.
2019-12-27 19:18:00 +01:00
Vincent Ambo
100f451b0a fix(dottime.el): Defer package-specific setup 2019-12-27 13:04:49 +01:00
Vincent Ambo
a54cc040ad fix(emacs.d): Load dottime.el after libraries that it modifies 2019-12-27 12:54:12 +01:00
Vincent Ambo
309d1ac50f feat(dottime.el): Support dottime display in notmuch thread view
Advises notmuch to display `Date` headers using dottime.
2019-12-27 12:54:12 +01:00
Vincent Ambo
29e71053be feat(dottime.el): Add arbitrary offsets & prefixes to dottime-format
This gives users from Elisp slightly more flexibility about the
display of dottime.
2019-12-27 12:54:12 +01:00
Vincent Ambo
5fd7de921b feat(dottime.el): Use dottime in telega.el, too 2019-12-27 12:54:12 +01:00
Vincent Ambo
b48797e4ce chore(emacs.d): Remove explicit FCC dir for private account
No longer required because Gmail does this automatically.
2019-12-26 15:14:37 +01:00
Vincent Ambo
cf9dd380f3 chore(emacs.d): Remove GPG agent warmup from mail sending
Fetching credentials is no longer handled by msmtp itself.
2019-12-26 14:13:40 +01:00
Vincent Ambo
29af390e8c fix(emacs.d): Slightly more reliable way of setting edwina prefix 2019-12-25 14:10:00 +01:00
Vincent Ambo
b59788779a fix(emacs.d): Display newest emails first in notmuch 2019-12-25 14:09:42 +01:00
Vincent Ambo
4a0cf0de98 fix(emacs.d): Avoid advising split-window-* functions
Advising these functions apparently breaks things internally.
2019-12-24 18:43:52 +01:00
Vincent Ambo
637cb0bd2a fix(emacs.d): Pick new buffers *into* the active window
Instead of splitting below and moving the target buffer into the new
split, split and move the buffer into the active window.

The other way around does (for some reason I don't fully understand)
not work because `split-window-below` may return invalid windows.
2019-12-24 18:15:30 +01:00
Vincent Ambo
ceaa0ff0ba feat(emacs.d): Advise split-window-below to call edwina-arrange 2019-12-24 15:43:24 +01:00
Vincent Ambo
86c2ebc096 feat(emacs.d): Enable edwina package for window management
Configures edwina using `s-w` as the key prefix (in line with my other
EXWM-related commands).

An additional function is added that switches to a buffer (borrowing
the implementation from `ivy-switch-buffer`) but splitting it into a
new window instead.
2019-12-24 15:30:25 +01:00
Vincent Ambo
ccd31b8898 feat(emacs.d): Bind 's-g m' to notmuch UI 2019-12-23 16:05:05 +01:00
Vincent Ambo
a36e9f817d fix(cheddar): Fix errors if filename does not have an extension 2019-12-22 15:21:23 +00:00
Vincent Ambo
f1c018af18 feat(emacs): Install edwina package
No configuration for this yet, I just want to try it out. It seems
like an interesting way of managing Emacs windows!
2019-12-22 14:30:28 +00:00
Vincent Ambo
03acae2f85 feat(cheddar): Style pre-elements for better readability
Uses GitHub-like styling for <pre> elements, i.e. slight padding and
background colour highlighting.
2019-12-21 16:18:26 +00:00
Vincent Ambo
f904951384 feat(cheddar): Render code blocks in Markdown via syntect
Implements fully static (i.e. no JavaScript!) highlighting of code
blocks when rendering Markdown.

This works by walking through the Comrak AST and replacing any code
blocks with pre-rendered HTML blocks.

Syntaxes are chosen based on the "block info", which is the string
users put after the block's opening fence. This can either be
a (case-insensitive) name of a syntax, or alternatively a file
extension associated with the desired syntax.

The theme is set to one that imitates GitHub.
2019-12-21 15:57:02 +00:00
Vincent Ambo
671dbc7f72 docs(cheddar): Mention cgit filter compatibility in README 2019-12-21 15:56:44 +00:00
Vincent Ambo
d6c255a35d feat(cheddar): Render Markdown via Comrak
Renders any ".md" file by pushing it through the Comrak rendering
pipeline.

This does not yet implement syntax highlighting of fenced blocks, but
we're getting there.
2019-12-21 14:49:35 +00:00
Vincent Ambo
d35aa4ae46 refactor(cheddar): Set up scaffolding for Markdown rendering
Generalises the two bits of the program that will be required either
way (extension parsing and syntax loading).

A dependency on Comrak is introduced as I think GitHub-flavoured
Markdown (with all its fancy extensions) is desirable!
2019-12-21 14:09:12 +00:00
Vincent Ambo
81d9b81b06 feat(cheddar): Use syntax highlighting assets from bat
This uses Nix to inject the path to the syntax highlighting assets
that ship with the bat source code into the cheddar build at compile
time, where the Rust compiler then inserts it into the binary via
macros.

bat has a lot of custom syntax highlighting definitions that they
collected from all over the place (including for languages like Nix!)
and this makes them accessible to cheddar.

Also if you're reading this, can you just take a moment to appreciate
how incredible it is that Nix just lets us do something like this?!
2019-12-21 04:55:10 +00:00
Vincent Ambo
4681c07bde feat(tools/cheddar): Check in new syntax highlighting tool
The first step with this tool will be to use it as a source-filter for
cgit. The second step is to use it as the Markdown renderer by
depending on one of the Markdown libraries, with integration for
rendering code snippets directly.
2019-12-21 04:54:57 +00:00
Vincent Ambo
03bfe08e1d chore: Significantly restructure folder layout
This moves the various projects from "type-based" folders (such as
"services" or "tools") into more appropriate semantic folders (such as
"nix", "ops" or "web").

Deprecated projects (nixcon-demo & gotest) which only existed for
testing/demonstration purposes have been removed.

(Note: *all* builds are broken with this commit)
2019-12-20 20:18:41 +00:00
Vincent Ambo
e900c4279b feat(journaldriver): Add Naersk-based build configuration 2019-12-20 20:03:36 +00:00
Vincent Ambo
384ade5e39 merge(journaldriver): Merge journaldriver into tools/journaldriver 2019-12-20 19:44:19 +00:00
Vincent Ambo
5c0f874229 chore(tools/journaldriver): Prepare tree for depot merge 2019-12-20 19:43:50 +00:00
Vincent Ambo
7158d56499 fix(tools): Correctly resolve kms_pass->pass
The binary name doesn't match the one inside of the derivation, but
this is intentional.
2019-12-20 13:13:06 +00:00
Vincent Ambo
da9dd45ebf fix(emacs.d): Correctly restore default font size in randr-single 2019-12-20 12:37:16 +00:00
Vincent Ambo
90ebd154ca feat(emacs.d): Add key binding for opening telega 2019-12-19 15:19:24 +00:00
Vincent Ambo
c1a5d7d62a feat(emacs): Add telega.el support
Packages the telega-server binary and adds the required mode into
Emacs.

Unread message count is displayed in the modeline, which is neat.

Probably need to figure out some key bindings for this.
2019-12-19 14:48:22 +00:00
Vincent Ambo
166aff23bc fix(nix-util): Correctly enable lexical scope in this library 2019-12-19 14:23:14 +00:00
Vincent Ambo
c2f97577f7 feat(emacs-pkgs): Introduce nix-util helper library
Adds an Emacs library with so far a grand total of one helper function
that can prefetch and insert a git repository at point.

This is very useful for the various Go repo imports I am doing at the
moment.
2019-12-19 01:40:27 +00:00
Vincent Ambo
a40c2474fe feat(emacs.d): Set screen-specific default text scales 2019-12-17 17:48:13 +00:00
Vincent Ambo
be28071e56 feat(emacs.d): Add functions & bindings to manage global font size
This makes it possible to quickly adjust the size of text in all
frames using one keyboard shortcut. Each of these functions
understands a prefix argument to mean "please only operate on the
current buffer", hence the following bindings and effects:

Global:
* `C-=`: Increase the global font size (chosen because `+` lies on the
  `=` key)
* `C--`: Decrease the global font size
* `C-x C-0`: Restore the global default font size

Local:
* `C-u C-=`: Increase the local font size
* `C-u C--`: Decrease the local font size
* `C-u C-x C-0`: Restore the local default font size
2019-12-17 17:48:13 +00:00
Vincent Ambo
caf23a9e2a chore(emacs.d): Clean up unused functions 2019-12-17 12:33:15 +00:00
Vincent Ambo
940a9c580c feat(emacs.d): Add function to insert TODO comments 2019-12-17 12:09:44 +00:00
Vincent Ambo
cc68e28514 chore(emacs.d): Include all git repos under ~/ in magit 2019-12-17 11:47:30 +00:00
Vincent Ambo
ef3e5072ef fix(emacs.d): Fix regex passing in counsel-rg 2019-12-17 11:46:58 +00:00
Vincent Ambo
5727b6d24f refactor(emacs.d): Move ivy configuration to use-package 2019-12-17 11:46:41 +00:00
Vincent Ambo
7965f3ba3a fix(emacs.d): Disable C-x C-c 2019-12-17 00:45:30 +00:00
Vincent Ambo
3a719993fc fix(emacs.d): Change binding for ivy-browse-repositories
This avoids a conflict with some work tooling.
2019-12-17 00:38:18 +00:00
Vincent Ambo
0b6365efe5 fix(emacs): withLocalEmacs now loads packages via $EMACSLOADPATH
For some reason setting the initial load-path via `--directory` does
not work and leads to an Emacs without any packages.
2019-12-17 00:37:49 +00:00
Vincent Ambo
05168b9a3c feat(emacs): Add 'withLocalEmacs' override function
This function makes it possible to build an Emacs instance that,
instead of launching an Emacs built by Nix, configures an Emacs
already present on the system to use the packages built by Nix.

This **requires** that the versions of the two Emacsen (i.e. the one
used by Nix to build and the one used to run the packages) are kept in
sync, otherwise byte-code incompatibilities may lead to undefined
behaviour.
2019-12-16 23:57:32 +00:00
Vincent Ambo
43f07099a7 feat(emacs): Install 'request.el', an HTTP client 2019-12-16 23:57:17 +00:00
Vincent Ambo
99909ddd04 feat(emacs): Add withLocalConfig override function
This function makes it possible to pass in a folder with additional
local configuration from somewhere downstream.
2019-12-16 13:33:29 +00:00
Vincent Ambo
6ae799e8c8 feat(emacs): Make list of packages overridable
Exposes an `overrideEmacs` which can take a package function that
receives the current package list and can make arbitrary modifications
to it.

This makes it possible for me to maintain a private overlay for e.g.
work purposes with packages that should not be visible in my public
repos.
2019-12-16 11:59:07 +00:00
Vincent Ambo
67a9b3cc52 feat(emacs.d): Install refine 2019-12-16 11:38:22 +00:00
Vincent Ambo
e5839306f8 fix(emacs.d): Set correct magit-repository-directories path
This was set to my old home directory name from a different machine
and I had low-key been wondering why it didn't work, but not enough to
go do something about it.
2019-12-16 10:04:46 +00:00
Vincent Ambo
6108759688 fix(term-switcher): Add dependency on vterm 2019-12-16 03:57:03 +00:00
Vincent Ambo
20ef6710e0 fix(emacs.d): Only highlight current line in prog-modes
This is extremely annoying in vterms, so now it's gone!
2019-12-16 03:56:12 +00:00
Vincent Ambo
d63996c3cb feat(term-switcher): Use emacs-libvterm instead of launching X-terms 2019-12-16 03:51:28 +00:00
Vincent Ambo
e991834275 feat(emacs.d): Install and configure emacs-libvterm
This incredible package provides a fully functional, libvterm based
terminal emulator inside of Emacs.

Killer feature: It's possible to switch the buffer into a full Emacs
text mode (read-only) for selections and such.
2019-12-16 03:51:28 +00:00
Vincent Ambo
6b72c45ad3 feat(aoc2019): Add solutions for day4
Finally an excuse to use some cl-defstruct magic in Elisp!
2019-12-16 01:42:29 +00:00
Vincent Ambo
cc51fb6ce7 feat(aoc2019): Add solution for day3/2 2019-12-16 00:45:25 +00:00
Vincent Ambo
458163910b chore(emacs.d): Use local term-switcher package 2019-12-15 22:55:43 +00:00
Vincent Ambo
f34c82f765 refactor(emacs-pkgs): Extract term-switcher into separate emacs pkg 2019-12-15 22:54:58 +00:00
Vincent Ambo
9f8542fe75 chore(emacs.d): Remove smartparens
I actually find a lot of its behaviour very annoying, rather than
smart. Good old electric-pair-mode usually does what I want.
2019-12-15 22:54:58 +00:00
Vincent Ambo
97555884e2 fix(emacs.d): Fix uses use-package's :config instead of :init
In several places I used `:init` instead of `:config` and things
worked due to lazy loading anyways.

That's no excuse not to fix it though ...
2019-12-15 22:54:58 +00:00
Vincent Ambo
6ee061e205 refactor(emacs.d): Use dottime from my dottime package
Instead of locally advicing format-time-string, use my dottime
package.
2019-12-15 22:54:58 +00:00
Vincent Ambo
2bf2567284 feat(emacs-pkgs): Check in separate dottime package
This package adds a toggle for displaying modeline time in dottime, as
well as a function for formatting time as dottime generically.
2019-12-15 22:54:58 +00:00
Vincent Ambo
02e2877193 chore(emacs): Move carp-mode to third-party 2019-12-15 20:55:46 +00:00
Vincent Ambo
00c9060c2f feat(emacs.d): Introduce custom mc/mark-dwim cursor marker
Adds a "do what I mean" multiple-cursor selection with the logic that
I find most useful:

* If there is no active region, mark the next line (or lines, based on
  prefix argument)
* If there is an active region that spans multiple lines, call
  `mc/edit-lines`
* If there is an active region on a single line, trigger a custom
  selection hydra with functionality equivalent to
  `mc/mark-more-like-this-extended` but a slightly improved user
  experience

Hopefully this will make it easier to get into the habit of actually
using multiple-cursors without calling the mc commands via M-x
2019-12-15 17:13:17 +00:00
Vincent Ambo
895deabb3a chore(emacs.d): Comment out not-currently-used email bindings 2019-12-15 17:09:39 +00:00
Vincent Ambo
c16d9bb9b8 chore(tools): Add 'rink' 2019-12-14 17:53:33 +00:00
Vincent Ambo
cbe7b47a96 refactor(tools): __dispatch.sh now uses relative paths
This enables usage of __dispatch.sh from anywhere, even outside of the
depot.

Specifically this means I can add `~/depot/bin` to my $PATH and all
the registered tools work from anywhere.
2019-12-14 17:40:21 +00:00
Vincent Ambo
f69dfbc40e fix(emacs.d): Fix loading of custom.el 2019-12-14 17:15:47 +00:00
Vincent Ambo
c8bf0f9008 feat(emacs.d): Add function to open depot in magit 2019-12-14 17:07:25 +00:00
Vincent Ambo
7315ca331e chore(emacs.d): Clean up <tab> bindings
One simple rule to rule them all: In prog-mode, <tab> tries to invoke
company.
2019-12-14 15:59:20 +00:00
Vincent Ambo
9df9fafd82 fix(emacs.d): Let emacs write to customize configuration
Configures Emacs' `customize` to write directly to my Emacs
configuration. This comes with the caveat that the new config will
only be loaded if my Emacs is rebuilt.
2019-12-14 15:38:03 +00:00
Vincent Ambo
fadf0619eb feat(tools/emacs): Build fully pre-configured Emacs
Builds an Emacs that is not only configured with the required packages
but with the entire Emacs configuration for my personal setup.

This means that `nix-env -iA tools.emacs` will install a
fully-configured Emacs that can be launched as the window manager from
my ~/.xsession.
2019-12-14 15:25:04 +00:00
Vincent Ambo
7d03ab7140 chore(emacs.d): Move init/* to config/* 2019-12-14 15:24:53 +00:00
Vincent Ambo
1584607fb9 chore: Remove some configuration from a previous life 2019-12-14 14:56:13 +00:00
Vincent Ambo
bd7903815d feat(emacs.d): Load local Emacs configuration if present
This allows me to have local Emacs configuration that is tracked
separately from the rest.
2019-12-14 13:35:03 +00:00
Vincent Ambo
c490bbf4b3 chore(emacs.d): Import several months of privately tracked changes
These changes were made in a fork of my configuration that is not
publicly available and detailed history will not be included in this
public commit.
2019-12-14 13:23:20 +00:00
Vincent Ambo
1f204f2891 feat(emacs.d): Display modeline time in dottime
See dotti.me for more information about dottime.

<3 edef
2019-12-14 13:22:28 +00:00
Vincent Ambo
c1ba41a62d feat(emacs.d): Generalise counsel terminal selector
This makes it possible to use my terminal selector with any X11
terminal, configurable via two simple variables.
2019-12-14 13:21:12 +00:00
Vincent Ambo
2b80265a1f refactor(emacs): Rewrite derivation to match depot layout 2019-12-14 12:47:07 +00:00
Vincent Ambo
863f5b146d chore(emacs): Move Nix derivation for building Emacs over from infra 2019-12-14 11:57:11 +00:00
Vincent Ambo
a80ebfb067 docs(emacs): Update README file 2019-12-14 11:32:54 +00:00
Vincent Ambo
b938d716ff merge(emacs): Merge (unrelated) emacs.d history into tools/emacs 2019-12-14 11:31:25 +00:00
Vincent Ambo
15c61c0bee chore(emacs): Move emacs config to tools/emacs 2019-12-14 11:30:56 +00:00
Vincent Ambo
fe01b82c5d refactor(tools/blog_cli): Build with buildGo.nix
This is the first Go package properly built with external dependencies
using buildGo.nix
2019-12-13 21:31:01 +00:00
Vincent Ambo
c93130c8c3 feat(aoc2019): Solve day 3 puzzles
Now this was quite something.
2019-12-10 14:45:28 +00:00
Vincent Ambo
c81be2f836 feat(aoc2019): Solve day 2 puzzles 2019-12-09 14:43:09 +00:00
Vincent Ambo
85fa12da33 feat(aoc2019): Solve day 1 puzzles 2019-12-09 14:43:01 +00:00
Vincent Ambo
ded10fbd83 feat(tools): Add auto-dispatch for Advent of Code solutions 2019-12-09 12:27:07 +00:00
Vincent Ambo
514c48fdde feat(tools): Set up Advent of Code 2019 scaffolding 2019-12-09 11:44:21 +00:00
Vincent Ambo
46e5a23937 fix: Amend import paths to use whitelisted nixpkgs 2019-12-09 02:55:22 +00:00
Vincent Ambo
01bad09eed refactor: Introduce new layout with nixpkgs in third_party
This is not the final layout yet, but makes it so that my top-level
attribute set is no longer overlaid into nixpkgs itself.

This is useful for other people who are importing my monorepo.
2019-12-09 02:40:48 +00:00
Vincent Ambo
2b82f1b71a refactor: Reshuffle file structure for better code layout
This gets rid of the package called "server" and instead moves
everything into the project root, such that Go actually builds us a
binary called `nixery`.

This is the first step towards factoring out CLI-based functionality
for Nixery.
2019-11-27 14:12:38 +00:00
Vincent Ambo
df88da126a fix(builder): Ensure "solo-metapackages" do not break builds
The previous logic failed because single meta-packages such as
"nixery.dev/shell" would not end up removing the meta-package itself
from the list of packages passed to Nix, causing a build failure.

This was a regression introduced in 827468a.
2019-11-27 13:28:13 +00:00
Vincent Ambo
a924093d09 test(builder): Add test coverage for name->image conversion
Adds tests to cover that packages & metapackages are parsed into image
names correctly.
2019-11-27 13:28:13 +00:00
Vincent Ambo
f1655c5cbb fix(tools): Enable CI runs for tools.gotest 2019-11-22 16:29:43 +00:00
Vincent Ambo
1619f58d78 feat(tools): Add 'gotest' program to demonstrate pkgs.buildGo
This is a tiny program that does nothing but exists to demonstrate
pkgs.buildGo by building a program that depends on a local library as
well as a protobuf definition.
2019-11-22 16:28:22 +00:00
Vincent Ambo
45d63bce17 feat(nix): Filter projects that should be built by CI
Instead of specifying CI projects manually, this filters them to move
the CI configuration into the derivations `meta` attributes.
2019-11-15 23:25:41 +00:00
Vincent Ambo
c1c379848a chore(nix): Move files around to conform to new read-tree layout
Broadly speaking, the following things are included:

* there is now a uniform `args` struct that is passed to all
  derivations, package headers have been changed appropriately
* overrides are now loaded from a separate `override` folder just
  using read-tree.nix
* third-party packages have moved into the `third_party` attribute set
2019-11-15 15:26:08 +00:00
Vincent Ambo
104c930040 chore(build): Use significantly fewer layers for Nixery itself
Nixery itself is built with the buildLayeredImage system, which takes
some time to create large numbers of layers.

This adjusts the default number of image layers from 96 to 20.

Additionally Nixery's image is often loaded with `docker load -i`,
which ignores layer cache hits anyways.

Additionaly the CI build is configured to use only 1, which speeds up
CI runs.
2019-11-09 15:35:58 +00:00
Vincent Ambo
9a8abeff97 feat(build): Integration test on both CPU architectures 2019-11-09 15:35:58 +00:00
Vincent Ambo
1d6898a7cc feat(build): Include arm64 in build matrix 2019-11-09 15:35:58 +00:00
Vincent Ambo
145b7f4289 fix(build-image): Allow "cross-builds" of images for different arch
Imports the package set twice in the builder expression: Once
configured for the target system, once configured for the native
system.

This makes it possible to fetch the actual image contents for the
required architecture, but use local tools to assemble the symlink
layer and metadata.
2019-11-09 15:35:58 +00:00
Vincent Ambo
d7ccf35149 feat(builder): Support 'arm64' meta-package
Specifying this meta-package toggles support for ARM64 images, for
example:

    # Pull a default x86_64 image
    docker pull nixery.dev/hello

    # Pull an ARM64 image
    docker pull nixery.dev/arm64/hello
2019-11-09 15:35:58 +00:00
Vincent Ambo
3c2de4c037 refactor(builder): Parameterise CPU architecture to use for images
Adds the CPU architecture to the image configuration. This will make
it possible to let users toggle architecture via meta-packages.

Relates to #13
2019-11-09 15:35:58 +00:00
Vincent Ambo
7afbc912ce chore(build): Add nixery-popcount to top-level package set 2019-11-03 01:33:36 +00:00
Vincent Ambo
05b5b1718a feat(popcount): Cache seen narinfos on disk 2019-11-03 01:33:36 +00:00
Vincent Ambo
6a2fb092a7 chore: Add missing copyright headers to source files 2019-11-03 01:33:36 +00:00
Vincent Ambo
b03f7a1b4d feat(popcount): Add new narinfo-based popcount implementation
Adds an implementation of popcount that, instead of realising
derivations locally, just queries the cache's narinfo files.

The downside of this is that calculating popularity for arbitrary Nix
package sets is not possible with this implementation. The upside is
that calculating the popularity for an entire Nix channel can now be
done in ~10 seconds[0].

This fixes #65.

[0]: Assuming a /fast/ internet connection.
2019-11-03 01:33:36 +00:00
Vincent Ambo
2d4a3ea307 chore(server): Remove outdated TODO
Real-life experience has shown that the weighting of the metric
produced here is appropriate.
2019-10-29 19:16:24 +01:00
Vincent Ambo
904c3dade0 refactor(server): Handle non-error errors safely
This case should not be possible unless something manually constructs
a logrus entry with a non-error value in the log.ErrorKey field, but
it's better to be safe than sorry.
2019-10-29 19:16:24 +01:00
Vincent Ambo
3a7c964a22 chore(build): Configure build to use new GCS configuration options 2019-10-28 22:31:44 +01:00
Vincent Ambo
ab190256ab fix(server): Use correct scope for GCS tokens 2019-10-28 22:31:44 +01:00
Vincent Ambo
3611baf040 docs(under-the-hood): Update builder & storage backend information
Both of these no longer matched the reality of what was actually going
on in Nixery.
2019-10-28 22:31:44 +01:00
Vincent Ambo
b736f5580d docs: Add storage configuration options to README 2019-10-28 22:31:44 +01:00
Vincent Ambo
d8fba23365 fix(server): Thread request context to all relevant places
Previously background contexts where created where necessary (e.g. in
GCS interactions). Should I begin to use request timeouts or other
context-dependent things in the future, it's useful to have the actual
HTTP request context around.

This threads the request context through the application to all places
that need it.
2019-10-28 22:31:44 +01:00
Vincent Ambo
30e618b65b chore(server): Move cache miss log statement to debug level
This is very annoying otherwise.
2019-10-28 22:31:44 +01:00
Vincent Ambo
4332d38f4f fix(server): Correctly construct filesystem paths for layer serving 2019-10-28 22:31:44 +01:00
Vincent Ambo
b60a8d007b fix(server): Ensure paths exist when renaming in filesystem storage
The point at which files are moved happens to also (initially) be the
point where the `layers` directory is created. For this reason
renaming must ensure that all path components exist, which this commit
takes care of.
2019-10-28 22:31:44 +01:00
Vincent Ambo
c08aa52558 fix(server): Ensure error messages are correctly printed in logs
I assumed (incorrectly) that logrus would already take care of
surfacing error messages in human-readable form.
2019-10-28 22:31:44 +01:00
Vincent Ambo
790bce219c feat(server): Add filesystem storage backend config options
The filesystem storage backend can be enabled by setting
`NIXERY_STORAGE_BACKEND` to `filesystem` and `STORAGE_PATH` to a disk
location from which Nixery can serve files.
2019-10-28 22:31:44 +01:00
Vincent Ambo
167a0b3263 refactor(server): Pass HTTP request to storage.ServeLayer
The request object is required for some serving methods (e.g. the
filesystem one).
2019-10-28 22:31:44 +01:00
Vincent Ambo
e5bb2fc887 feat(server): Implement initial filesystem storage backend
This allows users to store and serve layers from a local filesystem
path.
2019-10-28 22:31:44 +01:00
Vincent Ambo
e8fd6b6734 refactor(server): Change setup to create new storage backends 2019-10-28 22:31:44 +01:00
Vincent Ambo
20e0ca53cb feat(server): Implement GCS storage backend with new interface
Logical implementation is mostly identical to the previous one, but
adhering to the new storage.Backend interface.
2019-10-28 22:31:44 +01:00
Vincent Ambo
f7d16c5d45 refactor(server): Introduce pluggable interface for storage backends
This abstracts over the functionality of Google Cloud Storage and
other potential underlying storage backends to make it possible to
replace these in Nixery.

The GCS backend is not yet reimplemented.
2019-10-28 22:31:44 +01:00
Vincent Ambo
ffe58d6cb5 refactor(build): Do not expose nixery-server attribute
In most cases this is not useful for users without the wrapper script,
so users should always build nixery-bin anyways.
2019-10-28 22:31:44 +01:00
Vincent Ambo
7b7d21205f docs: Update GCS signing key documentation
This key is now taken straight from the configured service account
key.
2019-10-27 13:58:04 +01:00
Vincent Ambo
3a5db4f9f1 refactor(server): Load GCS signing key from service account key
The JSON file generated for service account keys already contains the
required information for signing URLs in GCS, thus the environment
variables for toggling signing behaviour have been removed.

Signing is now enabled automatically in the presence of service
account credentials (i.e. `GOOGLE_APPLICATION_CREDENTIALS`).
2019-10-27 13:58:04 +01:00
Vincent Ambo
cca835ae37 fix(build): Only take the first matching hash for source hashing
Some Nix download mechanisms will add a second hash in the store path,
which had been added to the source hash output (breaking argument
interpolation).
2019-10-11 14:23:41 +01:00
Vincent Ambo
1853c74998 refactor(server): Only compress symlink forest layer once
Instead of compressing & decompressing again to get the underlying tar
hash, use a similar mechanism as for store path layers for the symlink
layer and only compress it once while uploading.
2019-10-11 12:37:00 +01:00
Vincent Ambo
e22ff5d176 fix(server): Use uncompressed tarball hashes in image config
Docker expects hashes of compressed tarballs in the manifest (as these
are used to fetch from the content-addressable layer store), but for
some reason it expects hashes in the configuration layer to be of
uncompressed tarballs.

To achieve this an additional SHA256 hash is calculcated while
creating the layer tarballs, but before passing them to the gzip
writer.

In the current constellation the symlink layer is first compressed and
then decompressed again to calculate its hash. This can be refactored
in a future change.
2019-10-11 12:37:00 +01:00
Vincent Ambo
0693e371d6 feat(server): Apply GZIP compression to all image layers
This fixes #62
2019-10-11 12:37:00 +01:00
Vincent Ambo
bf2718cebb chore(build): Use separate GCS bucket for CI runs
This has become an issue recently with changes such as GZIP
compression, where CI runs no longer work because they conflict with
the production bucket for the public instance.
2019-10-11 12:37:00 +01:00
Vincent Ambo
d7ffbbdea4 refactor(server): Use logrus convenience functions for logs
Makes use of the `.WithError` and `.WithField` convenience functions
in logrus to simplify log statement construction.

This has the added benefit of making it easier to correctly log
errors.
2019-10-06 23:05:23 +01:00
Vincent Ambo
c1020754a2 fix(build-image): Import 'match' from builtins 2019-10-06 23:05:23 +01:00
Vincent Ambo
6f148f789f refactor(server): Convert existing log entries to structured format
This rewrites all existing log statements into the structured logrus
format. For consistency, all errors are always logged separately from
the primary message in a field called `error`.

Only the "info", "error" and "warn" severities are used.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f77c93b6ae feat(server): Add log level to severity mapping
The output format now writes a `severity` field that follows that
format that should be recognised by Stackdriver Logging.
2019-10-06 23:05:23 +01:00
Vincent Ambo
6912658c72 feat(server): Use hash of Nixery source as version
Uses a hash of Nixery's sources as the version displayed when Nixery
launches or logs an error. This makes it possible to distinguish
between errors logged from different versions.

The source hashes should be reproducible between different checkouts
of the same source tree.
2019-10-06 23:05:23 +01:00
Vincent Ambo
95abb1bcde feat(server): Initial Stackdriver-compatible log formatter
This formatter has basic support for the Stackdriver Error Reporting
format, but several things are still lacking:

* the service version (preferably git commit?) needs to be included in
  the server somehow
* log streams should be split between stdout/stderr as that is how
  AppEngine (and several other GCP services?) seemingly differentiate
  between info/error logs
2019-10-06 23:05:23 +01:00
Vincent Ambo
0642f7044d fix(server): Amend package path for Go tooling compatibility
With these changes it is possible to keep Nixery in $GOPATH and build
the server in there, while still having things work correctly via Nix.
2019-10-06 23:05:23 +01:00
Vincent Ambo
f4bf3518f6 refactor(server): Replace log calls with logrus
This introduces a structured logging library that can be used (next
step) to attach additional metadata to log entries.
2019-10-06 23:05:23 +01:00
Vincent Ambo
d9b329ef59 refactor(server): Always include 'cacert' & 'iana-etc'
These two packages almost always end up being required by programs,
but people don't necessarily consider them.

They will now always be added and their popularity is artificially
inflated to ensure they end up at the top of the layer list.
2019-10-03 22:50:02 +01:00
Vincent Ambo
9bb6d0ae25 fix(server): Ensure build cache objects are written to GCS
Cache writes might not be flushed without this call.
2019-10-03 22:50:02 +01:00
Vincent Ambo
48a5ecda97 feat(server): Order layers in image manifest based on merge rating
Image layers in manifests are now sorted in a stable (descending)
order based on their merge rating, meaning that layers more likely to
be shared between images come first.

The reason for this change is Docker's handling of image layers on
overlayfs2: Images are condensed into a single representation on disk
after downloading.

Due to this Docker will constantly redownload all layers that are
applied in a different order in different images (layer order matters
in imperatively created images), based on something it calls the
'ChainID'.

Sorting the layers this way raises the likelihood of a long chain of
matching layers at the beginning of an image.

This relates to #39.
2019-10-03 22:50:02 +01:00
Vincent Ambo
0d820423e9 chore(build-image): Remove nixery-build-layers
This functionality has been rolled into the server component and is no
longer required.
2019-10-03 22:29:50 +01:00
Vincent Ambo
6b06fe27be feat(server): Implement creation of layer tarballs in the server
This will create, upload and hash the layer tarballs in one disk read.
2019-10-03 22:29:50 +01:00
Vincent Ambo
1124b8c236 fix(server): Do not invoke layer build if no layers are missing
This previously invoked a Nix derivation that spent a few seconds on
making an empty object in JSON ...
2019-10-03 13:21:04 +01:00
Vincent Ambo
feba42e409 feat(server): Fetch popularity data on launch
The last missing puzzle piece for #50!
2019-10-03 13:21:04 +01:00
Vincent Ambo
43a642435b feat(server): Reimplement local manifest cache backed by files
Implements a local manifest cache that uses the temporary directory to
cache manifest builds.

This is necessary due to the size of manifests: Keeping them entirely
in-memory would quickly balloon the memory usage of Nixery, unless
some mechanism for cache eviction is implemented.
2019-10-03 13:21:04 +01:00
Vincent Ambo
313e5d08f1 refactor(builder): Streamline layer creation & reintroduce caching
The functions used for layer creation are now easier to follow and
have clear points at which the layer cache is checked and populated.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
53906024ff refactor: Remove remaining MD5-hash mentions and computations 2019-10-03 13:21:04 +01:00
Vincent Ambo
f6b40ed6c7 refactor(server): Cache manifest entries for layer builds
MD5 hash checking is no longer performed by Nixery (it does not seem
to be necessary), hence the layer cache now only keeps the SHA256 hash
and size in the form of the manifest entry.

This makes it possible to restructure the builder code to perform
cache-fetching and cache-populating for layers in the same place.
2019-10-03 13:21:04 +01:00
Vincent Ambo
355fe3f5ec feat(server): Reintroduce manifest caching to GCS
The new builder now caches and reads cached manifests to/from GCS. The
in-memory cache is disabled, as manifests are no longer written to
local file and the caching of file paths does not work (unless we
reintroduce reading/writing from temp files as part of the local
cache).
2019-10-03 13:21:04 +01:00
Vincent Ambo
1308a6e1fd refactor(server): Clean up cache implementation
A couple of minor fixes and improvements to the cache implementation.
2019-10-03 13:21:04 +01:00
Vincent Ambo
0698d7f2aa chore(server): Remove "layer seen" cache
This cache is no longer required as it is implicit because the layer
cache (mapping store path hashes to layer hashes) implies that a layer
has been seen.
2019-10-03 13:21:04 +01:00
Vincent Ambo
64fca61ea1 fix(server): Upload symlink layer created by first Nix build
This layer is needed in addition to those that are built in the second
Nix build.
2019-10-03 13:21:04 +01:00
Vincent Ambo
f4f2909573 fix(server): Specify correct authentication scope for GCS
When retrieving tokens for service service accounts, some methods of
retrieval require a scope to be specified.
2019-10-03 13:21:04 +01:00
Vincent Ambo
aa02ae1421 feat(server): Implement new build process core
Implements the new build process to the point where it can actually
construct and serve image manifests.

It is worth noting that this build process works even if the Nix
sandbox is enabled!

It is also worth nothing that none of the caching functionality that
the new build process enables (such as per-layer build caching) is
actually in use yet, hence running Nixery at this commit is prone to
doing more work than previously.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
17adda0355 fix(server): Minor fixes to updated new builder code 2019-10-03 13:21:04 +01:00
Vincent Ambo
ef2623d168 fix(nix): Minor fixes to derivations for new build process 2019-10-03 13:21:04 +01:00
Vincent Ambo
3f40c0a2d2 feat(server): Implement package for creating image manifests
The new manifest package creates image manifests and their
configuration. This previously happened in Nix, but is now part of the
server's workload.

This relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
87e196757b feat(server): Reimplement creation & uploading of layers
The new build process can now call out to Nix to create layers and
upload them to the bucket if necessary.

The layer cache is populated, but not yet used.
2019-10-03 13:21:04 +01:00
Vincent Ambo
61269175c0 refactor(server): Introduce a state type to carry runtime state
The state type contains things such as the bucket handle and Nixery's
configuration which need to be passed around in the builder.

This is only added for convenience.
2019-10-03 13:21:04 +01:00
Vincent Ambo
6e2b84f475 feat(server): Add cache for layer builds in GCS & local cache
This cache is going to be used for looking up whether a layer build
has taken place already (based on a hash of the layer contents).

See the caching section in the updated documentation for details.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
6262dec8aa feat(nix): Add derivation to create layer tars from a store path set
This introduces a new Nix derivation that, given an attribute set of
layer hashes mapped to store paths, will create a layer tarball for
each of the store paths.

This is going to be used by the builder to create layers that are not
present in the cache.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
2c8ef634f6 docs(caching): Add information about Nixery's caching strategies 2019-10-03 13:21:04 +01:00
Vincent Ambo
e60805c9b2 feat(server): Introduce function to hash contents of a layer
This creates a cache key which can be used to check if a layer has
already been built.
2019-10-03 13:21:04 +01:00
Vincent Ambo
9c3c622403 refactor(server): Expose layer grouping logic via a function
Refactors the layer grouping package (which previously compiled to a
separate binary) to expose the layer grouping logic via a function
instead.

This is the next step towards creating layers inside of the server
component instead of in Nix.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
8c79d085ae chore(server): Import layer grouping logic into server component 2019-10-03 13:21:04 +01:00
Vincent Ambo
0898d8a961 chore(build-image): Simplify wrapper build & remove layer grouping
Simplifies the wrapper script used to invoke Nix builds from Nixery to
just contain the essentials, since the layer grouping logic is moving
into the server itself.
2019-10-03 13:21:04 +01:00
Vincent Ambo
712b38cbbc refactor(build-image): Do not assemble image layers in Nix
This is the first step towards a more granular build process where
some of the build responsibility moves into the server component.

Rather than assembling all layers inside of Nix, it will only create
the symlink forest and return information about the runtime paths
required by the image.

The server is then responsible for grouping these paths into layers,
and assembling the layers themselves.

Relates to #50.
2019-10-03 13:21:04 +01:00
Vincent Ambo
ad9b3eb262 refactor(build): Add group-layers to top-level Nix derivations
This makes CI build the group-layers tool (and cache it to Cachix!)
2019-09-22 18:04:40 +01:00
Vincent Ambo
7b987530d1 refactor(build-image): Minor tweak to layer construction script 2019-09-22 18:04:40 +01:00
Vincent Ambo
21a17b33f4 fix(build): Ensure launch script compatibility with other runtimes
Fixes two launch script compatibility issues with other container
runtimes (such as gvisor):

* don't fail if /tmp already exists
* don't fail if the environment becomes unset
2019-09-21 15:25:28 +01:00
Vincent Ambo
0000b956bb feat(server): Log Nix output live during the builds
Instead of dumping all Nix output as one at the end of the build
process, stream it live as the lines come in.

This is a lot more useful for debugging stuff like where manifest
retrievals get stuck.
2019-09-21 15:25:28 +01:00
Vincent Ambo
c391a7b7f8 fix(build-image): Use absolute paths in tarballs 2019-09-21 15:25:28 +01:00
Vincent Ambo
da6fd1d79e fix(build): Ensure nixery-build-image is on Nixery's PATH
This is useful when running Nixery locally.
2019-09-21 12:44:40 +01:00
Vincent Ambo
f0b69638e1 chore(build): Bump nixpkgs version used in Travis
This version matches the updated popularity URL.
2019-09-21 12:44:40 +01:00
Vincent Ambo
64f74abc4d feat: Add configuration option for popularity data URL 2019-09-21 12:44:40 +01:00
Vincent Ambo
e4d03fdb17 chore(docs): Remove mdbook override
The change has been upstreamed in Nixpkgs.
2019-09-21 12:44:40 +01:00
Vincent Ambo
5a002fe067 refactor(builder): Calculate image cache key only once 2019-09-10 11:32:37 +01:00
Vincent Ambo
4a58b0ab4d feat(server): Cache built manifests to the GCS bucket
Caches manifests under `manifests/$cacheKey` in the GCS bucket and
introduces two-tiered retrieval of manifests from the caches (local
first, bucket second).

There is some cleanup to be done in this code, but the initial version
works.
2019-09-10 11:32:37 +01:00
Vincent Ambo
051eb77b3d refactor(server): Use package source specific cache keys
Use the PackageSource.CacheKey function introduced in the previous
commit to determine the key at which a manifest should be cached in
the local cache.

Due to this change, manifests for moving target sources are no longer
cached and the recency threshold logic has been removed.
2019-09-10 11:32:37 +01:00
Vincent Ambo
980f5e2187 refactor(server): Move package source management logic to server
Introduces three new types representing each of the possible package
sources and moves the logic for specifying the package source to the
server.

Concrete changes:

* Determining whether a specified git reference is a commit vs. a
  branch/tag is now done in the server, and is done more precisely by
  using a regular expression.

* Package sources now have a new `CacheKey` function which can be used
  to retrieve a key under which a build manifest can be cached *if*
  the package source is not a moving target (i.e. a full git commit
  hash of either nixpkgs or a private repository).

  This function is not yet used.

* Users *must* now specify a package source, Nixery no longer defaults
  to anything and will fail to launch if no source is configured.
2019-09-10 11:32:37 +01:00
Vincent Ambo
bcd7710be5 feat(tools): Introduce pass-compatible wrapper using Cloud KMS
Adds a shell script that supports a subset of the 'pass' interface for
compatibility with kontemplate, and wraps kontemplate in a script that
places this version on the PATH.

This makes it possible to use Cloud KMS encrypted secrets with kontemplate.
2019-09-03 16:12:30 +01:00
Vincent Ambo
496a4ab847 docs: Add information about NIX_TIMEOUT variable 2019-09-02 23:44:57 +01:00
Vincent Ambo
32b9b5099e feat(server): Add configuration option for Nix build timeouts
Adds a NIX_TIMEOUT environment variable which can be set to a number
of seconds that is the maximum allowed time each Nix builder can run.

By default this is set to 60 seconds, which should be plenty for most
use-cases as Nixery is not expected to be performing builds of
uncached binaries in most production cases.

Currently the errors Nix throws on a build timeout are not separated
from other types of errors, meaning that users will see a generic 500
server error in case of a timeout.

This fixes #47
2019-09-02 23:44:57 +01:00
Vincent Ambo
f2e0f3ee27 chore(third_party): Remove git-appraise
Not actually in use here ...
2019-09-02 20:01:25 +01:00
Vincent Ambo
640b497950 feat(tools): Add stern, a k8s log watcher 2019-09-02 18:18:28 +01:00
Vincent Ambo
ce8635833b refactor(build-image): Remove implicit import of entire package set
Explicitly refer to where things come from, and also don't import
dockerTools as it is no longer used for anything.
2019-09-02 00:08:11 +01:00
Vincent Ambo
92270fcbe4 refactor(build-image): Simplify customisation layer builder
Moves the relevant parts of the customisation layer construction from
dockerTools.mkCustomisationLayer into the Nixery code base.

The version in dockerTools builds additional files (including via
hashing of potentially large files) which are not required when
serving an image over the registry protocol.
2019-09-02 00:08:11 +01:00
Vincent Ambo
6450347bf1 chore(tools): Remove ormolu from tools
The Ormolu derivation is too large to be instantiated on-demand. I've
resorted to installing this tool into my profile instead.
2019-08-25 20:17:03 +01:00
Vincent Ambo
4f88851971 fix(blog_cli): Quote JSON output for post dates 2019-08-23 11:24:03 +01:00
Vincent Ambo
57059fec2b feat(blog_cli): Add flag to set post date and serialise as date only 2019-08-23 10:57:15 +01:00
Vincent Ambo
306e12787a chore(build): Add iana-etc to Nixery's own image
This package is used by a variety of programs that users may want to
embed into Nixery in addition, for example cachix, but those packages
don't refer to it explicitly.
2019-08-21 10:35:32 +01:00
Vincent Ambo
bb5427a47a chore(docs): Update embedded nix-1p version
The new version of the document has syntactic fixes that render pipes
in code blocks in tables correctly across dialects.

Fixes #44
2019-08-21 10:35:32 +01:00
Vincent Ambo
b856648127 feat(tools): Introduce ormolu for formatting Haskell code 2019-08-20 00:26:54 +01:00
Vincent Ambo
1d5b53abf8 chore(tools/blog_cli): Shorten metadata key names
Every character counts because I want to keep metadata below 255
bytes.
2019-08-19 19:11:32 +01:00
Vincent Ambo
d5710db900 refactor(tools/blog_cli): More sensible chunk encoding as records
Instead of many rrdatas in one record, make many records!
2019-08-19 17:51:29 +01:00
Vincent Ambo
2a16740445 feat(tools): Add a horrifying tool to write blog posts into DNS 2019-08-19 16:59:55 +01:00
Vincent Ambo
daa6196c2a fix(build): Force nix-env to use NIX_PATH
Thanks to clever!
2019-08-19 01:56:17 +01:00
Vincent Ambo
ccf6a95f94 chore(build): Pin nixpkgs to a specific commit
This is the same commit for which Nixery has popularity data, but that
isn't particularly relevant.
2019-08-19 01:56:17 +01:00
Vincent Ambo
e7d7f73f7d feat(build): Add 'extraPackages' parameter
This makes it possible to inject additional programs (e.g. Cachix)
into a Nixery container.
2019-08-19 01:56:17 +01:00
Vincent Ambo
ffae282eac fix(docs): Correct link to layering strategy 2019-08-19 01:56:17 +01:00
Vincent Ambo
745b7ce0b8 fix(build): Ensure root user is known inside of container
This is required by git in cases where Nixery is configured with a
custom git repository.

I've also added a shell back into the image to make debugging a
running Nixery easier. It turns out some of the dependencies already
pull in bash anyways, so this is just surfacing it to $PATH.
2019-08-17 10:42:13 +01:00
Vincent Ambo
9a95c4124f fix(server): Sort requested packages in image name & spec
Before this change, Nixery would pass on the image name unmodified to
Nix which would lead it to cache-bust the manifest and configuration
layers for images that are content-identical but have different
package ordering.

This fixes #38.
2019-08-17 10:31:41 +01:00
Vincent Ambo
0ee239874b docs(README): Update links to layering strategy 2019-08-17 10:10:41 +01:00
Florian Klink
3b65fc8c72 feat(server): add iana-etc and cacert to the shell convenience package
These probably should be part of every container image by default, but
adding it to the "shell" convenience name probably is our best bet for
now.
2019-08-16 22:47:22 +01:00
Vincent Ambo
27036e18f5 feat(tools): Add kontemplate 2019-08-16 16:50:50 +01:00
Vincent Ambo
b652ea2f03 feat(tools): Add Terraform 0.12 to local tools
This is pre-configured with the GCP provider.
2019-08-15 16:28:42 +01:00
Vincent Ambo
38493b83ea chore(tools): Remove bazel & ghc 2019-08-15 16:15:59 +01:00
Vincent Ambo
501e6ded5f fix(build): Ensure GCS signing is used in CI 2019-08-15 16:04:34 +01:00
Vincent Ambo
3f232e0170 docs: Add asciinema demo to README & book 2019-08-15 16:04:34 +01:00
Vincent Ambo
0ec369d76c docs(book): Update information on new layering strategy 2019-08-15 16:04:34 +01:00
Vincent Ambo
ca1ffb397d feat(build): Add an integration test that runs on Travis
This test, after performing the usual Nixery build, loads the built
image into Docker, runs it, pulls an image from Nixery and runs that
image.

To make this work, there is some configuration on the Travis side.
Most importantly, the following environment variables have special
values:

* `GOOGLE_KEY`: This is set to a base64-encoded service account key to
  be used in the test.
* `GCS_SIGNING_PEM`: This is set to a base64-encoded signing key (in
  PEM) that is used for signing URLs.

Both of these are available to all branches in the Nixery repository.
2019-08-15 00:26:57 +01:00
Vincent Ambo
85b9c30749 chore(server): Add 'go vet' to build process 2019-08-14 20:18:41 +01:00
Vincent Ambo
36d50d1f19 fix(server): Print correct project name during startup
They grow up so fast :')
2019-08-14 20:18:41 +01:00
Vincent Ambo
cf227c153f feat(builder): Implement build cache for manifests & layers
Implements a cache that keeps track of:

a) Manifests that have already been built (for up to 6 hours)
b) Layers that have already been seen (and uploaded to GCS)

This significantly speeds up response times for images that are full
or partial matches with previous images served by an instance.
2019-08-14 20:18:41 +01:00
Vincent Ambo
58380e3313 refactor(server): Extract build logic into separate module
This module is going to get more complex as the implementation of #32
progresses.
2019-08-14 20:18:41 +01:00
Vincent Ambo
d9168e3e4d refactor(build-image): Extract package set loading into helper
Some upcoming changes might require the Nix build to be split into
multiple separate nix-build invocations of different expressions, thus
splitting this out is useful.

It also fixes an issue where `build-image/default.nix` might be called
in an environment where no Nix channels are configured.
2019-08-14 00:02:04 +01:00
Vincent Ambo
3939722063 style: Apply nixfmt to trivial Nix files
ALl the ones except for build-image.nix are considered trivial. On the
latter, nixfmt makes some useful changes but by-and-large it is not
ready for that code yet.
2019-08-14 00:02:04 +01:00
Vincent Ambo
6285cd8dbf feat(build-image): Use new image layering algorithm for images
Removes usage of the old layering algorithm and replaces it with the
new one.

Apart from the new layer layout this means that each layer is now
built in a separate derivation, which hopefully leads to better
cacheability.
2019-08-14 00:02:04 +01:00
Vincent Ambo
7214d0aa4f feat(build-image): Introduce a terrifying hack to build group-layers
The issue is described in detail in a comment in
`build-image/default.nix`, please read it.
2019-08-14 00:02:04 +01:00
Vincent Ambo
f60f702274 feat: Add shell.nix for running a local Nixery 2019-08-14 00:02:04 +01:00
Vincent Ambo
6035bf36eb feat(popcount): Clean up popularity counting script
Adds the script used to generate the popularity information for all of
nixpkgs.

The README lists the (currently somewhat rough) usage instructions.
2019-08-14 00:02:04 +01:00
Vincent Ambo
6d718bf271 refactor(server): Use wrapper script to avoid path dependency
Instead of requiring the server component to be made aware of the
location of the Nix builder via environment variables, this commit
introduces a wrapper script for the builder that can simply exist on
the builders $PATH.

This is one step towards a slightly nicer out-of-the-box experience
when using `nix-build -A nixery-bin`.
2019-08-14 00:02:04 +01:00
Vincent Ambo
819b460278 chore(docs): Move mdBook derivation to docs/default.nix 2019-08-14 00:02:04 +01:00
Vincent Ambo
1fa93fe6f6 refactor: Move registry server to a subfolder 2019-08-14 00:02:04 +01:00
Vincent Ambo
d699f7f91c chore(build): Update Go dependencies & add gonum 2019-08-14 00:02:04 +01:00
Vincent Ambo
56a426952c feat(group-layers): Finish layering algorithm implementation
This commit adds the actual logic for extracting layer groupings and
merging them until the layer budget is satisfied.

The implementation conforms to the design doc as of the time of this
commit.
2019-08-14 00:02:04 +01:00
Vincent Ambo
590ce994bb feat(group-layers): Add initial popcount scripts
This script generates an entry in a text file for each time a
derivation is referred to by another in nixpkgs.

For initial testing, this output can be turned into group-layers
compatible JSON with this *trivial* invocation:

cat output | awk '{ print "{\"" $2 "\":" $1 "}"}' | jq -s '. | add | with_entries(.key |= sub("/nix/store/[a-z0-9]+-";""))' > test-data.json
2019-08-14 00:02:04 +01:00
Vincent Ambo
92078527db feat(group-layers): Add preliminary size & popularity considerations
As described in the design document, this adds considerations for
closure size and popularity. All closures meeting a certain threshold
for either value will have an extra edge from the image root to
themselves inserted in the graph, which will cause them to be
considered for inclusion in a separate layer.

This is preliminary because popularity is considered as a boolean
toggle (the input I generated only contains the top ~200 most popular
packages), but it should be using either absolute popularity values or
percentiles (needs some experimentation).
2019-08-14 00:02:04 +01:00
Vincent Ambo
ce31598f42 feat(group-layers): Implement first half of new layering strategy
The strategy is described in-depth in the comment at the top of the
implementation file, as well as in the design document:

https://storage.googleapis.com/nixdoc/nixery-layers.html
2019-08-14 00:02:04 +01:00
Vincent Ambo
3e385dc379 docs: Update embedded nix-1p
The new commit has an operator table, which is nice to have!
2019-08-08 21:15:50 +01:00
Vincent Ambo
c727b3ca9e chore(nix): Increase maximum number of layers to 96
This uses a significantly larger percentage of the total available
layers (125) than before, which means that cache hits for layers
become more likely between images.
2019-08-08 21:15:50 +01:00
Vincent Ambo
993fda3377 fix(go): Fix breakage in unsigned URLs
This affected the public instance which is still running without URL
signing. Should add some monitoring!
2019-08-08 18:58:23 +01:00
Vincent Ambo
12a853fab7 docs: Minor fixes to README after new website release 2019-08-05 01:26:19 +01:00
Vincent Ambo
d87662b7b5 docs: Add a section on Nix itself 2019-08-05 00:32:53 +01:00
Vincent Ambo
6293d69fd9 docs: Add a section on running your own Nixery 2019-08-05 00:32:53 +01:00
Vincent Ambo
a3f6278913 docs: Add an "under-the-hood" page explaining the build process
This page describes the various steps that Nixery goes through when
"procuring" an image.

The intention is to give users some more visibility into what is going
on and to make it clear that this is not just an image storage
service.
2019-08-05 00:32:53 +01:00
Vincent Ambo
2bef0ba240 feat(build): Build Nixery book and embed it into Nixery image
Executes the previously added mdBook on the previously added book
source to yield a directory that can be served by Nixery on its index
page.

This is one of those 'I <3 Nix' things due to how easy it is to do.
2019-08-05 00:32:53 +01:00
Vincent Ambo
85e8d760fc feat(build): Add mdBook 0.3.1 to build environment
Upstream nixpkgs currently only has an older versin of mdBook. Until
that changes, we keep a different version in here.
2019-08-05 00:32:53 +01:00
Vincent Ambo
7c41a7a872 docs: Replace static page with mdBook site
Uses mdBook[1] to generate a documentation overview page instead of
the previous HTML site.

This makes it possible to add more elaborate documentation without
having to deal with finicky markup.

[1]: https://github.com/rust-lang-nursery/mdBook
2019-08-05 00:32:53 +01:00
Vincent Ambo
099c99b7ad feat(build): Configure Cachix for build caching in CI
The CI setup is configured with an appropriate key to enable pushes to
the nixery.cachix.org binary cache.
2019-08-04 01:41:35 +01:00
Vincent Ambo
a0d7d693d3 feat(build): Support additional pre-launch commands in image
This makes it possible for users to hook basically arbitrary things
into the Nixery container image.
2019-08-04 01:41:35 +01:00
Vincent Ambo
20103640fa fix(nix): Support retrieving differently cased top-level attributes
As described in issue #14, the registry API does not allow image names
with uppercase-characters in them.

However, the Nix package set has several top-level keys with uppercase
characters in them which could previously not be retrieved using
Nixery.

This change implements a method for retrieving those keys, but it is
explicitly only working for the top-level package set as nested
sets (such as `haskellPackages`) often contain packages that differ in
case only.
2019-08-03 23:11:40 +01:00
Vincent Ambo
aa260af1ff docs: Add GCS signing envvars to README 2019-08-03 01:31:51 +01:00
Vincent Ambo
07ef06dcfa feat(go): Support signed GCS URLs with static keys
Google Cloud Storage supports granting access to protected objects via
time-restricted URLs that are cryptographically signed.

This makes it possible to store private data in buckets and to
distribute it to eligible clients without having to make those clients
aware of GCS authentication methods.

Nixery now uses this feature to sign URLs for GCS buckets when
returning layer URLs to clients on image pulls. This means that a
private Nixery instance can run a bucket with restricted access just
fine.

Under the hood Nixery uses a key provided via environment
variables to sign the URL with a 5 minute expiration time.

This can be set up by adding the following two environment variables:

* GCS_SIGNING_KEY: Path to the PEM file containing the signing key.
* GCS_SIGNING_ACCOUNT: Account ("e-mail" address) to use for signing.

If the variables are not set, the previous behaviour is not modified.
2019-08-03 01:31:51 +01:00
Vincent Ambo
3347c38ba7 fix(go): Registry API acknowledgement URI has a trailing slash
Previously the acknowledgement calls from Docker were receiving a
404 (which apparently doesn't bother it?!). This corrects the URL,
which meant that acknowledgement had to move inside of the
registryHandler.
2019-08-03 01:25:36 +01:00
Vincent Ambo
ecee1ec1b8 chore: Prevent accidental key leaks via gitignore 2019-08-03 01:25:36 +01:00
Vincent Ambo
62ade18b7b fix(static): Fix logo nitpick (smoothened λ edges) 2019-08-03 01:25:36 +01:00
Vincent Ambo
c84543a9b5 style(static): Fix favicon background colour 2019-08-03 01:25:36 +01:00
Vincent Ambo
a4c0d3e8d3 chore(go): Remove 'builder' metapackage
This metapackage isn't actually particularly useful (stdenv is rarely
what users want).
2019-08-03 01:25:36 +01:00
Vincent Ambo
da5df525c8 docs: Update all nixery.appspot.com references to nixery.dev
Shiny, new domain is much better and eliminates the TLS redirect issue
because there is a HSTS preload for the entire .dev TLD (which, by the
way, is awesome!)
2019-08-02 17:14:06 +01:00
Vincent Ambo
1f885e43b6 style(static): Update Nixery logo to a healthier version
This might not yet be the final version, but it's going in the right
direction.

Additionally the favicon has been reduced to just the coloured Nix
logo, because details are pretty much invisible at that size anyways.
2019-08-02 17:14:06 +01:00
Vincent Ambo
02dfff393a fix(build): coreutils are still required by launch script
Mea culpa!
2019-08-02 01:33:39 +01:00
Vincent Ambo
92f1758014 docs(static): Note that the demo instance is just a demo
People should not start depending on the demo instance. There have
been discussions around making a NixOS-official instance, but the
project needs to mature a little bit first.
2019-08-02 01:17:36 +01:00
Vincent Ambo
bf34bb327c fix(nix): Calculate MD5 sum of config layer correctly
The MD5 sum is used for verifying contents in the layer cache before
accidentally re-uploading, but the syntax of the hash invocation was
incorrect leading to a cache-bust on the manifest layer on every
single build (even for identical images).
2019-08-02 01:08:14 +01:00
Vincent Ambo
3d0596596a feat(go): Return error responses in registry format
The registry specifies a format for how errors should be returned and
this commit implements it:

https://docs.docker.com/registry/spec/api/#errors
2019-08-02 01:08:14 +01:00
Vincent Ambo
119af77b43 feat(go): Return errors with correct status codes to clients
Uses the structured errors feature introduced in the Nix code to
return more sensible errors to clients. For now this is quite limited,
but already a lot better than before:

* packages that could not be found result in 404s
* all other errors result in 500s

This way the registry clients will not attempt to interpret the
returned garbage data/empty response as something useful.
2019-08-02 01:08:14 +01:00
Vincent Ambo
2f1bc55597 fix(go): Return response code 500 if Nix builds fail 2019-08-02 01:08:14 +01:00
Vincent Ambo
3070d88051 feat(nix): Return structured errors if packages are not found
Changes the return format of Nixery's build procedure to return a JSON
structure that can indicate which errors have occured.

The server can use this information to send appropriate status codes
back to clients.
2019-08-02 01:08:14 +01:00
Vincent Ambo
ec8e9eed5d docs(README): Revamp with updated information on package sources
Adds documentation for configuration options and supported features.
2019-07-31 15:28:36 +01:00
Vincent Ambo
3bc04530a7 feat(go): Add environment configuration for package set sources
Adds environment variables with which users can configure the package
set source to use. Not setting a source lets Nix default to a recent
NixOS channel (currently nixos-19.03).
2019-07-31 15:28:36 +01:00
Vincent Ambo
2db92243e7 feat(nix): Support package set imports from different sources
This extends the package set import mechanism in
build-registry-image.nix with several different options:

1. Importing a nixpkgs channel from Github (the default, pinned to
   nixos-19.03)

2. Importing a custom Nix git repository. This uses builtins.fetchGit
   and can thus rely on git/SSH configuration in the environment (such
   as keys)

3. Importing a local filesystem path

As long as the repository pointed at is either a checkout of nixpkgs,
or nixpkgs overlaid with custom packages this will work.

A special syntax has been defined for how these three options are
passed in, but users should not need to concern themselves with it as
it will be taken care of by the server component.

This relates to #3.
2019-07-31 15:28:36 +01:00
Vincent Ambo
a83701a14b feat(build): Add dependencies for custom repo clones
Adds git & SSH as part of the Nixery image, which are required to use
Nix's builtins.fetchGit.

The dependency on interactive tools is dropped, as it was only
required during development when debugging the image building process
itself.
2019-07-31 15:28:36 +01:00
Vincent Ambo
2e4b1f85ee fix(nix): Add empty image config to allow k8s usage
Introduce an empty runtime configuration object in each built layer.

This is required because Kubernetes expects the configuration to be
present (even if it's just empty values).

Providing an empty configuration will make Docker's API return a full
configuration struct with default (i.e. empty) values rather than
`null`, which works for Kubernetes.

This fixes issue #1. See the issue for additional details.
2019-07-31 00:02:27 +01:00