builtins.genericClosure is a quite powerful (and undocumented) Nix
primop: It repeatedly applies a function to values it produces and
collects them into a list. Additionally individual results can be
identified via a key attribute.
Since genericClosure only ever creates a single list value internally,
we can eliminate a huge performance bottleneck when building a list in a
recursive algorithm: list concatenation. Because Nix needs to copy the
entire chunk of memory used internally to represent the list, building
big lists one element at a time grinds Nix to a halt.
After rewriting decode using genericClosure decoding the LaTeX source
of my 20 page term paper now takes 2s instead of 14min.
Change-Id: I33847e4e7dd95d7f4d78ac83eb0d74a9867bfe80
I think it’s solid enough to use in a wider context.
Change-Id: If53e8bbb6b90fa88d73fb42730db470e822ea182
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3055
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
This should ease migrating to a distinction between depot.third_party
and pkgs (as in nixpkgs) in the future.
Ref cl/2910, b/108.
Change-Id: I53a854071fddd7c0d0526cc4c5b16998202082c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2913
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
users.sterni.nix.utf8 implements UTF-8 decoding in pure nix. We
implement the decoding as a simple state machine which is fed one byte
at a time. Decoding whole strings is possible by subsequently calling
step. This is done in decode which uses builtins.foldl' to get around
recursion restrictions and a neat trick using builtins.deepSeq puck
showed me limiting the size of the thunks in a foldl' (which can also
cause a stack overflow).
This makes decoding arbitrarily large UTF-8 files into codepoints using
nix theoretically possible, but it is not really practical: Decoding a
36KB LaTeX file I had lying around takes ~160s on my laptop.
Change-Id: Iab8c973dac89074ec280b4880a7408e0b3d19bc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2590
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
