tvl-depot/nix/buildLisp

buildLisp.nix

This is a build system for Common Lisp, written in Nix.

It aims to offer an alternative to ASDF for users who live in a Nix-based ecosystem. This offers several advantages over ASDF:

• Simpler (almost logic-less) package definitions
• Easy linking of native dependencies (from Nix)
• Composability with Nix tooling for other languages
• Effective, per-system caching strategies
• Easy overriding of dependencies and whatnot
• Convenient support for multiple Common Lisp implementations
• ... and more!

The project is still in its early stages and some important restrictions should be highlighted:

• Extending buildLisp with support for a custom implementation currently requires some knowledge of internals and may not be considered stable yet.
• Parallel compilation is not possible: Since buildLisp doesn't encode dependencies between components (i. e. source files) like ASDF, it must compile source files in sequence to avoid errors due to undefined symbols.

Usage

buildLisp exposes four different functions:

• buildLisp.library: Builds a collection of Lisp files into a library.

  parameter	type	use	required?
  name	string	Name of the library	yes
  srcs	list<path>	List of paths to source files	yes
  deps	list<drv>	List of dependencies	no
  native	list<drv>	List of native dependencies	no
  test	see "Tests"	Specification for test suite	no
  implementation	see "Implementations"	Common Lisp implementation to use	no

  The output of invoking this is a directory containing a FASL file that is the concatenated result of all compiled sources.

• buildLisp.program: Builds an executable program out of Lisp files.

  parameter	type	use	required?
  name	string	Name of the program	yes
  srcs	list<path>	List of paths to source files	yes
  deps	list<drv>	List of dependencies	no
  native	list<drv>	List of native dependencies	no
  main	string	Entrypoint function	no
  test	see "Tests"	Specification for test suite	no
  implementation	see "Implementations"	Common Lisp implementation to use	no

  The main parameter should be the name of a function and defaults to ${name}:main (i.e. the exported main function of the package named after the program).

  The output of invoking this is a directory containing a bin/${name}.

• buildLisp.bundled: Creates a virtual dependency on a built-in library.

  Certain libraries ship with Lisp implementations, for example UIOP/ASDF are commonly included but many implementations also ship internals (such as SBCLs various sb-* libraries).

  This function takes a single string argument that is the name of a built-in library and returns a "package" that simply requires this library.

Tests

Both buildLisp.library and buildLisp.program take an optional argument tests, which has the following supported fields:

parameter	type	use	required?
name	string	Name of the test suite	no
expression	string	Lisp expression to run tests	yes
srcs	list<path>	List of paths to source files	no
native	list<drv>	List of native dependencies	no

the expression parameter should be a Lisp expression and will be evaluated after loading all sources and dependencies (including library/program dependencies). It must return a non-NIL value if the test suite has passed.

Example

Using buildLisp could look like this:

{ buildLisp, lispPkgs }:

let libExample = buildLisp.library {
    name = "lib-example";
    srcs = [ ./lib.lisp ];

    deps = with lispPkgs; [
      (buildLisp.bundled "sb-posix")
      iterate
      cl-ppcre
    ];
};
in buildLisp.program {
    name = "example";
    deps = [ libExample ];
    srcs = [ ./main.lisp ];
    tests = {
      deps = [ lispPkgs.fiveam ];
      srcs = [ ./tests.lisp ];
      expression = "(fiveam:run!)";
    };
}

Development REPLs

buildLisp builds loadable variants of both program and library derivations (usually FASL files). Therefore it can provide a convenient way to obtain an instance of any implementation preloaded with buildLisp-derivations. This is especially useful to use as a host for Sly or SLIME.

• buildLisp.sbcl.lispWith, buildLisp.ccl.lispWith, ...: Creates a wrapper script preloading a Lisp implementation with various dependencies.

  This function takes a single argument which is a list of Lisp libraries programs or programs. The desired Lisp implementation will load all given derivations and all their dependencies on startup.

  The shortcut buildLisp.sbclWith for buildLisp.sbcl.lispWith is also provided.

• repl passthru attribute: derivation.repl is provided as a shortcut for buildLisp.${implementationName}.lispWith [ derivation ]. derivation.ccl.repl, derivation.sbcl.repl etc. work as well, of course (see also "Implementations" section).

Implementations

Both buildLisp.library and buildLisp.program allow specifying a different Common Lisp implementation than the default one (which is SBCL). When an implementation is passed, buildLisp makes sure all dependencies are built with that implementation as well since build artifacts from different implementation will be incompatible with each other.

The argument taken by implementation is a special attribute set which describes how to do certain tasks for a given implementation, like building or loading a library. In case you want to use a custom implementation description, the precise structure needed is documented in buildLisp's source code for now. buildLisp also exposes the following already working implementation sets:

• buildLisp.sbcl: SBCL, our default implementation

• buildLisp.ccl: CCL, similar to SBCL, but with very good macOS support

• buildLisp.ecl: ECL setup to produce statically linked binaries and libraries. Note that its runtime library is LGPL, so extra conditions must be fulfilled when distributing binaries produced this way.

• Support for ABCL is planned.

For every of these “known” implementations, buildLisp will create a passthru attribute named like the implementation which points to a variant of the derivation built with said implementation. Say we have a derivation, myDrv, built using SBCL: While myDrv and myDrv.sbcl are built using SBCL, myDrv.ecl, myDrv.ccl etc. build the derivation and all its dependencies using ECL and CCL respectively.

This is useful to test portability of your derivation, but is also used internally to speed up the “normalization” of the dependency graph. Thus it is important to make sure that your custom implementation's name doesn't clash with one of the “known” ones.

Handling Implementation Specifics

When targeting multiple Common Lisp implementation, it is often necessary to handle differing interfaces for OS interaction or to make use of special implementation features. For this reason, buildLisp allows specifying dependencies and source files for specific implementations only. This can be utilized by having an attribute set in the list for the deps or srcs argument: buildLisp will pick the value of the attribute named like the used implementation or default and ignore the set completely if both are missing.

{ buildLisp, lispPkgs }:

buildLisp.library {
  name = "mylib";

  srcs = [
    # These are included always of course
    ./package.lisp
    ./portable-lib.lisp

    # Choose right impl-* file
    {
      sbcl = ./impl-sbcl.lisp;
      ccl = ./impl-ccl.lisp;
      ecl = ./impl-ecl.lisp;
    }

    # We can also use this to inject extra files
    { ecl = ./extra-ecl-optimizations.lisp; }
  ];

  deps = [
    # Use SBCL's special bundled package, flexi-streams otherwise
    {
      sbcl = buildLisp.bundled "sb-rotate-byte";
      default = lispPkgs.flexi-streams;
    }
  ];
}

Additionally a brokenOn parameter is accepted which takes a list of implementation names on which the derivation is not expected to work. This only influences meta.ci.targets which is read by depot's CI to check which variants (see "Implementations") of the derivation to build, so it may not be useful outside of depot.

Influencing the Lisp Runtime

Lisp implementations which create an executable by dumping an image usually parse a few implementation-specific command line options on executable startup that influence runtime settings related to things like GC. buildLisp generates a wrapper which makes sure that this never interferes with the argument parsing implemented in the actual application, but sometimes it is useful to run an executable with special settings. To allow this, the content of NIX_BUILDLISP_LISP_ARGS is passed to the lisp implementation.

For example, you can make the underlying SBCL print its version for any executable built with buildLisp (and SBCL) like this:

$ env NIX_BUILDLISP_LISP_ARGS="--version" ./result/bin/🕰️
SBCL 2.1.2.nixos

In practice you'd probably want to specify options like --dynamic-space-size or --tls-limit (try passing --help for a full list). Naturally, these options are completely different for different implementations.