Commit graph

38 commits

Author SHA1 Message Date
sterni
62fa36c9c2 fix(nix/buildLisp): preserve argv[0] in buildLisp.program
I *thought* I was being clever with the (cdr (member …)) call, but
somehow completely forgot that *posix-argv* and
*command-line-argument-list* are equivalent to argv, so they also
contain the program name as the first element. Dropping that made
argument parsing completely break down, so let's revert back to the
older solution which works quite well.

Change-Id: If7d3321cda0ca512bc8c23b6541ce390b81a3e24
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3538
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-09-12 21:39:49 +00:00
sterni
2f44743824 docs(nix/buildLisp): document NIX_BUILDLISP_LISP_ARGS
Change-Id: I54294da08ee08bcf6cba6c792a2a2235b988a778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3422
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-26 11:26:52 +00:00
sterni
ea6ee1e219 docs(nix/buildLisp): move example further up
Make "Example" the second section again since it got a bit buried under
a lot of detailed documentation you won't necessarily need right away.

Change-Id: I481354d1761c590e5872dfce8c3cf9934e278673
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3421
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-08-26 11:26:52 +00:00
sterni
2dd8bc7977 fix(nix/buildLisp): prevent image loader from parsing arguments
CCL and SBCL create executables by dumping their image. As a
consequence, some part of the respective compiler is embedded in the
resulting executable which is executed and loads the image. For CCL and
SBCL this piece of software seems to unconditionally parse arguments
which can't be prevented since it happens before any lisp is loaded.

Luckily in both cases the parsing stops at `--`, so we can just pass
this via the wrapper — we just need to work around the problem that this
will of course be left in argv and confuse any later code. This can be
rectified by deleting everything prior to the first `--` in the global
argument list on startup in both cases.

In cases we do want to pass arguments to the image loader, we can use
the special NIX_BUILDLISP_LISP_ARGS environment variable which is
understood by the wrapper.

Note: This fix doesn't interfere with ECL since it is not using the
wrapper script at the moment.

Fixes b/136.

Change-Id: I3f95aa61e945e51428021ca18232ff15c923f870
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3357
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-08-26 11:26:52 +00:00
sterni
2fa32b563f docs(nix/buildLisp): document recent changes to buildLisp
Doing this in a separate CL to avoid having to track the intermediate
changes no one will ever see in documentation as well which would be
unnecessary effort.

* Multi-implementation support introduced in cl/3292 and refined in
  cl/3368 in terms of the user interface.

* Implementation specific srcs and deps introduced in cl/3321

* Implementation passthru attrs and rename from .sbcl -> .repl was done
  in cl/3359

* ECL added in cl/3297, CCL in cl/3350

Change-Id: Ia13f2aea4e7e091c00991fcbfc601de364413979
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3380
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
7df7cd6257 feat(nix/buildLisp): pass implementation description instead of name
Instead of using a string to refer to an internal set defined in
buildLisp, we just expose the relevant sets (as nix.buildLisp.sbcl,
nix.buildLisp.ecl, …) and receive them as the `implementation`
argument directly. This has several advantages:

* It becomes easier to extend buildLisp, even for downstream users:
  Since you can just pass your own set, there's nothing stopping you
  from adding support for another implementation in a downstream
  derivation without having to edit the buildLisp file in any way which
  is great if you're using e. g. builtins.fetchGit to import it.

* Users can mess with the implementation set by changing out some parts
  of it for customization purposes. Note that currently the sets use a
  lot of self-references which aren't even bound by a fix-point, so to
  make this work smoothly, we'd need to add some overriding mechanism.

* The buildLisp code becomes quite a bit clearer. Since we're now always
  dealing with the implementation set, the confusing distinction between
  `impl`, `impl.name` and `implementation` no longer exists. `impl` is
  now exclusively an abbreviation of `implementation` (we could make
  this more consistent in the future even).

Change-Id: I36d68069dd1315610b2f7159941507b465469b7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3368
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2021-08-24 22:00:15 +00:00
sterni
d7e70b1d72 feat(nix/buildLisp): add ccl
This adds support for Clozure's CL implementation to buildLisp. This is
quite trivial in comparison to ECL since SBCL and CCL have very similar
in how they work (so much so that CCL also suffers from b/136).

Also the similarities in the code actually added here are striking, so
I'll try to make an effort to reduce the code duplication in the
future.

To fix builds with CCL the following changes were made:

* //3p/lisp/nibbles: The double inclusion of the types.lisp file was
  fixed. CCL doesn't like double definitions and refuses to compile
  otherwise.

* //3p/lisp/physical-quantities: Update to a new bug fix release which
  contains a compilation fix for CCL.

* //3p/lisp/routes: apply a patch fixing the build which was previously
  failing due to a double definition.

* //3p/lisp/usocket: only depend on sb-bsd-sockets for SBCL and ECL, the
  latter of which seems to have a SBCL compatible implementation of the
  package.

* Conditionally include a few CCL-specific source files and add
  `badImplementation` entries for the remaining failures which are
  //fun/gemma (to be expected) and //web/panettone which fails with an
  incredibly vague message.

Change-Id: I666efdc39a0f16ee1bb6e23225784c709b04e740
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3350
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
02566cdcfb feat(nix/buildLisp): add ecl
Adds ECL as a second supported implementation, specifically a statically
linked ECL. This is interesting because we can create statically linked
binaries, but has a few drawbacks which doesn't make it generally
useful:

* Loading things is very slow: The statically linked ECL only has byte
  compilation available, so when we do load things or use the REPL it is
  significantly worse than with e. g. SBCL.

* We can't load shared objects via the FFI since ECL's dffi is not
  available when linked statically. This means that as it stands, we
  can't build a statically linked //web/panettone for example.

Since ECL is quite slow anyways, I think these drawbacks are worth it
since the biggest reason for using ECL would be to get a statically
linked binary. If we change our minds, it shouldn't be too hard to
provide ecl-static and ecl-dynamic as separate implementations.

ECL is LGPL and some libraries it uses as part of its runtime are as
well. I've outlined in the ecl-static overlay why this should be of no
concern in the context of depot even though we are statically linking.

Currently everything is building except projects that are using cffi to
load shared libaries which have gotten an appropriate
`badImplementations` entry. To get the rest building the following
changes were made:

* Anywhere a dependency on UIOP is expressed as `bundled "uiop"` we now
  use `bundled "asdf"` for all implementations except SBCL. From my
  testing, SBCL seems to be the only implementation to support using
  `(require 'uiop)` to only load the UIOP package. Where both a
  dependency on ASDF and UIOP exists, we just delete the UIOP one.
  `(require 'asdf)` always causes UIOP to be available.

* Where appropriate only conditionally compile SBCL-specific code and
  if any build the corresponding files for ECL.

* //lisp/klatre: Use the standard condition parse-error for all
  implementations except SBCL in try-parse-integer.

* //3p/lisp/ironclad: disable SBCL assembly optimization hack for all
  other platforms as it may interfere with compilation.

* //3p/lisp/trivial-mimes: prevent call to asdf function by substituting
  it out of the source since it always errors out in ECL and we hardcode
  the correct path elsewhere anyways.

As it stands ECL still suffers from a very weird problem which happens
when compiling postmodern and moptilities:
https://gitlab.com/embeddable-common-lisp/ecl/-/issues/651

Change-Id: I0285924f92ac154126b4c42145073c3fb33702ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3297
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: eta <tvl@eta.st>
2021-08-24 22:00:15 +00:00
sterni
0285ea7eac feat(nix/buildLisp): expose drvs built w/ the other implementations
For every implementation we support an extra passthru attribute with the
name of the implementation is created which points to a version of the
derivation built with that implementation. E. g. if we support CCL, ECL
and SBCL, third_party.lisp.alexandria would have:

* third_party.lisp.alexandria.sbcl
* third_party.lisp.alexandria.ecl
* third_party.lisp.alexandria.ccl

To make this possible, the REPL derivation which was called `sbcl`
originally has been renamed to `repl`.

Since some things won't build with all implementations, we introduce a
brokenOn argument which influences the meta.targets list that is
created, but won't prevent the passthru attrs from being created to
ease debugging failures.

Change-Id: Icd6af345143593fac30ded10deabf31172e5d48a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3359
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
ee6b2003fc feat(nix/buildLisp): implementation specific deps and srcs
Both the deps and srcs arguments may now have special “filter sets” in
the lists they receive as arguments. When building, buildLisp checks if
such sets either have a attribute named like the current implementation
or a "default" attribute. If yes, the set is replaced by the respective
attribute's value. If no, the set is removed from the list without
replacement.

This can be used to add elements for (a) specific implementation(s):

  { sbcl = buildLisp.bundled "sb-posix"; }

  { sbcl = ./sbcl/optional-sbcl.lisp; }

or to switch between files for different implementations:

  # If a implementation case is missing and no default set present,
  # no file will be added. Compilation will likely fail as a result.
  {
    ecl = ./tf-ecl.lisp;
    ccl = ./tf-ccl.lisp;
    sbcl = ./tf-sbcl.lisp;
  }

or to account for special behavior for a certain implementation:

  {
    ccl = ./ccl-quirk-impl.lisp
    default = ./ansi-impl.lisp;
  }

Change-Id: I082c3701d1f5063b92100bf336a83425471c269d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3321
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
acb5994e87 feat(nix/buildLisp): allow implementation-specifc bundled functions
By implementing a bundled function for an implementation, we can use a
custom one for a specific implementation. This is useful for
implementations like ECL where a require will be compiled as an
instruction rather than importing all new symbols into a dump, so using
the underlying static or shared object directly would be beneficial.

overrideLisp for bundled libraries now only allows overriding the name
and implementation arguments.

Change-Id: I9036b29157e8daa4d86ff87d603b044373711dbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3301
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
d344637fe2 refactor(nix/buildLisp): prepare multi implementation support
Concept is roughly:

* receive extra argument `implementation` that refers to the name of an
  implementation or rather an attribute in an internal attribute set
  telling buildLisp how to do certain build steps.

* We assume an implementation can execute lisp files as scripts and that
  we can implement the following main tasks in lisp:

  - Building a library (`genCompileLisp`)

  - Building an executable (`genDumpLisp`)

  - Loading a library dynamically (`genLoadLisp`)

  Based on that we can implement:

  - Running a test suite (`genTestLisp`)

  - A REPL preloaded with a libraries and their dependencies (`lispWith`)

  Additional attributes for implementing these parts genericly  are
  added as needed (`faslExt` and `runScript`).

* `genCompileLisp` no longer prints a shell script which concatenates
  the individual FASLs. Instead it does the step previously done by the
  shell script itself. In essence `genCompileLisp` now writes a lisp
  script which compiles and installs the library to build.
  This will allow us extra freedom for different implementations, e. g.
  for ECL we'll want to build a object file archive additionally to fasl
  files in order to be able to link proper executables.

* `genLoadLisp` and `genTestLisp` are almost generic (the former just
  sometimes would need to use different file extensions), but we
  integrate them into the implementation “API” to facilitate minor
  tweaks we need to do like the `fasc` extension for ECL's native FASL
  files.

Change-Id: I1b8ccc0063159638ec7af534e9a6b5384e750193
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3292
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-24 22:00:15 +00:00
sterni
59195e76fa docs(nix/buildLisp): mention drawback of non-parallel compilation
To be fair this hardly matters since SBCL is quite fast, but compiling
ironclad with ECL is quite the experience…

Change-Id: Ib89cc50e5d557acec51fdb085bcbdfc99736221e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3342
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-12 15:08:15 +00:00
sterni
0298ec862e fix(nix/buildLisp): remove misplaced makeOverridables
`makeOverriddable` doesn't work for bundled sbclWith as is because it
uses the `//` operator internally which doesn't work with the types
`bundled` and `sbclWith` accept as arguments (string and list
respectively).

What's more, `bundled` already uses `makeOverridable` and allows to
override the internal call to `library` via `overrideLisp`. For
`sbclWith` no such mechanism exists, but this seems to be no concern for
now: Using `overrideLisp` for this hasn't worked so far (and failed with
a _hideous_ evaluation error), so there doesn't seem to be any real
demand for this feature. Maybe a feature for another CL.

Change-Id: I0b2f34c00a2143cd66dd43a6b1b2880af997ee50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3296
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-10 14:31:52 +00:00
sterni
c8f521e96a refactor(nix/buildLisp): use passthru for extra attributes
Using passthru and appending the attributes via `//` have the same
effect with a subtle difference: In the latter case re-evaluating
the derivation when using the underlying `mkDerivation`'s
`overrideAttrs` will delete all appended attributes. Using
passthru at least preserves the attributes although the self
reference to the derivation in `passthru.sbcl` will become
outdated (unless updated manually).

Change-Id: I8b85009f386b9375b86a23fd50c4ec8c6a9dea7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3257
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-08-01 23:37:03 +00:00
Vincent Ambo
31939acd6c fix(buildLisp): Override LANG in sbclWith for misconfigured machines
Machines on which LANG is misconfigured have trouble with SBCL loading
files that contain characters in certain encodings. This overrides
whichever local LANG (if any) is set.

Change-Id: Ic4341a01c4393e7f697de6cecc58dea4f2d85987
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2076
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-08 18:35:28 +00:00
Griffin Smith
7a344fbb5e feat(nix/buildLisp): Expose an sbcl attribute
Expose an `sbcl` attribute on packages and programs, to allow for easier
development either with SLY or on a REPL.

Change-Id: Ide4d087a5223561e1fe192ef32dc593c54b5a20e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1834
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-23 22:21:17 +00:00
Griffin Smith
3089f6b6ce feat(nix/buildLisp): Add abstraction for test suites
Add support for explicitly specifying tests as part of a buildLisp
program or library.

Change-Id: I733213c1618f0fa60f645465560bce0522641efd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1481
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-27 14:18:32 +00:00
Griffin Smith
2b7fe6f957 fix(nix/buildLisp): Set LANG to C.UTF-8 on program builds
This was already happening for libraries, but not for programs - as a
result, programs built with libraries that contained unicode (eg
cl-unicode, uax-15, ...) would fail to build due to character encoding
issues when loading the FASLs.

Change-Id: I66149b585e85b213d0c026153140a1925536bd29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1469
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-26 21:56:54 +00:00
eta
4dca44ac72 fix(buildLisp): use full store path in FASL output pathnames
If you compiled dirA/test.lisp and dirB/test.lisp in the same library,
they'd both get written to /test.fasl and the second would overwrite the
first. Instead, use the whole store path (with / swapped for -) as the fasl
filename.

Change-Id: I4eb88b5d33757751e1f67e72ed328bd58079b1b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/944
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-06 22:01:09 +00:00
Vincent Ambo
4bbbb58cb5 chore: Rename pkgs->depot in all Nix file headers 2020-02-21 13:54:53 +00:00
Vincent Ambo
8e8bbbca04 fix(nix/buildLisp): Ensure SBCL uses UTF-8 encoding 2020-01-29 10:12:07 +00:00
Vincent Ambo
0001dfd7f6 docs(nix/buildLisp): Update the README with actual docs 2020-01-29 00:03:24 +00:00
Vincent Ambo
ca60eafa80 feat(nix/buildLisp): Add 'bundled' function for built-in libraries
Makes it possible to add virtual dependencies on built-in libraries,
e.g. `buildLisp.bundled "sb-posix"`.
2020-01-26 23:58:31 +00:00
edef
30286d5454 chore(buildLisp): use lib.optionalString where applicable 2020-01-24 22:18:19 +00:00
Vincent Ambo
21e9a65a35 fix(nix/buildLisp): Don't load binaries in sbclWith
Adds an attribute on each Lisp derivation that specifies whether it is
a binary or not. This attribute is then filtered for in sbclWith.
2020-01-17 16:41:57 +00:00
Vincent Ambo
e1cc4966b7 feat(nix/buildLisp): Support passing programs to sbclWith
Adds the necessary attributes on derivations created by
buildLisp.program for them to be passed to buildLisp.sbclWith.

This makes it possible to easily spin up Lisp environments that
contain everything needed for a given program.
2020-01-17 12:44:24 +01:00
Vincent Ambo
09720e2da2 fix(buildLisp): Wrap executables to set load paths correctly
I can not currently find a way to set the CFFI variables correctly to
get it to load libraries from Nix.

In the absence of that feature, a wrapper also does the trick.
2020-01-09 03:32:29 +00:00
Vincent Ambo
44820827d1 feat(buildLisp): Initial implementation of foreign library loading
Adds a new 'native' parameter to the buildLisp functions in which
libraries can be passed in.

This does not yet work with CFFI packages.
2020-01-09 02:57:02 +00:00
Vincent Ambo
2a170f1ed7 fix(buildLisp): Perform a topological sort of dependencies
This ensures that dependencies are loaded in the correct order in
larger dependency graphs.
2020-01-09 00:46:20 +00:00
Vincent Ambo
e3a8dc9500 fix(buildLisp): Cursed code to fix load ordering
It's not enough to compile in the right order - turns out you also
have to load the compiled objects in the right order.

To achieve this some cursed code has been added that changes the Lisp
generated by Nix to compile the other Lisp so that it also generates
some bash, which Nix can then use to concatenate the FASLs in the
right order to feed them to Lisp again.

It works but I'll replace it with a more elegant solution once one is
needed.
2020-01-08 23:57:34 +00:00
Vincent Ambo
7bc10eb9b7 feat(buildLisp): Add initial, tiny example program 2020-01-08 21:41:43 +00:00
Vincent Ambo
bdad8f6642 feat(buildLisp): Implement buildLisp.program to dump executables
Dumps the executable image from SBCL to $out/bin/$name.

Image compression is disabled.
2020-01-08 21:39:26 +00:00
Vincent Ambo
2bfe073eb2 refactor(buildLisp): Inline dependency loading in genCompileLisp 2020-01-08 21:39:06 +00:00
Vincent Ambo
b5e1e81a3d feat(buildLisp): Add function to wrap SBCL with dependencies
Adds `buildLisp.sbclWith` which creates an SBCL wrapper the contains
all the requested dependencies.
2020-01-08 19:38:29 +00:00
Vincent Ambo
ca199a57d9 feat(buildLisp): Implement dependency loading & propagation
Similar to buildGo.nix, the library derivations carry information
about their dependencies which is merged when a load file is
instantiated.

The load files are created when compiling libraries, but will in the
future also be created when wrapping SBCL and dumping images.
2020-01-08 18:40:53 +00:00
Vincent Ambo
1297afec4b fix(buildLisp): Fail the build on compilation errors
This needs to be handled explicitly in the COMPILE-FILE form.
2020-01-08 17:53:06 +00:00
Vincent Ambo
a954bd8d5e feat(nix/buildLisp): Add initial sketch including buildLisp.library
Adds a Nix function to build a Lisp library out of a specified set of
Nix files. All files are combined into a single FASL.

This is by design only compatible with SBCL (for now).
2020-01-08 02:00:54 +00:00