decode-RFC2047 used babel's octets-to-string, but we can replace it with
the function of the same name from flexi-streams. This doesn't make a
difference for the moment, but will be useful in the future:
flexi-streams provides de- and encoding streams that we'll be able to
use to replace and augment some of the stream based MIME part handling
code in mime4cl. babel doesn't have as powerful stream functionality
although it seems to be planned.
Another big upside of flexi-streams is that we'll be able to replace
delimited-input-string using it. This should allow us to slowly work
towards correct and more efficient decoding of MIME bodies.
Change-Id: I17174f1c96c5be7d103d396564e6aa0fe24c80fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8371
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Add missing dependency alexandria. This update adds a feature to disable
the cffi which would be neat for ECL.
Change-Id: Iad5a4646317fb26bb2dec7bcf3d883075ab24842
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7564
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The problem went away once again, let's see how long it'll last this time.
As it turns out, CCL has a Unicode Standard conforming string
implementation that doesn't allow the use of (lone) surrogate code
points, requiring us to disable a test in cl-json which tested the
behavior of en- and decoding of such a (technically illegal) string.
Change-Id: I8bfa482934bbf94f86cecdde02d5c3d4e77950a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6204
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Pin cl-json to a branch on my fork of cl-json which implements encoding
and decoding of non-BMP Unicode codepoints (>= U+10000) using UTF-16
surrogate points as the JSON specs prescribes. While we're at it, also
enable the test suite of the package.
This resolves the annoying issue of panettone garbling up some Unicode
characters by sending invalid JSON to cheddar and then incorrectly
decoding the returned result.
Fixes: b/145.
Change-Id: I44cb38c2775c0e5512d75f51dff0d1deb39f8f78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5884
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
SCLF is quite a big utility library (almost 3€ LOC) with limited
portability (CMUCL, SBCL and CLISP to an extent). Continuing to maintain
it is an unnecessary burden, as depot only uses a fraction of it which
is now inlined into the respective users (mime4cl and mblog).
In the future trimming down ex-sclf.lisp may make sense either by
refactoring the code that uses it or by moving interesting utilities
into e.g. klatre.
Change-Id: I2e73825b6bfa372e97847f25c30731a5aad4a1b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5922
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
This switches upstream from hankhero/cl-json to
sharplispers/cl-json (the former of which had its last commit in 2014).
Sadly the new upstream hasn't decided on an appropriate fix for b/145
yet (due to concern about backwards compatibility, apparently). I did
not look before working on a fix, so I have an 90% finished fix which
is (I think) better than the already proposed ones, so I'll patch it in
here eventually.
Change-Id: I9e39e138fa655794b864db5f268bdfdc35788fcc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5795
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Accessing the headers of a MIME message feels like something mime4cl
should handle. We implemented this ad hoc in mblog before in order to
not need to worry about doing it in a sensible way. Now we introduce a
decent-ish interface for getting a header from a MIME message,
mime-message-header-values:
* It returns a list because MIME message headers may appear multiple
times.
* It decodes RFC2047 only upon request, as you may want to be stricter
about parsing certain fields.
* It checks header name equality case insensitively.
The code for decoding the RFC2047 string is retained and still uses
babel for doing the actual decoding.
Change-Id: I58bbbe4b46dbded04160b481a28a40d14775673d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5150
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
By computing the amount the stream position advanced we can save a
syscall on every read which speeds up mime:mime-body-stream by /a lot/,
e.g. extracting a ~3MB attachment drops from over 15s to under ~0.5s.
There's still a lot to be gained and correctness left to be desired
which can be addressed as described in the newly added comment.
Change-Id: I5e1dfd213aac41203f271cf220db456dfb95a02b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5073
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Seems like some issues to do with bytecode compilation have been fixed
at HEAD. closer-mop compiles again and an ironclad failure with the
next quicklisp/channel bump is avoided.
In this change pathname handling in ECL also changed somehow, causing it
to make the :directory part absolute by prefixing it with a slash which
made ld.bfd unhappy while linking an output path that began with a
double slash. This problem can be avoided by constructing the path as
ANSI Common Lisp intended. The truename on the out path is important to
make it recognize that it is indeed a directory.
Change-Id: I5e744022b92502f99ac0b33411a6be443707e200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5076
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Having #+cmu all over the place suggests that we maintain CMUCL support
or test with CMUCL which is not the case.
Change-Id: Ia0828cb1ac48e49acdee6fef7a0fa2c04c1805b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5068
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This should be a net positive for portability and lets us drop some of
the CMUCL cruft (which we don't test anyway, CMU support may have
regressed regardless).
Change-Id: I85664d82d211177da1db9eebea65c956295b09f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5067
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Moves to the derivation-based git fetchers everywhere in third-party.
This might help with forward-compatibility with newer Nix versions,
though that's not our primary concern right now.
Change-Id: I565bb72585b8639893e9ea3a9e233338aede63a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3903
Tested-by: BuildkiteCI
Reviewed-by: zseri <zseri.devel@ytrizja.de>
* 3p/lisp/closer-mop: closer-mop no longer builds with ECL (see linked
issue), so let's mark it as broken for now.
Change-Id: I97c29d718682cec4ecc682ff1593d0ce9aca0010
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4607
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
this one was a little more difficult because it needs a patch, there's
something wonky with the definition order
fwiw, the upstream cvs repository ... server errors.
Change-Id: I2d99359edec36b578389f1be1fcf077743c29c4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4342
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
nixpkgs includes a lispPackages set which is generated from something.
In the meantime, we pretty much never update our Lisp deps.
This commit ties our sources to nixpkgs.lispPackages where the desired
package is included in nixpkgs (which is actually most of them!)
Change-Id: I520a006535980271b2fa4e0ed4e34029475dcbef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4331
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
* move packages and adapt them for the depot structure instead of
briefcase
* drop linear-programming package, it didn't build anyways
Note that at least some of these packages (e.g. prove) are deprecated
upstream, but lets sort that out later.
Change-Id: I7f5a5faa29d57f060b21ac8e1706090866a82000
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4330
Autosubmit: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Adds a simple generic function find-mime-text-part which returns the
first suitable text/* part in any MIME part it is given.
Has no meaningful alternatives handling at the moment: It will pick the
first text part and doesn't allow specifying a preference.
Change-Id: Id9b113b3ef3ca1a575ce8f3582a4f85e30edfb43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3379
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This seems to be unnecessary: It doesn't muffle any SBCL warnings that
affect a current version and does nothing special otherwise.
Change-Id: I36efde761fc95d9df735f29d2eb369c6b61853c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3486
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Luckily we don't need to deal with this mess since all our
implementations work similarly wrt streams and “wide” characters.
Change-Id: I3ccc606a59c42791f2591d752673c867d848a332
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3485
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
The following changes are required to make mime4cl build:
* file-position doesn't like to be called with NIL as the position
argument, so we have to make sure to not do that in
stream-file-position. My workaround is a bit clunky, but works.
* Tests discover the sample file via relative path resolution. This
doesn't work when they are imported into the nix store as individual
files. Instead we make use of the fact that DEFVAR is a no-op if the
variable is already defined and inject a file via the nix build that
sets the relevant ones. For the path to sample1.msg, we need to create
a new variable.
Change-Id: I74eeda7bf2c2a4f64cc2b90e72081513ec3285d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3270
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Used http://wcp.sdf-eu.org/software/mime4cl-20150207T211851.tbz (sha256
5a914669bba7561efe59a4fd0817204c07ad2add98b03ae206ef185ac04affb3).
Importing seems sensible since there's no upstream repo nor has their
been a release since 2015.
This is just an import commit, so the changes made to make it build are
more discoverable as their own commit.
Change-Id: I2ff28c3c7433abdf7857204bc89eaf9edc0b1cbc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3378
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Used http://wcp.sdf-eu.org/software/npg-20150517T144652.tbz (sha256
42e88f6067128fbdb3a3d578371c9b0ee2a34f1d36daf80be8a520094132d828).
There's no upstream repository nor a release since 2015, so importing
seems to make a lot of sense.
Since we can't subtree making any depot-related changes in a separate CL
-- this is only the source import.
Change-Id: I64c984ca0a84b9e48c6f496577ffccce1d7bdceb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3377
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Adding the default.nix is quite straightforward, however we have to make
today's SBCL happy: due to package locking it no longer likes sclf using
an sb-impl internal constant for some reason. This is however a good
opportunity to clean up the stat-*-time code: It converted the times in
an implementation specific way even though time.lisp does provide a
generic way to convert between unix and universal time. Note that the
updated ASDF file is untested, but should be a trivial enough change.
Change-Id: If193bf830ac704cc53e0855d8e9fff2b5a5ef291
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3268
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Used http://wcp.sdf-eu.org/software/sclf-20150207T213551.tbz (sha256
a231aeecdb9e87c72642292a1e083fffb33e69ec1d34e667326c6c35b8bcc794).
There's no upstream repository nor a release since 2015, so importing
seems to make a lot of sense.
Since we can't subtree making any depot-related changes in a separate CL
to make them more discoverable -- this is only the source import.
Change-Id: Ia51a7f4029dba3abd1eee4eeebcf99aca5c5ba4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3376
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This one requires a bit of jumping through hoops. Patching the dtd /
catalog lookup is quite straightforward and similar to cxml, but the
CLOSURE-HTML:*html-dtd* variable gives us a bit of trouble: It is
defined quite late in `html-parser.lisp`, but files that need to be
built first already reference it. SBCL has apparently decided to be
particular about this and emits a `WARNING` (!) condition for this
which is also worthy of `failure-p` of `compile-file` being true,
so that `buildLisp` will abort compilation. We workaround this issue
by injecting an extra source file which `defvar`s the desired symbol.
A similar issue exists with `dump-dtd` which references
`CL-USER:*HTML-DTD*` for some reason. Since this is a helper intended
for development (?) and not exported we just throw it away via a
patch.
Change-Id: Ic0f92815a21f3793925c49a70a72f4a86791efe4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3263
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This adds support for Clozure's CL implementation to buildLisp. This is
quite trivial in comparison to ECL since SBCL and CCL have very similar
in how they work (so much so that CCL also suffers from b/136).
Also the similarities in the code actually added here are striking, so
I'll try to make an effort to reduce the code duplication in the
future.
To fix builds with CCL the following changes were made:
* //3p/lisp/nibbles: The double inclusion of the types.lisp file was
fixed. CCL doesn't like double definitions and refuses to compile
otherwise.
* //3p/lisp/physical-quantities: Update to a new bug fix release which
contains a compilation fix for CCL.
* //3p/lisp/routes: apply a patch fixing the build which was previously
failing due to a double definition.
* //3p/lisp/usocket: only depend on sb-bsd-sockets for SBCL and ECL, the
latter of which seems to have a SBCL compatible implementation of the
package.
* Conditionally include a few CCL-specific source files and add
`badImplementation` entries for the remaining failures which are
//fun/gemma (to be expected) and //web/panettone which fails with an
incredibly vague message.
Change-Id: I666efdc39a0f16ee1bb6e23225784c709b04e740
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3350
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Adds ECL as a second supported implementation, specifically a statically
linked ECL. This is interesting because we can create statically linked
binaries, but has a few drawbacks which doesn't make it generally
useful:
* Loading things is very slow: The statically linked ECL only has byte
compilation available, so when we do load things or use the REPL it is
significantly worse than with e. g. SBCL.
* We can't load shared objects via the FFI since ECL's dffi is not
available when linked statically. This means that as it stands, we
can't build a statically linked //web/panettone for example.
Since ECL is quite slow anyways, I think these drawbacks are worth it
since the biggest reason for using ECL would be to get a statically
linked binary. If we change our minds, it shouldn't be too hard to
provide ecl-static and ecl-dynamic as separate implementations.
ECL is LGPL and some libraries it uses as part of its runtime are as
well. I've outlined in the ecl-static overlay why this should be of no
concern in the context of depot even though we are statically linking.
Currently everything is building except projects that are using cffi to
load shared libaries which have gotten an appropriate
`badImplementations` entry. To get the rest building the following
changes were made:
* Anywhere a dependency on UIOP is expressed as `bundled "uiop"` we now
use `bundled "asdf"` for all implementations except SBCL. From my
testing, SBCL seems to be the only implementation to support using
`(require 'uiop)` to only load the UIOP package. Where both a
dependency on ASDF and UIOP exists, we just delete the UIOP one.
`(require 'asdf)` always causes UIOP to be available.
* Where appropriate only conditionally compile SBCL-specific code and
if any build the corresponding files for ECL.
* //lisp/klatre: Use the standard condition parse-error for all
implementations except SBCL in try-parse-integer.
* //3p/lisp/ironclad: disable SBCL assembly optimization hack for all
other platforms as it may interfere with compilation.
* //3p/lisp/trivial-mimes: prevent call to asdf function by substituting
it out of the source since it always errors out in ECL and we hardcode
the correct path elsewhere anyways.
As it stands ECL still suffers from a very weird problem which happens
when compiling postmodern and moptilities:
https://gitlab.com/embeddable-common-lisp/ecl/-/issues/651
Change-Id: I0285924f92ac154126b4c42145073c3fb33702ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3297
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: eta <tvl@eta.st>
Seems to fix weird issues related to CCL I encountered.
Change-Id: Id5c34c7c98e22b2bc56d6723af85cac1e031ed72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3365
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Also allows us to enable the SBCL opt modules. Upstream changes as
sharplispers has the only maintained nibbles fork atm.
Change-Id: I6f0d1b9e4e570169e5f5c584364948e2031063af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3364
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This was previously propagated from somewhere else, but is actually
needed here.
Change-Id: I921758320ff5567b451291c69c8532d43a5c898c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3358
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Rename my //users directory and all places that refer to glittershark to
grfn, including nix references and documentation.
This may require some extra attention inside of gerrit's database after
it lands to allow me to actually push things.
Change-Id: I4728b7ec2c60024392c1c1fa6e0d4a59b3e266fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2933
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
sbcl 2.0.9 introduced a new warning:
> minor incompatible change: the compiler signals a warning at
> compile-time when an initform of T, NIL or 0 does not match
> a STANDARD-CLASS slot's declared type.
This broke a few packages, but they all have been fixed upstream in the
meantime and we only need to bump their versions. The culprits are:
* defclass-std which possibly has become unmaintained since the fix
(december 2020).
* cl-prevalence which also needs one symbol from bt now
* lisp-binary which also includes a new file now
Change-Id: I06bb47a129d5ef912a623315c1281aedd1ceac2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2934
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
