This refactor is driven by the following (ultimate) aims:
- Get rid of as much of the custom stream code in mime4cl which makes
less code to maintain in the future.
- Lay the groundwork for correct handling of 8bit transfer encoding:
The mime4cl we inherited assumes that any MIME message can be decoded
completely by the CL implementation (in SBCL's case using latin1)
into CHARACTERs. This is not necessarily the case. flexi-streams
allows changing how the stream is decoded on the fly and also has
support for reading the underlying bytes which is perfect for the
requirements decoding MIME has.
- Since flexi-streams uses trivial-gray-streams, it supports
READ-SEQUENCE. Taking advantage of this may improve decoding
performance significantly in the future.
This incurs the following changes:
- Naturally we now open given files as binary files in MIME-MESSAGE.
Given strings are encoded using STRING-TO-OCTETS and then passed on
to a new octet vector method. Instead of MY-STRING-INPUT-STREAM this
now uses flexi-streams' WITH-INPUT-FROM-SEQUENCE.
- OPEN-FILE-PORTION and OPEN-DECODED-FILE-PORTION need to be merged,
since the transfer encoding not only implies an extra decoder stream
that needs to be attached after file portion stream, but also imply a
certain encoding of the stream itself (mostly binary vs. ASCII).
As flexi-streams can change their encoding on the fly this could be
untangled again, but it is not strictly necessary.
As before, we use the DATA slot of the file portion to create a fresh
stream if possible. Instead of strings we now use an vector of octets
to match MIME-MESSAGE.
The actual portioned stream relies on POSITIONED-FLEXI-INPUT-STREAM, a
subclass of the stock FLEXI-INPUT-STREAM class, described below.
- POSITIONED-FLEXI-INPUT-STREAM replaces DELIMITED-INPUT-STREAM. It is
created using MAKE-POSITIONED-FLEXI-INPUT-STREAM which accepts the
same arguments as MAKE-FLEXI-STREAMS and, additionally, :IGNORE-CLOSE.
A POSITIONED-FLEXI-INPUT-STREAM works the same as an
FLEXI-INPUT-STREAM, but upon creation, the underlying stream is
rewinded or forwarded to the argument given by :POSITION using
FILE-POSITION.
If :IGNORE-CLOSE is T, a call to CLOSE is not forwarded to the
underlying stream.
Change-Id: I2d48c769bb110ca0b7cf52441bd63c1e1c2ccd04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8559
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Since rt.lisp seems to start tests in parallel, the informational output
about which sample file is being tested gets mangled in all sorts of
ways. The solution is to just loop over the sample files outside a test
and schedule a single test case per sample file from there.
Change-Id: I4494e4a526ce6d92a298cf7daf06c8013c7ca605
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8569
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This makes sure that initializing coder-stream-mixin (for the most part)
has the same interface as initializing qbase64:decode-stream. This will
make integrating that as a faster replacement to
mime4cl:base64-decoder-stream a bit easier.
The idea is to replace the char by char base64 decoder with one that
supports read-sequence. After that deliminited-input-stream needs to
gain support for read-sequence as well, so we can actually take
advantage of this fact. Finally, we'll have to evaluate the remaining
decoders and think about switching the (base64) encoders over as well.
Change-Id: If971da02437506e00a7c9fab2b94efc42725e62d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8555
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
For whatever reason, there were two sort of identical tests, mime.1 and
mime.2, in the mime4cl test suite: The former tested *sample1-file* and
the latter all messages *samples-directory*—in the same way, parsing the
original and a re-rendered version of the message to check if they were
equal.
We can just move sample1.msg into *samples-directory*, get rid
of *sample1-file* and thus pave the way for more test messages in the
future.
Change-Id: I843be331682b731af6ae02a4648ba1c64aaf59a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8546
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
The generic function itself needs to be defined using defgeneric,
defmethod is used for a defining method of a generic function, i.e. how
it should behave when confronted with a certain class.
Change-Id: Idd38afa02b56c5002e215decfff7f0c25267eab5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8532
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
decode-RFC2047 used babel's octets-to-string, but we can replace it with
the function of the same name from flexi-streams. This doesn't make a
difference for the moment, but will be useful in the future:
flexi-streams provides de- and encoding streams that we'll be able to
use to replace and augment some of the stream based MIME part handling
code in mime4cl. babel doesn't have as powerful stream functionality
although it seems to be planned.
Another big upside of flexi-streams is that we'll be able to replace
delimited-input-string using it. This should allow us to slowly work
towards correct and more efficient decoding of MIME bodies.
Change-Id: I17174f1c96c5be7d103d396564e6aa0fe24c80fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8371
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
SCLF is quite a big utility library (almost 3€ LOC) with limited
portability (CMUCL, SBCL and CLISP to an extent). Continuing to maintain
it is an unnecessary burden, as depot only uses a fraction of it which
is now inlined into the respective users (mime4cl and mblog).
In the future trimming down ex-sclf.lisp may make sense either by
refactoring the code that uses it or by moving interesting utilities
into e.g. klatre.
Change-Id: I2e73825b6bfa372e97847f25c30731a5aad4a1b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5922
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Accessing the headers of a MIME message feels like something mime4cl
should handle. We implemented this ad hoc in mblog before in order to
not need to worry about doing it in a sensible way. Now we introduce a
decent-ish interface for getting a header from a MIME message,
mime-message-header-values:
* It returns a list because MIME message headers may appear multiple
times.
* It decodes RFC2047 only upon request, as you may want to be stricter
about parsing certain fields.
* It checks header name equality case insensitively.
The code for decoding the RFC2047 string is retained and still uses
babel for doing the actual decoding.
Change-Id: I58bbbe4b46dbded04160b481a28a40d14775673d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5150
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
By computing the amount the stream position advanced we can save a
syscall on every read which speeds up mime:mime-body-stream by /a lot/,
e.g. extracting a ~3MB attachment drops from over 15s to under ~0.5s.
There's still a lot to be gained and correctness left to be desired
which can be addressed as described in the newly added comment.
Change-Id: I5e1dfd213aac41203f271cf220db456dfb95a02b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5073
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Having #+cmu all over the place suggests that we maintain CMUCL support
or test with CMUCL which is not the case.
Change-Id: Ia0828cb1ac48e49acdee6fef7a0fa2c04c1805b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5068
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This should be a net positive for portability and lets us drop some of
the CMUCL cruft (which we don't test anyway, CMU support may have
regressed regardless).
Change-Id: I85664d82d211177da1db9eebea65c956295b09f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5067
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Adds a simple generic function find-mime-text-part which returns the
first suitable text/* part in any MIME part it is given.
Has no meaningful alternatives handling at the moment: It will pick the
first text part and doesn't allow specifying a preference.
Change-Id: Id9b113b3ef3ca1a575ce8f3582a4f85e30edfb43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3379
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
The following changes are required to make mime4cl build:
* file-position doesn't like to be called with NIL as the position
argument, so we have to make sure to not do that in
stream-file-position. My workaround is a bit clunky, but works.
* Tests discover the sample file via relative path resolution. This
doesn't work when they are imported into the nix store as individual
files. Instead we make use of the fact that DEFVAR is a no-op if the
variable is already defined and inject a file via the nix build that
sets the relevant ones. For the path to sample1.msg, we need to create
a new variable.
Change-Id: I74eeda7bf2c2a4f64cc2b90e72081513ec3285d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3270
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Used http://wcp.sdf-eu.org/software/mime4cl-20150207T211851.tbz (sha256
5a914669bba7561efe59a4fd0817204c07ad2add98b03ae206ef185ac04affb3).
Importing seems sensible since there's no upstream repo nor has their
been a release since 2015.
This is just an import commit, so the changes made to make it build are
more discoverable as their own commit.
Change-Id: I2ff28c3c7433abdf7857204bc89eaf9edc0b1cbc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3378
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
