Commit graph

33 commits

Author SHA1 Message Date
Vincent Ambo
5e08f9b6c6 refactor(cheddar): Support multiple shortlinks & custom link sets
This makes it possible for users of cheddar as a library to supply
their own shortlinks. In practice it is unlikely anyone will do this,
but the change also allows us to (relatively) easily add additional
shortlinks to the set used by TVL.

Note that Cheddar is primarily intended for use by TVL and the default
rendering function interfaces have not changed, and will default to
using TVL shortlinks.

A new public function `format_markdown_with_shortlinks` has been added
with which users can use an alternative set of shortlinks. This
function should not be used in TVL depot code.

Change-Id: I4ddab28cbcf45d07c51323b7b730b96e62922816
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3083
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-10 11:32:33 +00:00
Vincent Ambo
3c7d4c74be chore(3p/overlays): Remove LLVM11 bump from overlay
* This was mostly for //third_party/nix and its dependencies which now
  have been set to use llvmPackages_11 manually.

* For //users/grfn/achilles we also manually select the newer LLVM version.

* //tools/cheddar doesn't seem to need llvm anymore.

* //third_party/buzz also compiles with clang 7.1.0

* replace clang-tools everywhere with new attribute clang-tools_11

For the future we may want to have something similar again, but it may
not be necessary to invest too much time into it: nixpkgs is set to
upgrade their default llvmPackages to LLVM 11 as well at some point in
the near future.

Co-Authored-By: sterni <sternenseemann@systemli.org>
Change-Id: Id83868dbc476a6c776b59518b856c933f30ea79d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-05-24 18:15:05 +00:00
sterni
c8ecea1032 docs(cheddar): fix typo in build.rs comment
Change-Id: Id323b4a187b3837d60ac6b8917f3e8571954b8e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3119
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-05-12 15:38:57 +00:00
Vincent Ambo
a9902dadcf feat(cheddar): Implement highlighting of CL and bug shortlinks
Implements highlighting of shortlinks like cl/123, or b/123.

Highlighting works by replacing the input Markdown using a simple
regular expression replacement.

We also considered parsing and replacing these links in the Markdown
AST, but it would have been significantly more complex (due to arena
allocation) for little upside and no obvious performance benefit.

Change-Id: I53f03fb17491046d89d0b7f605929571c11ee9a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3082
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
2021-05-04 15:50:26 +00:00
Vincent Ambo
5e6685c689 feat(cheddar): Add a build script that injects BAT_SYNTAXES
The BAT_SYNTAXES environment variable is injected during Nix builds
and points to the build result of //third_party/bat_syntaxes, which is
included in the cheddar binary.

During development this is a bit annoying: Opening a Nix shell will
set the right variable, but it will also mess with the Cargo
environment in ways that make it annoying to use (*and* require a
pre-build of all dependencies as required by naersk).

For an easier development flow, this adds a build script that
automatically runs Nix, fetches the path to the syntax set and sets
the environment variable. It is a no-op if the variable is already
set.

This makes for a smoother development experience when working on
cheddar in a depot checkout.

Change-Id: Ib5435518897e3443247edd9f362821c0b9ff756b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3081
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-05-03 20:18:33 +00:00
Vincent Ambo
d311af9bc0 refactor(cheddar): Split out a library with rendering logic
Splits `main.rs` into `lib.rs` and `bin/cheddar.rs`, which enables
reuse of cheddar's rendering logic in other Rust applications.

Change-Id: Ifd1a44a8d1620c595550a0a497a25b0563e917ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3060
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-05-03 13:57:43 +00:00
Vincent Ambo
d62fab4168 refactor(cheddar): Consistent use of depot.third_party vs. pkgs
In preparation for the solution of b/108, we need to consistently use
`depot.third_party` for packages that are only packed in the TVL depot
and `pkgs` for things that come from nixpkgs.

Change-Id: I49d82726b2f3bd7d4923effdd9a7e3f67ddc0659
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2916
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 12:06:40 +00:00
Vincent Ambo
057331aa64 feat(cheddar): Set front_matter_delimiter = '---' extension
This chops off "front matter", which we have - for example - in Tvix
documentation that is rendered through Pandoc.

Change-Id: I65c7a4ad8171423a50aaf47c046dab8c774346d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2717
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2021-03-31 22:20:49 +00:00
Vincent Ambo
ca578c04ec chore(cheddar): Bump crate dependencies
This required some minor Comrak-related refactoring.

Change-Id: I5c5898eb895bd5d8743949458ee9406087fcff22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2690
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2021-03-31 22:06:50 +00:00
Vincent Ambo
c710509078 test(cheddar): Add simple Markdown rendering tests
Covers some of the odd things we do, specifically callouts and code
rendering.

Change-Id: Ib8542373b434b53d277b0d8c9ddb78ac7c5176a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2689
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-03-31 22:06:50 +00:00
Vincent Ambo
87d70d2a80 style(cheddar): Apply rustfmt
Change-Id: I378b5ae618b01244edc090ad1495fbd56857d7c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1562
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-02 21:57:35 +00:00
Vincent Ambo
6936ee40af feat(tools/cheddar): Add endpoint for Markdown rendering
Similar to the source code highlighting endpoint, but for Markdown.
This is to be used by the bug tracker, as well as Sourcegraph in the
future.

Change-Id: I4bea5c46ba969ba9965b61409e1c19c2edf1246c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1424
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-25 01:42:03 +00:00
Vincent Ambo
082ed2a74f chore(cheddar): Clean up all clippy lints
Change-Id: I5544c10e964b57891fce6ea01f09eeec406e9bef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/510
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-20 03:00:46 +00:00
Vincent Ambo
e27b0a3013 feat(cheddar): Add Sourcegraph-compatible highlighting server
Sourcegraph uses a component called syntect_server to syntax-highlight
source files.

Since we already have custom syntaxes, overrides and configuration we
might as well use them with Sourcegraph!

This implements the syntect_server "protocol" (it's just a single
endpoint) so that we can swap out the syntect_server component with
cheddar.

Note: There's a few clippy lints here, but they're being solved in a
followup commit because I wanted to take care of all of them at
once (not just the ones introduced in this change).

Change-Id: Ib518a2fa1b9fee299fe599482403599583cac479
Reviewed-on: https://cl.tvl.fyi/c/depot/+/509
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-20 02:59:20 +00:00
Vincent Ambo
dbb4a57bbf style(cheddar): Reformat with rustfmt
Change-Id: I9a0764f23c1269447a8b18f3128074fc58f70456
Reviewed-on: https://cl.tvl.fyi/c/depot/+/492
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-19 02:49:41 +00:00
Vincent Ambo
d1524a5a6f refactor(cheddar): Pass references readers & writers into renderers
This paves the way for using other things than stdin/stdout as
sources/sinks, which is required for example for implementing a
syntect_server replacement based on cheddar.

Change-Id: I5779db8dbf7b7ced109c26b940f721d237d60785
Reviewed-on: https://cl.tvl.fyi/c/depot/+/491
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-19 02:49:41 +00:00
Vincent Ambo
5e5e02d6a7 refactor(cheddar): Switch to clap-rs for command line arguments
The complexity of the arg parsing is increasing somewhat because we're
adding more features to cheddar, so to set us up for that this
switches the arg parsing to the somewhat more flexible clap.

Change-Id: I187bc0c1b6c6bd596fa0f6bb494b04e335262ba9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/445
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-06-17 01:18:16 +00:00
Vincent Ambo
51594b0594 chore(cheddar): Bump all dependencies
Manual updates:
  comrak 0.6 -> 0.7
  syntect 3.3 -> 4.2.0

Leading to these automatic updates:
    Updating crates.io index
    Updating adler32 v1.0.4 -> v1.1.0
      Adding autocfg v1.0.0
    Updating base64 v0.10.1 -> v0.12.2
    Removing bindgen v0.53.3
    Removing cexpr v0.4.0
      Adding chrono v0.4.11
    Removing clang-sys v0.29.3
    Updating comrak v0.6.2 -> v0.7.0
    Removing env_logger v0.7.1
    Removing glob v0.3.0
    Updating hermit-abi v0.1.13 -> v0.1.14
    Removing humantime v1.3.0
      Adding indexmap v1.4.0
    Updating itoa v0.4.5 -> v0.4.6
    Updating libc v0.2.70 -> v0.2.71
    Removing libloading v0.5.2
    Removing log v0.4.8
    Updating miniz_oxide v0.3.6 -> v0.3.7
    Removing nom v5.1.1
      Adding num-integer v0.1.43
      Adding num-traits v0.2.12
    Updating onig v5.0.0 -> v6.0.0
    Removing peeking_take_while v0.1.2
    Updating plist v0.4.2 -> v1.0.0
    Updating proc-macro2 v1.0.17 -> v1.0.18
    Removing quick-error v1.2.3
    Updating quote v1.0.6 -> v1.0.7
    Updating regex v1.3.7 -> v1.3.9
    Updating regex-syntax v0.6.17 -> v0.6.18
    Removing rustc-hash v1.1.0
    Updating ryu v1.0.4 -> v1.0.5
    Updating serde v1.0.110 -> v1.0.112
    Updating serde_derive v1.0.110 -> v1.0.112
    Updating serde_json v1.0.53 -> v1.0.55
    Removing shlex v0.1.1
    Updating syn v1.0.25 -> v1.0.31
    Updating syntect v3.3.0 -> v4.2.0
    Removing termcolor v1.1.0
    Removing version_check v0.9.2
    Removing which v3.1.1
    Updating yaml-rust v0.4.3 -> v0.4.4

Change-Id: I2c2ee6a45bf7a5286fe30d61de52d7993871e7e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/444
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-16 23:59:48 +00:00
Vincent Ambo
9885b70b7a feat(cheddar): Override syntax highlighting for 'rules.pl' to Prolog
Adds a mechanism for per-filename overrides of the chosen language
syntax and configures it for Gerrit's submit rule file.

This also switches the syntax set used to the one from
//third_party/bat_syntaxes, which contains custom additions such as
Prolog support.

Change-Id: I2023dbad5b326305ef2ef0ecf34ef66a3f7575ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/349
Reviewed-by: riking <rikingcoding@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-15 16:52:27 +00:00
Vincent Ambo
6f8fbf4aa4 chore(tools/cheddar): Bump dependencies & upgrade to LLVM 10 2020-05-25 23:37:39 +01:00
Vincent Ambo
4bbbb58cb5 chore: Rename pkgs->depot in all Nix file headers 2020-02-21 13:54:53 +00:00
Vincent Ambo
14b5447aac Merge branch 'feat/cheddar-extensions' 2020-02-09 12:07:10 +00:00
Vincent Ambo
d752cbecc0 feat(cheddar): Add toggle flag for about-filter behaviour
Cheddar now needs to be passed the --about-filter flag to toggle the
behaviour for rendering Markdown into HTML.

By default Markdown will be highlighted like normal source code (i.e.
cgit source-filtering is the default behaviour).
2020-01-19 23:54:30 +00:00
Vincent Ambo
2236d43ff7 feat(cheddar): Implement callout paragraphs
Implements support for tagging paragraphs that begin with a callout
word (TODO, WARNING, QUESTION, TIP) with an additional `cheddar-*`
class that makes it possible to render these callouts specially.

This is currently not the nicest implementation, but it works.
2020-01-11 05:06:36 +00:00
Vincent Ambo
d4e469508f refactor(cheddar): Extract code block highlighting into function
Since I am going down the path of adding additional Markdown
extensions it makes sense to avoid letting `format_markdown` turn into
a giant beast of a function.

Therefore this commit extracts the logic for rendering code blocks via
syntect and changes the innards of `format_markdown` to instead
provide arbitrary AST value replacements.
2020-01-11 02:17:19 +00:00
Vincent Ambo
a36e9f817d fix(cheddar): Fix errors if filename does not have an extension 2019-12-22 15:21:23 +00:00
Vincent Ambo
03acae2f85 feat(cheddar): Style pre-elements for better readability
Uses GitHub-like styling for <pre> elements, i.e. slight padding and
background colour highlighting.
2019-12-21 16:18:26 +00:00
Vincent Ambo
f904951384 feat(cheddar): Render code blocks in Markdown via syntect
Implements fully static (i.e. no JavaScript!) highlighting of code
blocks when rendering Markdown.

This works by walking through the Comrak AST and replacing any code
blocks with pre-rendered HTML blocks.

Syntaxes are chosen based on the "block info", which is the string
users put after the block's opening fence. This can either be
a (case-insensitive) name of a syntax, or alternatively a file
extension associated with the desired syntax.

The theme is set to one that imitates GitHub.
2019-12-21 15:57:02 +00:00
Vincent Ambo
671dbc7f72 docs(cheddar): Mention cgit filter compatibility in README 2019-12-21 15:56:44 +00:00
Vincent Ambo
d6c255a35d feat(cheddar): Render Markdown via Comrak
Renders any ".md" file by pushing it through the Comrak rendering
pipeline.

This does not yet implement syntax highlighting of fenced blocks, but
we're getting there.
2019-12-21 14:49:35 +00:00
Vincent Ambo
d35aa4ae46 refactor(cheddar): Set up scaffolding for Markdown rendering
Generalises the two bits of the program that will be required either
way (extension parsing and syntax loading).

A dependency on Comrak is introduced as I think GitHub-flavoured
Markdown (with all its fancy extensions) is desirable!
2019-12-21 14:09:12 +00:00
Vincent Ambo
81d9b81b06 feat(cheddar): Use syntax highlighting assets from bat
This uses Nix to inject the path to the syntax highlighting assets
that ship with the bat source code into the cheddar build at compile
time, where the Rust compiler then inserts it into the binary via
macros.

bat has a lot of custom syntax highlighting definitions that they
collected from all over the place (including for languages like Nix!)
and this makes them accessible to cheddar.

Also if you're reading this, can you just take a moment to appreciate
how incredible it is that Nix just lets us do something like this?!
2019-12-21 04:55:10 +00:00
Vincent Ambo
4681c07bde feat(tools/cheddar): Check in new syntax highlighting tool
The first step with this tool will be to use it as a source-filter for
cgit. The second step is to use it as the Markdown renderer by
depending on one of the Markdown libraries, with integration for
rendering code snippets directly.
2019-12-21 04:54:57 +00:00