Skip to content

Latest commit

 

History

History

Rouge

README.md

Rouge Tests

Testing how to employ custom Rouge lexers in Asciidoctor projects.

Table of Contents

Directory Contents

  • /sass/ — CUstom Sass/CSS for Asciidoctor.
  • /themes/ — themes references and work.
  • sample.alan — sample Alan source (UTF-8 + BOM).

Custom Ruby scripts, lexers and libraries:

  • alan3.rb — custom Rouge lexer for Alan (WIP, developed here)
  • custom-rouge-adapter.rb — tweaks the Rouge adapter for Asciidoctor that loads (requires) our alan3.rb lexer.

Tests:

Sample documents:

Objectives

In order to integrate the Rouge highlighter with our ALAN Docs toolchain, we'll need to enforce our custom Rouge lexers on Asciidoctor, and provide custom themes for each language:

  • Bypass the native Asciidoctor Rouge API with a custom adapter, which instructs Rouge on where to find and load the custom lexer definitions.
  • Define custom themes on a per language basis:
    • HTML Backend
      • Sass/CSS — for the HTML backend.
      • A native Rouge theme can still be specified via the :rouge-style: attribute (in doc header or via CLI opts) which will be converted to CSS as part of the default Asciidoctor CSS, this theme will be used as a fallback theme for code snippets in languages not covered by our custom CSS.
    • PDF Backend
  • Invoke Asciidoctor via its Ruby API, which offers a higher degree of control in terms of options when interacting with the Rouge library, replacing Bash scripts with a Rake build system in our toolchain.

In this directory we'll be testing various approaches and solutions, to see which one works better for us, and so that we may weigh the pros and cons of each.

Overview

Rouge is a Ruby syntax highlighter which is natively supported by Asciidoctor in various backends. Our goal would be to employ Rouge for the HTML backend, instead of Highlight, due to lack of callouts support with the latter (see alan-docs#107 and alan-docs#36).

We're also planning to migrate from asciidoctor-fopub to asciidoctor-pdf for the PDF backend, once the latter is more mature for our needs (v2.0). When this will happen, Rouge could be used for syntax highlighting PDF documents too, which lessens our dependencies and maintenance work considerably, beside ensuring stylistic consistency across the various documents formats.

We'd also like to be able to use our own Rouge lexers (i.e. syntax definitions), tailored to our needs, instead of having to submit them to the upstream Rouge repository in order for Asciidoctor to be able to use them via the Rouge gem. E.g. we might need some in-house custom lexers for BNF, ALAN transcripts, etc., none of which is likely to qualify for integration in the official Rouge gem.

Furthermore, we'd like to bypass Asciidoctor's inclusion of Rouge themes in the generated HTML, and use our custom CSS stylesheets for colouring code blocks, in order to provide different themes for each language, as well as alternative themes for Alan, based on the block's role.

The ALAN Lexer

Initially I started to work on the ALAN lexer (alan3.rb) elsewhere, in my own fork of the Rouge repository. I've now started to develop the syntax here, because the current test toolchain found in this folder is simpler to use, and I can check the result using a custom theme (themes/alan-b16-eighties.rb), which also simplifies tests since it colours some token classes not covered by Rouge's default themes, but required for ALAN.

The lexer filename (alan3.rb) and its associated syntax name (alan3) and aliases (alan-if and alan) are all temporary and might change in the final version. If we're going to submit the lexer to the Rouge gem repository, we'll have to drop the alan alias and adopt alan-if as the syntax name instead, to avoid clashes with the Alan programming language developed by Alan Technologies, Inc (see alantech/alan#548).

The problem is that currently all the documents in the ALAN Docs and ALAN StdLib projects (and others) are using alan as the source syntax name, so we'd have to change all its occurrences in the Asciidoctor sources, and also change the syntax name in the other highlighters too (e.g. the one used for the asciidoctor-fopub PDF backend, which we'll be still using even after switching to Rouge, at least until we drop it in favour of the asciidoctor-pdf backend).

We can always keep alan3 as an alias, in case in the future we implement a lexer for ALAN 2.

As for the lexer features, I'm planning to use the full power of Rouge and Ruby to create a semantically detailed lexer, which will match tokens as accurately as possible.

For our use in the ALAN Docs project, we'll probably end up hiding all these semantic details by assigning the same colour to related tokens, in order to keep the syntax appearance simpler for the reader.

Tech Info

Some info on the nitty gritty technicalities surrounding Rouge, the Asciidoctor API and their usage.

Custom Lexers with Rougify via CLI

  • rougify-term.sh — highlights sample.alan in the terminal, via CLI, using our custom lexer.

Assuming our lexer is called alan3.rb, to highlight the sample.alan file from the command line we only need to use the --require/-r option:

To highlight an Alan source file using a custom Rouge lexer from the command line, we only need to use the --require/-r option:

$ rougify sample.alan --require ./alan3.rb

From rougify help:

$ rougify help highlight
usage: rougify highlight <filename> [options...]
       rougify highlight [options...]

[...]

--require|-r <filename>     require a filename or library before
                            highlighting

Custom Lexers via Rouge API

I'm still not entirely sure how to instruct the Rouge API to require a custom lexer, but here's how the command line parameter --require is being handled by lib/rouge/cli.rb (L235):

          when '-r', '--require'
            opts[:requires] << argv.shift

Custom Lexers with Asciidoctor

Thanks to Dan Allen (@mojavelinux) for helping us out with the solution on how to make Rouge require a custom lexer.

  • Create the file custom-rouge-adapter.rb: 
    require 'rouge'
require './alan3.rb'

class CustomRougeAdapter < (Asciidoctor::SyntaxHighlighter.for 'rouge')
  register_for 'rouge'
end
  • Invoke Asciidoctor with -r ./custom-rouge-adapter.rb.

An alternative code for custom-rouge-adapter.rb, defers loading Rouge until the load_library method is called:

class CustomRougeAdapter < (Asciidoctor::SyntaxHighlighter.for 'rouge')
  register_for 'rouge'

  def load_library
    require 'rouge'
    require './alan3.rb'
    :loaded
  end
end

They both produce equal results for our scope.

Acknowledgements

We'd like to thank Dan Allen (@mojavelinux) from the Asciidoctor Project for having helped us out with the solution on how to make Rouge require a custom lexer:

Links

Rouge

Asciidoctor