Better syntax highlighting: switching from highlight.js to shiki

[julio4]

Problem

The current highlighting library highlight.js is lightweight and can run on both server and client, but provides a very basic syntax parsing that could be greatly improved.

Supersede #2313 #2292 #2268 #2238 #2237 #2360 #2445 #1652 ...

Proposed Solution

Consider switching to shiki, a high quality syntax highlighter that uses TextMate grammars and themes (same as VS Code), yet stays performant. It is also actively maintained.

There have been significant efforts and improvements that led to shiki-v1, and I believe that it's worthwile to make this migration and make shiki the default highlighter for mdbook. Today, mdbook is used to document a lot of code snippets and this change would definitely improve readability.

Example

Let's take Listing 20-24 from the rust book:

impl Worker {
    fn new(id: usize, receiver: Arc<Mutex<mpsc::Receiver<Job>>>) -> Worker {
        let thread = thread::spawn(move || loop {
            let message = receiver.lock().unwrap().recv();

            match message {
                Ok(job) => {
                    println!("Worker {id} got a job; executing.");

                    job();
                }
                Err(_) => {
                    println!("Worker {id} disconnected; shutting down.");
                    break;
                }
            }
        });

        Worker {
            id,
            thread: Some(thread),
        }
    }
}

mdbook w/ highlight.js (Coal theme)	Shiki (github-dark-dimmed theme)
Open in browser	Open in browser

Vs-code reference (theme Github Dark Dimmed):

Implementation example

@cestef overriding of theme/book.js for shiki compatibility as explained here

Notes

Shiki v1: https://nuxt.com/blog/shiki-v1
Another alternative: https://github.com/wooorm/starry-night

Further ideas

Twoslash allows to extend Typescript code snippets from markdown file with extended Typescript compiler's information and vs code syntax engine. It is maintained and integrated perfectly with shiki.

With the strong rust-analyzer and the recent abstraction of the twoslash API in twoslash-protocol, it would make sense to build a twoslash-rs for mdbook. Some preliminary explorations have been done: by ayazhafiz/twoslash-rust and @jxom

This first implementation of Shiki in mdbook would pave the way for this later on.

[julio4]

@ehuss Any update on this? Would this be something that you would consider?

[ehuss]

We are likely going to switch to build-side rendering with syntect (#1652), so probably not going to add any other systems.

[julio4]

We are likely going to switch to build-side rendering with syntect (#1652), so probably not going to add any other systems.

Looks really nice but the issue seems to be stale, so not sure if it can be included soon... It's been a pain for years to manually provide custom highlight.js file for unsupported languages.

Also, shiki provide build-side rendering, with the possibility of exploring further ideas as I mentioned in the issue and with better syntax highlighting as, IMHO, TextMate grammars are slightly better than Sublime Text syntax with oniguruma.
Won't be in rust but build time overhead is not significant. So I would still like to consider Shiki if possible