Remark vs MarkdownIt

[Saul-Mirone]

Hi benrbray,

I'm really curious about the reason why you switch the markdown parser to remark from markdown-it.

The reason why I ask this is that I'm writing a markdown editor: milkdown these days and I use markdown-it, it works well. However I heard about remark and I'm comparing them. I think our projects have something in common so I think your opinion well be very helpful.

[benrbray]

Hi, Saul-Mirone, milkdown looks great!

I chose markdown-it at first since it seems to be the de-facto standard for Markdown syntax extensions. So, switching to remark was a big deal, and it was a massive undertaking that involved rewriting a lot of the Noteworthy backend.

Noteworthy originally used Marijn's proof-of-concept markdown conversion code to convert the markdown-it token stream to a ProseMirror document. I hacked on a few extra features but I knew I'd have to eventually rewrite it from scratch, which gave me an opportunity to evaluate if markdown-it was the best choice.

The main reasons I switched:

• markdown-it prioritizes HTML output, while remark has first-class support for abstract syntax trees.

  • Having clean ASTs to work with on the backend makes it much simpler to examine / transform Markdown documents in various ways, e.g.
    • convert a markdown AST to/from a ProseMirror document
    • plugins to Noteworthy can define a pair of tree transformations for converting between AST/ProseMirror
    • Noteworthy watches your workspace for changes, and maintains an index for things like wikilinks/citations. Using remark makes it easy to parse the documents behind-the-scenes and extract information from the AST.
  • I considered writing a library to convert the markdown-it token stream to/from an AST, but decided against it, since many existing syntax extensions don't produce tokens that would be easy to consume. Some of them go direct to HTML without even generating tokens!

• I found the syntax-extension authoring experience for markdown-it to be too confusing / complicated.

  • The plugin process isn't very well-specified, so existing syntax plugins are all quite different in their implementation.
  • I wrote a few of my own markdown-it syntax extensions (either from scratch, or based on an existing plugin). Maybe it's just me, but I found them hard to modify and maintain. I dreaded having to touch those files.
  • By comparison, remark has a bit of a steep learning curve (mostly due to lack of documentation), but once I learned how, I found it really easy to spin up new syntax extensions as I needed them!

• remark has a very well-specified process for converting Markdown -> AST -> HTML

  • micromark tokenizes the input
  • mdast converts the token stream to an abstract syntax tree and back
  • remark packages everything up nicely as a library
  • rehype converts ast to html
  • all existing syntax extensions support all three levels of abstraction, so I can pick and choose!

There are a few cons, but so far I'm much happier using remark. I was able to add

• Remark is relatively young, so the API is a bit unstable and there is not a lot of documentation. However, the maintainer is quite responsive to questions / issues on GitHub. By comparison, the markdown-it maintainer has been actively hostile to users on the issues page, which is sad to see.
• Writing syntax extensions involves manually writing a state machine. I suspect this can be improved with some wrapper libraries that generate state machine instructions from extension code written in a parser-combinator style.
• The type annotations for remark leave something to be desired. But the same was true of many of the markdown-it plugins I used, so I found myself wrestling with the same issues.

What goals do you have for milkdown? Here's my Noteworthy "manifesto", for comparison.

[Saul-Mirone]

There are mainly 2 goals for milkdown I think:

1. Free, open source, easy to be integrated and can be used anywhere.
2. Provide a WYSIWYG markdown editor that can be used both out of box and customized by plugin and theme.

[Saul-Mirone]

Milkdown has been migrated to remark too👻

[benrbray]

Taking a look at your docs, I think we share the common goal of being plugin-driven, and it looks like you're quite far ahead of me with plugin support :). It'll be fun to dive into some of your plugin code to see how it works, and I'd be interested to hear about any lessons learned setting up your plugin system.

Noteworthy currently takes inspiration from remirror's plugin system for defining schema extensions, and for Markdown parser extensions I let plugins define transformations between the mdast and ProseMirror tree formats.

[Saul-Mirone]

In fact milkdown take everything as an Atom.
So a prosemirror node as actually an atom. The core part of milkdown just load the atom one by one, and I provide some internal atoms such as parser and serializer, and I also provide some helpers for users to quickly create their own atoms, such as createProsemirrorPlugin.

All atoms will share a common context, they can read and write properties on it. So as you can imagine, a Paragraph atom will add a node to node list on context. And later, a ProsemirrorSchema atom will use the node list to create a prosemirror schema. This sequence is controlled by LoadState, which has a more clear description in this document page.

[Saul-Mirone]

🍼Check our latest Milkdown document to know how to build your own custom syntax:

https://saul-mirone.github.io/milkdown/#/example-custom-syntax