Async processing of markdown

[ronald-mz]

Goals

1. allow for in-browser async processing of markdown files into mdast by composing micromark exports and syntax-tree exports
2. allow flexibility in how markdown content is processed

Non-Goals

1. interception and manipulation of data between calls to preprocessor, parser, postprocessor, and compile

How

1. by exporting the compiler in mdast-util-from-markdown

Why

Both micromark and mdast-util-from-markdown parse the markdown file in one go. micromark exports a streaming option but that utilizes node duplex streams and only converts markdown to html rather than mdast.

By exporting the compiler from mdast-util-from-markdown, devs will be able to control the steps and timings of the processing.

Event Loop

Yielding to the event loop allows for the render process to proceed, garbage collection to occur (minor and major), and other events to occur.

Parity with micromark

micromark exports its compiler but mdast-util-from-markdown does not

Alternatives Considered

The usual suggestion is to parse markdown in a separate thread. This produces other problems and doesn't address others.

Even if the parser is in another thread.

• We still aren't allowing the garbage collector to run outside of an expensive runtime 'stop the world' to clean up.
• Serializing objects between threads is expensive.
• We'd need to open up another worker to process more than one markdown file at a time.

Related Prior Discussion

syntax-tree/mdast-util-from-markdown#29 (different use case, but same result)

PS

The unified ecosystem is amazing. Thank you all for your dedication and work to make, maintain, responding to discussions, and dealing with issues.

[wooorm]

Heya! Thanks for your kind words :)

The thing with markdown (and HTML) is that it actually cannot stream. Definitions can exist at the end of a document. Whether they exist or not affects how earlier markdown is parsed. Whether something is a link or not affects more than just emitting <a> and </a>:

![[*x*][]](asd)
![[*y*][]](asd)

[*x*]: https://example.com

Yields:

<p><img src="asd" alt="x" />
<img src="asd" alt="[y][]" /></p>

Yielding to the event loop allows for the render process to proceed, garbage collection to occur (minor and major), and other events to occur.

I am not sure that it is useful to control this. The alternative threads/workers, seems like a better abstraction level, that is already able to control this.

micromark exports its compiler but mdast-util-from-markdown does not

That is because ASTs allow arbitrary processing. Which is very easy when things are in their entirety as JSON. And very complex otherwise. The entire ecosystem is built around whole ASTs. It would mean none of the utilities or plugins work. And there’s no standard for streaming ASTs?

We still aren't allowing the garbage collector to run outside of an expensive runtime 'stop the world' to clean up.

I am under the impression that GC always works like that?

Serializing objects between threads is expensive.

I mean, there are costs with everything. Expensive is subjective. A whole pause/resume process seems expensive too?

We'd need to open up another worker to process more than one markdown file at a time.

I think you can send things to and from workers?

(different use case, but same result)

What is the use case? Why do you want this?

[ronald-mz]

Heya! Thanks for your kind words :)

The thing with markdown (and HTML) is that it actually cannot stream. Definitions can exist at the end of a document. Whether they exist or not affects how earlier markdown is parsed. Whether something is a link or not affects more than just emitting <a> and </a>:

There's a small misunderstanding. I do not want to stream through the entire process. I know from prior discussions that making the AST requires the full context.

What I'd like is similar to the streaming API of micromark. Stream into a buffer, then create the AST from the buffer. But, remark has no streaming API nor does it export the compiler to make the AST like micromark does. So I can't make my own streaming API with the library (unless I pull the code out wholesale, but I'd prefer to avoid doing that if I can)

Yielding to the event loop allows for the render process to proceed, garbage collection to occur (minor and major), and other events to occur.

I am not sure that it is useful to control this. The alternative threads/workers, seems like a better abstraction level, that is already able to control this.

It is if you want to avoid majorgcs. Even in an alternative thread, we hit them since we aren't letting go of the thread while building the AST.

micromark exports its compiler but mdast-util-from-markdown does not

That is because ASTs allow arbitrary processing. Which is very easy when things are in their entirety as JSON. And very complex otherwise. The entire ecosystem is built around whole ASTs. It would mean none of the utilities or plugins work. And there’s no standard for streaming ASTs?

I think my clarification at the beginning addresses this portion

We still aren't allowing the garbage collector to run outside of an expensive runtime 'stop the world' to clean up.

I am under the impression that GC always works like that?

The garbage collector only stops the world when it runs out of memory. Otherwise it it goes through phases/generations.

Nursery -> Intermediate -> Old

In an over simplification, it is faster to reclaim objects from the Nursery than from the Old generation.

Serializing objects between threads is expensive.

I mean, there are costs with everything. Expensive is subjective. A whole pause/resume process seems expensive too?

A pause/resume process would take longer in real time, but use less memory and CPU time.

We'd need to open up another worker to process more than one markdown file at a time.

I think you can send things to and from workers?

One worker would not be able to share CPU time to create an AST. I could send another file to it, but until the first is done the second would wait.

(different use case, but same result)

What is the use case? Why do you want this?

Sharing markdown between users in a P2P web context, viewing them as HTML rather than text.

Utilizing unified, remark, and rehype plugins to manipulate the tree after it's created.

[wooorm]

That is part of what I'm looking for. The other part sits between the tokenizer and compiler in micromark.

Why not investigate that first? That’s where I’d estimate 96% of your win is. And it’s already possible.

Here is where I'd ideally like to do an interrupt to bail out.

Why only pause there? Why not earlier or later?

It seems to me like you are asking about a solution (Y) where there is a different problem (X).

If the compiler were exported from mdast-util-from-markdown, you could introduce a bail out anywhere in this process or even mid-way through the preprocess or parse steps of micromark. Micromark's stream API supports this.

Have you tried what you want? Have you tested and measured it? I have strong doubts.

Consumers can't make their own since the compiler isn't exported from the util. (Unless they copy the compiler from the repository)

Hidden internals are hidden. I thought it would be good to architecture the parser in a streaming way, even though every steps buffers everything.

That's the case for small documents, but throttling won't work for documents as they grow to a certain size.

From what you are saying, it seems that you want to prevent superfluous work from being done when a new change happens.
Perhaps adding support for AbortController is an alternative solution to your problem.

[ronald-mz]

From what you are saying, it seems that you want to prevent superfluous work from being done when a new change happens.
Perhaps adding support for AbortController is an alternative solution to your problem.

An AbortController would be beneficial.

---

Incorrect Profiles

I went back and looked at the profiles I'd made. They were flawed. They included functions stripped out with micromark-build. In those profiles, consume is expensive. Without, it barely shows up in the profiler.

My apologies for the back and forth caused by this error and thank you all for the patience.

Conclusions

• An Abort Controller should handle my use case.
  • I am unsure where this discussion would occur, if any would need to involve y'all at all? Maybe in unified? But, this might be do-able with a plugin.
• Splicing takes longer the bigger the markdown.
  • I'll drop over to Performance improvement: linked lists for events micromark/micromark#23 to provide some time to help out.

[wooorm]

It could be in micromark, which has been your focus here, because you can already do each plugin manually if you want to.
It could also be in unified, then it would have to expose the controller to every transform and to parsers and compilers, which can then choose to support it or not.

Given that the previous benchmarks were incorrect, I do wonder whether it’s needed though?

Re the benchmarks, it seems that at 500x there is much less time spent on splicing. Perhaps that’s just v8 warming up.

Also, I do wonder whether these plugins represent your real-world case. remark-directive integrates with micromark and otherwise doesn’t transform, remark-directive-rehype
and remark-rehype map trees 1-to-1 to another, and rehype-stringify builds a string. Only rehype-format has to do more complex tree walking and mutation. Typically in my experience folks have more plugins like rehype-format in there!

[ronald-mz]

It could be in micromark, which has been your focus here, because you can already do each plugin manually if you want to. It could also be in unified, then it would have to expose the controller to every transform and to parsers and compilers, which can then choose to support it or not.

The way I imagined it is something like...

class UnifiedBail {
    #shouldBailCheck = null;

    constructor(shouldBailCheck) {
        this.#shouldBailCheck = shouldBailCheck;
    }

   // returns a unifiedjs plugin
    bailPoint() {
       if(shouldBailCheck()) {
         // bail out!
       }
    }
}

...

unified.use(remarkParse)
  .use(unifiedBail.bailPoint())
  .use(somePlugin)
  .use(unifiedBail.bailPoint())
  .use(someOtherPlugin)

I'd need to look closer at how Unified works though.

Given that the previous benchmarks were incorrect, I do wonder whether it’s needed though?

It might not be. I posted the flame charts from the correct benchmarks in a comment below. That splicing really eats into time and if brought under control, micromark's time spent would be negligible compared to plugins. Even on large markdown files.

Most of the time is spent in postprocess (splicing in the sub-tokenizer), and compiler (splicing for prepareList).

Re the benchmarks, it seems that at 500x there is much less time spent on splicing. Perhaps that’s just v8 warming up.

The x500 runs are using the regular readme.md while the x10 are using a readme.md repeated 50 times. Splicing the smaller event arrays do seem to have negligible impacts while splicing larger ones cause more and more delay.

Also, I do wonder whether these plugins represent your real-world case. remark-directive integrates with micromark and otherwise doesn’t transform, remark-directive-rehype and remark-rehype map trees 1-to-1 to another, and rehype-stringify builds a string. Only rehype-format has to do more complex tree walking and mutation. Typically in my experience folks have more plugins like rehype-format in there!

I'd usually have two others in there.

• A plugin that takes inspiration from Commonmark Markdown Forms, using block directives to enclose it.
• My own take on rehype-components, but lazily loads its components from html files. Parsing them into hast and dropping children into slots.

I'll run more profiles with those. The ones below were thrown together as a quick way to check on things.

For good measure, I'll also run a set of profiles with your preset and markdown dump. Though, I might brew some coffee for that. The ('./readme.md).repeat(50) takes awhile as is. I only run 10 times for that reason.

[wooorm]

I'd need to look closer at how Unified works though.

Hmm, this seems to be different from AbortController, and unrelated to parsing markdown?
Unified pipelines are for several files too. This seems to be for one.

And, if you want to create your own bail plugins like you show: you can. Nothing needs to change in unified for that.
As mentioned earlier: you can also already choose to stop after every plugin. Call .parse, then call run for each plugin. And then call stringify.

That splicing really eats into time and if brought under control, micromark's time spent would be negligible compared to plugins. Even on large markdown files.

Work being done that needs to be done is not nessecarily a problem? In my experience, which I think I mentioned from the start, is that parsing is indeed negligable compared to inspect/transform plugins.

Splicing the smaller event arrays do seem to have negligible impacts while splicing larger ones cause more and more delay.

Well that makes sense? Markdown is not a nice smart grammar that can be parsed linearly. And yes, merging many separate things into one giant thing is expensive.
Also seems like something V8 could improve?

The ones below were thrown together as a quick way to check on things.

I feel like I’m spending much time as a rubber duck here. Can you take some time learning the ecosystem and trying things out? Happy to discuss more actionable points! Thanks.

[ronald-mz]

In case y'all are interested.

Flame graphs

Remark - Basic (no plugins, readme.md) x500

Remark - Basic (no plugins, 50x readme.md) x10

Remark - Plugins (readme.md) x500

Remark - Plugins(50x readme.md) x10

Notes

Profiles made on Firefox

Noteable Plugins:

remark-directive
remark-directive-rehype
remark-rehype
rehype-document
rehype-format
rehype-stringify