Design lexical structure for documentation comments

[kritzcreek]

I don't think this is high priority.
The current implementation works well enough that we can document base and users can document their libraries, but it would be nice to do things "properly" at some point.
Right now the rules for what constitutes a documentation comment only exist as code inside mo_frontend/lexer.ml -> doc_comment_from_trivia.
Once we've come up with a coherent design we should implement it and document the rules in the language manual.

Current state and tricky issues

Currently documentation comments must start with /// followed by a space and text until the end of line.
An exception about the trailing space is made for /// without text after it. That constitutes a newline.

We also have /// @deprecated comments which use the same rules as above, and can be used to deprecate fields.

We also allow /** doc_comment */ but only strip leading and trailing whitespace for the whole multiline comment not for individual lines. There is no @deprecated equivalent for multi-line comments.
If we want multiline documentation comments to work well we'll have to come up with a ruleset for stripping indentation.

The comments contents are treated as Markdown for the Html output and passed through verbatim for the Asciidoc output.
This means we're trying to use the subset of Markdown that is also valid Asciidoc in the base library documentation.
Part of the reason for this being tricky is that I couldn't find a good Markdown, or any Asciidoc parser for OCaml.

[nomeata]

As for the markers, maybe worth doing what rust does, and hoping they thought a lot about it, so we don’t have to