token module documentation is incomplete

[encukou]

Documentation

The token docs use a generated list of tokens, with short blurbs for the OP tokens but nothing for the other (more interesting) ones.
Some tokens (COMMENT, NL, etc.) have secondary, hand-written definitions. However, these are excluded from the index; links go to the empty entries.

I propose adding hand-written prose for the non-OP tokens, and changing the automation to check the entries rather than generate them.
For the OP tokens, generating the docs is the correct thing to do, but the list can be more compact since they're all the same.

Linked PRs

• gh-130587: Add hand-written docs for non-OP tokens #130588

[StanFromIreland]

Also another issue in the file:

The '/' are not escaped.