gh-130587: Add hand-written docs for non-OP tokens

[encukou]

• Add hand-written docs for non-OP tokens

• Make the automation (generate_token.py) check that the hand-written docs are present, and only generate docs for the OP tokens

• Switch to list-table for the OP tokens, to make their docs more compact

• Add ENDMARKER to the grammar docs where it appears (toplevel components)

• Add forgotten versionchanged entry for EXCLAMATION

• Remove docs for NT_OFFSET

• Issue: token module documentation is incomplete #130587

---

📚 Documentation preview 📚: https://cpython-previews--130588.org.readthedocs.build/

[encukou]

The rendered docs are at https://cpython-previews--130588.org.readthedocs.build/en/130588/library/token.html

[encukou]

@lysnikolaou, does this look reasonable to you?

[AA-Turner]

Suggested change

	A generic token value returned by the :mod:`tokenize` module for
	:ref:`operators <operators>` and :ref:`delimiters <delimiters>`.
	A generic token value that indicates an
	:ref:`operator <operators>` or :ref:`delimiter <delimiters>`.

[lysnikolaou]

Let's include the information that this is only returned by the tokenize module. It's not used internally in the tokenizer at all. Maybe add it as an implementation detail?

[encukou]

The tokenize docs have some more information; how much should be repeated here?

[lysnikolaou]

Maybe add an .. impl-detail:: like you did for ENCODING?

[AA-Turner]

Does compile() with the AST flag not produce them?

[lysnikolaou]

I think it does.

[AA-Turner]

I'm not sure how these changes are related to this PR?

[encukou]

The ENDMARKER docs now refer here, and it would be curious if the marker is missing.

The actual gramar does use ENDMARKER.

[AA-Turner]

I think we can avoid re:

Suggested change

	tokendef_re = re.compile(r'.. data:: (\w+)')
	for line in fileobj:
	if match := tokendef_re.fullmatch(line.strip()):
	if match[1].isupper():
	has_handwritten_doc.add(match[1])
	for line in fileobj:
	if not line.startswith('.. data:: '):
	continue
	tok_name = line.removeprefix('.. data:: ').rstrip()
	if tok_name.isidentifier() and tok_name.isupper():
	has_handwritten_doc.add(tok_name)

[encukou]

Yes, but at the cost of duplicating the prefix. Is that a good tradeoff?

[AA-Turner]

Perhaps, though clarity may be lost in removing duplication:

Suggested change

	tokendef_re = re.compile(r'.. data:: (\w+)')
	for line in fileobj:
	if match := tokendef_re.fullmatch(line.strip()):
	if match[1].isupper():
	has_handwritten_doc.add(match[1])
	for line in map(str.rstrip, fileobj):
	name = line.removeprefix('.. data:: ')
	# Token names must be uppercase and made up of alphanumerics or '_'
	if line != name and name.isidentifier() and name.isupper():
	has_handwritten_doc.add(name)

[lysnikolaou]

This looks like a good improvement to me! Thanks @encukou!

I've left some inline comments regarding some specifics in the docs.

[lysnikolaou]

Let's include the information that this is only returned by the tokenize module. It's not used internally in the tokenizer at all. Maybe add it as an implementation detail?

[lysnikolaou]

I think it does.

[encukou]

Thank you for the reviews! I addressed some; I'll continue next week.