Improve Python syntax highlighting

[fabiospampinato]

monaco-editor version: 0.19.2
Browser: Chrome 79 maybe
OS: macOS Mojave
Playground code that reproduces the issue:

monaco.editor.create(document.getElementById("container"), {
	value: "from functions import sum, average",
	language: "python"
});

Basically some import statements in Python aren't highlighted properly.

How Monaco/Monarch highlights things:

How VSCode highlights things:

[alexdima]

PR welcome. The Python tokenizer is here.

[blois]

Part of the issue is that Monaco's default theme differs from VSCode's Dark+ and Light+.

It would be great if these could be reconciled to make it possible to implement Monarch tokenization that follows the same TextMate patterns.

For example, if you try to update the Python tokenizer to support themes such as https://github.com/brijeshb42/monaco-themes/blob/master/themes/Monokai.json then it will break with the default Monaco themes.

[fabiospampinato]

make it possible to implement Monarch tokenization that follows the same TextMate patterns.

My understanding is that that's not possible, at least not without sacrificing performance or platforms where WASM isn't available, TextMate uses a different regex engine and its regexes can't just be converted to JS regexes.

[blois]

It should be possible to implement a Monarch tokenizer which results in a token stream similar to what the TextMate grammars result in. Right now implementing a Monarch tokenizer which approximates TextMate will result in a loss of syntax highlighting as Monaco's default theme does not support all of the tokens.

I did extend the Monarch Python syntax highlighting to be closer to TextMate, but this required a number of customizations to the theme as well, would be great if I could make a PR to monaco-languages without it requiring theme customizations.

Python editor with my customizations: https://colab.research.google.com/gist/blois/00fa28cdb0433e62c8106a4b5812d024/syntax-highlighting.ipynb

[bolinfest]

More importantly, this seems to be missing Python 3 features like recognizing async/await as keywords and supporting f-strings.

[samstrohkorbatt]

@hediet, @bpasero, @alexdima
Could any of you approve my PR to add Python F-String syntax highlighting? This is the second half of what @bolinfest asked for over 4 years ago that still isn't in Monaco Editor yet

[saiyalamarty]

Is it expected that the f-string support that was merged works in v0.49.0-dev-20240524 but not in v0.49.0 or v0.49.0-rc?