Surrogates in UTF-8 encoding causes an unhandled exception in [p]debug

[Kreusada]

What Red version are you using?

3.5.13

Cog name

Dev

Command name

debug

What did you expect to happen?

A traceback to be gracefully sent as exception detail.

What actually happened?

The command itself errored:

[15:55:34] ERROR    [red] Exception in command 'debug'
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 6-7: surrogates not allowed

The above exception was the direct cause of the following exception:

╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮
│ C:\Users\Kreusada\redenv\Lib\site-packages\discord\ext\commands\core.py:235 in wrapped                                   │
│ ❱  235             ret = await coro(*args, **kwargs)                                                                 │
│ C:\Users\Kreusada\redenv\Lib\site-packages\redbot\core\dev_commands.py:486 in debug                                      │
│ ❱ 486         await output.send()                                                                                    │
│ C:\Users\Kreusada\redenv\Lib\site-packages\redbot\core\dev_commands.py:167 in send                                       │
│ ❱ 167         await self.ctx.send_interactive(get_pages(str(self)), box_lang="py")                                   │
│ C:\Users\Kreusada\redenv\Lib\site-packages\redbot\core\commands\context.py:183 in send_interactive                       │
│ ❱ 183         return await self.bot.send_interactive(                                                                │
│ C:\Users\Kreusada\redenv\Lib\site-packages\redbot\core\bot.py:2377 in send_interactive                                   │
│ ❱ 2377                 msg = await channel.send(box(page, lang=box_lang))                                            │
│ C:\Users\Kreusada\redenv\Lib\site-packages\discord\abc.py:1618 in send                                                   │
│ ❱ 1618             data = await state.http.send_message(channel.id, params=params)                                   │
│ C:\Users\Kreusada\redenv\Lib\site-packages\discord\http.py:600 in request                                                │
│ ❱  600             kwargs['data'] = utils._to_json(kwargs.pop('json'))                                               │
│ C:\Users\Kreusada\redenv\Lib\site-packages\discord\utils.py:644 in _to_json                                              │
│ ❱  644         return orjson.dumps(obj).decode('utf-8')                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: str is not valid UTF-8: surrogates not allowed

How can we reproduce this error?

1. Load a Red-DiscordBot instance with the --dev flag.
2. Run [p]debug "\uD800" in any location

Anything else?

While the traceback suggests that this error can occur more broadly through Context.send, the root cause in that case would lie with the developer, as surrogate characters are prohibited in UTF-8 encoding (they are designed for UTF-16). For debugging purposes, I feel it would be necessary for the exception to be included in the debug output rather than being raised, so that the command doesn't fail. This would allow this invalid debugging statement to be gracefully evaluated. This exception may be reproduced for any surrogate characters (unicode code points beyond the basic multilingual plane).

Possible workarounds could include pre-validation/sanitization of the raw source, probably best fitted inside dev_commands.sanitize_output.