UnicodeDecodeError or SystemError when using f-string with lambda and non-ASCII characters

[gaesa]

Bug report

Bug description:

When using an f-string in combination with lambda functions that return non-ASCII characters, either a UnicodeDecodeError or a SystemError is raised, depending on the specific modification made to the code. These errors do not occur consistently, but rather vary based on how the lambdas and string content are altered.

Steps to Reproduce

def test1(foo, bar):
    return ""

def test2():
    return f"{test1(
        foo=lambda: '、、、、、、、、、、、、、、、、、',
        bar=lambda: 'abcdefghijklmnopqrstuvwxyz 123456789 123456789',
    )}"

Run the code with python <file.py>, which triggers the following error:

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 28-29: unexpected end of data

Expected Behavior

The code should execute without raising any errors.

Actual Behavior

The following behaviors are observed when making the specified modifications. Each of these cases is independent:

1. Removing one 、 character from the first string (foo): No error.
2. Removing all 、 characters from the first string: SystemError: Negative size passed to PyUnicode_New.
3. Removing foo= (i.e., not passing foo by keyword): No error.
4. Removing lambda: (i.e., making either argument or both into a str type instead of a Callable[[], str]): No error.
5. Removing any character from the second string (bar): No error.

Other Relevant Information

The bug does not reproduce in Python 3.13.1, but it does reproduce in Python 3.13.2 and 3.12.9.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-130618: Fix parser error when using lambdas inside f-strings #130638
• [3.13] gh-130618: Fix parser error when using lambdas inside f-strings (GH-130638) #130642
• [3.12] gh-130618: Fix parser error when using lambdas inside f-strings (GH-130638) #130644

[vstinner]

It seems to be a parser bug.

cc @lysnikolaou @pablogsal

[pablogsal]

On it

[pablogsal]

The problem was introduced by:

❯ git bisect bad
a3797492179c249417a06d2499a7d535d453ac2c is the first bad commit
commit a3797492179c249417a06d2499a7d535d453ac2c (HEAD)
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date:   Wed Jan 22 01:50:22 2025 +0100

    [3.13] gh-129093: Fix f-string debug text sometimes getting cut off when expression contains `!` (GH-129159) (#129163)

    gh-129093: Fix f-string debug text sometimes getting cut off when expression contains `!` (GH-129159)
    (cherry picked from commit 767cf708449fbf13826d379ecef64af97d779510)

    Co-authored-by: Tomas R <tomas.roun8@gmail.com>

 Lib/test/test_fstring.py                                                          | 18 ++++++++++++++++++
 Misc/NEWS.d/next/Core_and_Builtins/2025-01-21-23-35-41.gh-issue-129093.0rvETC.rst |  2 ++
 Parser/lexer/lexer.c                                                              |  4 +---
 3 files changed, 21 insertions(+), 3 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2025-01-21-23-35-41.gh-issue-129093.0rvETC.rst