Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

gh-122291: Intern latin-1 one-byte strings at startup #122303

Merged
merged 1 commit into from
Jul 27, 2024
Merged

gh-122291: Intern latin-1 one-byte strings at startup #122303

merged 1 commit into from
Jul 27, 2024

Conversation

encukou
Copy link
Member

@encukou encukou commented Jul 26, 2024 •
edited
Loading

Intern 1-character latin1 strings during startup.

This makes _PyRuntime.cached_objects.interned_strings immutable: it's not modified after Py_Initialize() until shutdown.

@bedevere-app bedevere-app bot mentioned this pull request Jul 26, 2024
Closed
colesbury
colesbury approved these changes Jul 26, 2024
View reviewed changes
Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Should this also be backported to 3.13?

@encukou encukou added the needs backport to 3.13 bugs and security fixes label Jul 27, 2024
@encukou encukou merged commit bb09ba6 into python:main Jul 27, 2024
40 checks passed
@miss-islington-app
Copy link

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 27, 2024
@encukou @miss-islington
pythongh-122291: Intern latin-1 one-byte strings at startup (pythonGH…
8b26a3b 
…-122303)

(cherry picked from commit bb09ba6)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
@bedevere-app
Copy link

bedevere-app bot commented Jul 27, 2024

GH-122347 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jul 27, 2024
@encukou
Copy link
Member Author

encukou commented Jul 27, 2024

Yes, it fixes a thread-safety bug.

@encukou encukou deleted the intern-ūnus branch July 27, 2024 08:34
encukou added a commit that referenced this pull request Jul 27, 2024
@miss-islington @encukou
[3.13] gh-122291: Intern latin-1 one-byte strings at startup (GH-122303…
4092510 
…) (GH-122347)

(cherry picked from commit bb09ba6)

Co-authored-by: Petr Viktorin <encukou@gmail.com>
@encukou encukou mentioned this pull request Jul 29, 2024
Closed
encukou added a commit to encukou/cpython that referenced this pull request Aug 16, 2024
@encukou
[3.12] pythongh-122291: Intern latin-1 one-byte strings at startup (p…
1d4a531 
…ythonGH-122303)
@encukou encukou mentioned this pull request Aug 16, 2024
Merged
@ZeroIntensity ZeroIntensity mentioned this pull request Aug 17, 2024
Closed
Yhg1s pushed a commit that referenced this pull request Sep 27, 2024
@encukou @ericsnowcurrently
[3.12] gh-113993: Make interned strings mortal (GH-120520, GH-121364, G…
49f6beb 
…H-121903, GH-122303) (#123065)

This backports several PRs for gh-113993, making interned strings mortal so they can be garbage-collected when no longer needed.

* Allow interned strings to be mortal, and fix related issues (GH-120520)

  * Add an InternalDocs file describing how interning should work and how to use it.

  * Add internal functions to *explicitly* request what kind of interning is done:
    - `_PyUnicode_InternMortal`
    - `_PyUnicode_InternImmortal`
    - `_PyUnicode_InternStatic`

  * Switch uses of `PyUnicode_InternInPlace` to those.

  * Disallow using `_Py_SetImmortal` on strings directly.
    You should use `_PyUnicode_InternImmortal` instead:
    - Strings should be interned before immortalization, otherwise you're possibly
      interning a immortalizing copy.
    - `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
      `SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
      backports, as they are now part of public API and version-specific ABI.

  * Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.

   Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
    - `_Py_ID`
    - `_Py_STR` (including the empty string)
    - one-character latin-1 singletons

    Now, when you intern a singleton, that exact singleton will be interned.

  * Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).

  * Intern `_Py_STR` singletons at startup.

  * Beef up the tests. Cover internal details (marked with `@cpython_only`).

  * Add lots of assertions

* Don't immortalize in PyUnicode_InternInPlace; keep immortalizing in other API (GH-121364)

  * Switch PyUnicode_InternInPlace to _PyUnicode_InternMortal, clarify docs

  * Document immortality in some functions that take `const char *`

  This is PyUnicode_InternFromString;
  PyDict_SetItemString, PyObject_SetAttrString;
  PyObject_DelAttrString; PyUnicode_InternFromString;
  and the PyModule_Add convenience functions.

  Always point out a non-immortalizing alternative.

  * Don't immortalize user-provided attr names in _ctypes

* Immortalize names in code objects to avoid crash (GH-121903)

* Intern latin-1 one-byte strings at startup (GH-122303)

There are some 3.12-specific changes, mainly to allow statically allocated strings in deepfreeze. (In 3.13, deepfreeze switched to the general `_Py_ID`/`_Py_STR`.)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@colesbury colesbury colesbury approved these changes

Assignees
No one assigned
Labels
skip news
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@encukou @colesbury