gh-113993: Allow interned strings to be mortal, and fix related issues

[encukou]

I've spent too much time looking at this myself, it wants more eyes :)

I spent a week learning about the string interning mechanism, and wrote up how I think it should work in an InternalDocs file I'm adding here.

I found a bunch of ... quirks if not outright bugs. For example, we have duplicate singletons (e.g. _Py_ID(a) and the latin1 short string a). I don't think I can bring back mortal interned strings without getting my idea of the de# sync with the code, so, this ended up being a big PR.

---

• Add an InternalDocs file describing how interning should work and how to use it.
  (Please review this first!)

• Add internal functions to explicitly request what kind of interning is done:

  • _PyUnicode_InternMortal
  • _PyUnicode_InternImmortal
  • _PyUnicode_InternStatic

• Switch uses of PyUnicode_InternInPlace to those.

• Disallow using _Py_SetImmortal on strings directly.
  You should use _PyUnicode_InternImmortal instead:

  • Strings should be interned before immortalization, otherwise you're possibly
    interning a immortalizing copy.
  • _Py_SetImmortal doesn't handle the SSTATE_INTERNED_MORTAL to
    SSTATE_INTERNED_IMMORTAL update, and those flags can't be changed in
    backports, as they are now part of public API and version-specific ABI.

• Add private _only_immortal argument for sys.XXX, used in refleak test machinery.

• Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:

  • _Py_ID
  • _Py_STR (including the empty string)
  • one-character latin-1 singletons

  Now, when you intern a singleton, that exact singleton will be interned.

• Add a _Py_LATIN1_CHR macro, use it instead of _Py_ID/_Py_STR for one-character latin-1 singletons everywhere (including Clinic).

• Intern _Py_STR singletons at startup.

  Try this in 3.12: (click to expand)

  import sys
  
  a = sys.intern('<module>')  # normal string
  print('a', id(a), sys.getrefcount(a))
  try:
      raise Exception()
  except Exception as err:
      b = err.__traceback__.tb_frame.f_code.co_name  # same string via _Py_STR
  
  assert sys.intern(a) is sys.intern(b)

  In 3.13 the reproducer doesn't work but I don't think the underlying unsoundness was fixed.

• For free-threaded builds, intern _Py_LATIN1_CHR singletons at startup.

• Beef up the tests. Cover internal details (marked with @cpython_only).

• Add lots of assertions

• Issue: Interned strings are immortal, despite what the documentation says #113993

[encukou]

The buildbot failures are unrelated; I'll merge.

Thank you for the reviews!

[miss-islington-app]

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @encukou, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6f1d448bc110633eda110310fd833bd46e7b30f2 3.13

[miss-islington-app]

Sorry, @encukou, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6f1d448bc110633eda110310fd833bd46e7b30f2 3.12

[ericsnowcurrently]

Thanks for taking care of this, @encukou!

[bedevere-app]

GH-120945 is a backport of this pull request to the 3.13 branch.

[eduardo-elizondo]

This is great! Quick follow-up @encukou, should we rebase this one now: #113601?

[encukou]

@eduardo-elizondo Yup, I've updated it, and left some notes about the docs.
If you want to delegate the writing, let me know and I'll take over the PR :)

[eduardo-elizondo]

@eduardo-elizondo Yup, I've updated it, and left some notes about the docs. If you want to delegate the writing, let me know and I'll take over the PR :)

Fixing them now! Let's close out in the other PR

[iritkatriel]

Would be good to have a link to this file from the index in InternalDocs/README.md.