tp_dict slot of static builtin types is NULL in 3.12, without mention in the changelog or an alternative

[ronaldoussoren]

PyObjC contains some functionality that needs to walk the entire MRO of classes and access the tp_dict slot. As of Python 3.12 beta 1 this code crashes due to the slot being NULL on static builtin types.

The code in PyObjC walks the MRO to implement tp_getattro for the proxy of Objective-C classes to replicate PyObject_GenericGetAttr with some customisations, as wel as in the implementation of an alternative to super (again with customisations in the attribute resolution path). All code paths only need read-only access to the tp_dict of static builtin types, although some code needs to be able to update the tp_dict of classes created by PyObjC.

There is currently no documented alternative for directly accessing tp_dict, and because of this I've changed PyObjC to use the private API _PyType_GetDict.

Some alternatives I've looked into:

• Use PyObject_GetAttrString(someType, "__dict__"): This increases the complexity of the PyObjC code bases and requires changes to long stable code:

  • Needs to introduce a different code path for accessing the __dict__ slot of non-PyObjC subclasses

  • Needs to switch to the abstract Mapping API instead of the PyDict API, which likely has a performance impact

  • Changes to reference counting invariants in the code

• Use _PyType_GetDict on Python 3.12 or later: This works, but introduces a reliance on a private CPython API that might disappear at any moment (even micro releases), for example by no longer exporting the function from shared libraries.

Proposal: Rename _PyType_GetDict to either PyUnstable_Type_GetDict or PyType_GetDict, while keeping the current interface (e.g. returning a borrowed reference and never raising an exception).

This is a follow-up from #105020

@ericsnowcurrently

Linked PRs

• gh-105227: Add PyType_GetDict() #105747
• [3.12] gh-105227: Add PyType_GetDict() (GH-105747) #106600

[ronaldoussoren]

@Yhg1s : Would it be acceptable to introduce such a new API in an upcoming 3.12 beta release?

[ronaldoussoren]

@Yhg1s, I've added the deferred blocker label because this issue is a seemingly intentional regression from 3.11 without a documented replacement.

I'm currently using a private API and that doesn't feel right.

[ericsnowcurrently]

FWIW, I'm in favor of adding PyType_GetDict(). I've put up a PR: gh-105747.

[ronaldoussoren]

I agree that its is better to return a new reference instead of a borrowed reference in APIs like this, but that can make life harder for anyone transitioning to the new way of working. In my particular case it does make transitioning harder because I'll have to add refcount updates where I didn't have to do before. That said, this is not a big problem.

[encukou]

@ericsnowcurrently, when you get back, could you review (and maybe improve) the docs?

[ericsnowcurrently]

The docs seem fine to me, after your changes. My original changes were likely unnecessarily specific. :) Was there something in particular you were concerned about?

[ericsnowcurrently]

If you think more should be done, feel free to re-open this.