Convert type check macro's to functions on the limited api

[bschoenmaeckers]

As discussed at https://discuss.python.org/t/c-api-add-py-getbuildintype-and-py-getbuildintypeborrowed/107010/12.

It is desirable for alternative Python implementations to avoid using static exported type objects. Their primary usage is in type checking macros, so I propose converting them to exported functions instead.

See python/cpython#149558 for a draft to convert PyBool_Check.

[encukou]

• @encukou
• @erlend-aasland
• @serhiy-storchaka
• @vstinner
• @zooba

[vstinner]

I don't get the purpose of this change for CPython. If RustPython has specific needs, they should modify their C API implementation. In the associated PR, PyAPI_DATA(PyTypeObject) PyBool_Type; remains exposed in Include/object.h, so why not using it? The change can slow down C extensions calling PyBool_Check/PyBytes_Check.

[bschoenmaeckers]

RustPython is not able to implement these static type exports. Because RustPython's PyObject layout requires data to be allocated in front of the pointer (prefix allocation). This makes me belief that these static type export are leaking to much of cpython's implementation details.

Ref https://github.com/RustPython/RustPython/blob/320355f6335cc2628477ad7f5347f5ff6783059c/crates/vm/src/object/core.rs#L1052-L1141

[encukou]

@vstinner, see https://discuss.python.org/t/c-api-add-py-getbuildintype-and-py-getbuildintypeborrowed/107010/8

[vstinner]

@vstinner, see https://discuss.python.org/t/c-api-add-py-getbuildintype-and-py-getbuildintypeborrowed/107010/8

This message doesn't answer my questions. I don't see the benefits for CPython, it can only make C extensions (and CPython itself) slower.

I cannot find the C API implementation in https://github.com/RustPython/RustPython/. Is it already implemented or not? What is the current RustPython code for PyBool_Check()?

[bschoenmaeckers]

I cannot find the C API implementation in https://github.com/RustPython/RustPython/. Is it already implemented or not? What is the current RustPython code for PyBool_Check()?

I'm planning to implement all type check functions using the following macro: RustPython/RustPython#7871

[zooba]

Since the plan seems to be to boil the frog slowly as we move the stable ABI to where we want it, I think we should make the current macros available as functions as an unfortunate transition step and plan to get to a Py_GetBuiltinType(enumeration) and equivalent functions to isinstance and issubclass for the limited C API¹.

If we could go back in time, we wouldn't use data exports or macros or static PyObject definitions². Since we don't have the time machine, we can only add the ones we want and can't remove the ones we don't.

I'm not thrilled about adding so many functions for each specific type, and I'd put a note with them saying that "this isn't a sign you should add more" to any future contributors, but unless we want to force everyone to change their source code (spoiler: we don't), we have to keep them available.

I wouldn't add GetBuiltinTypeBorrowed, because we don't want to use it in the macro, and we don't want to be returning borrowed references. Making the entire type check macro into a function avoids problems here. I think adding a borrowed ref function would be actively making things worse.

I don't have any issue with RustPython choosing not to support modules built with pre-3.16 CPython (or whatever guarantees they want to give), or using dirty hacks to support them. But it's a reasonable request that we update the stable ABI definition to not require dirty hacks from other languages just to emulate linker features.

Footnotes

1. Which may exist already, I didn't check and haven't had to use them recently. ↩

2. Since PyTypeObject is a subclass of PyObject and so has to support refcounting, which is why we invented the hideously complex immortalisation system instead of just isolating them properly - again, lack of time machine making us do stuff we don't like. ↩

[zooba]

If someone confirms that the isinstance and issubclass equivalents are already in the stable ABI, then consider me voting +1 for the proposal (based on the single conversion in the linked PR).

[encukou]

@vstinner: the current API locks in an implementation detail: if PyLong_Type is a process-global constant, int must be a static type that's shared across all interpreters.
(Which, in turn, means that any implementation of the Stable ABI currently must support static types.)

---

@zooba

If someone confirms that the isinstance and issubclass equivalents are already in the stable ABI, then consider me voting +1 for the proposal (based on the single conversion in the linked PR).

Note that this API is used are looking at memory layout compatibility, so unlike the Python functions, it ignores __instancecheck__/__subclasscheck__.
All the variants exist though: PyObject_IsInstance/PyObject_IsSubclass honour the hooks; PyObject_TypeCheck/PyType_IsSubtype are the C-level type checks.

---

I guess we could skip the transition step? We can add Py_GetBuiltinType, and switch the macros to use it ¹ -- keeping them as macros?
Then language bindings would either need to do more work reimplementing the macros, or just go without them and let users type Py_GetBuiltinType instead.

Footnotes

1. and PyType_IsSubtype, or PyType_FastSubclass, as applicable ↩

[zooba]

All the variants exist though

I assume in the stable ABI, so good 👍

I guess we could skip the transition step?

The macros as they stand can't handle strong references (no way to do the decref on the type object), so we either need to assume that they're all immortal and just not decref (and assume we'll ~never have a reason to switch back to ref-counted type objects, which is the actual assumption I'm not confident in), or we need to add an API that doesn't incref (which we previously agreed we didn't want to add).

Having dedicated functions for the existing macros protects us from all of those. For a limited set of primitive types, I think that's okay.

[vstinner]

@vstinner: the current API locks in an implementation detail: if PyLong_Type is a process-global constant, int must be a static type that's shared across all interpreters.

Modifying PyLong_CheckExact() is not enough to be able to remove PyLong_Type from the stable ABI. The type remains exported by PyAPI_DATA(PyTypeObject) PyLong_Type;. If someone wants to remove PyLong_Type from the stable ABI, I would prefer to see a complete plan, not just fix PyLong_CheckExact().

(Which, in turn, means that any implementation of the Stable ABI currently must support static types.)

Python implementations other than CPython can implement PyLong_CheckExact() differently, they don't have to use exactly:

#define PyLong_CheckExact(op) Py_IS_TYPE((op), &PyLong_Type)

[zooba]

If someone wants to remove PyLong_Type from the stable ABI, I would prefer to see a complete plan, not just fix PyLong_CheckExact().

I agree with you, but other than the type check and arbitrary uses by users, where else is it? Once we no longer rely on it ourselves, the rest of the plan is to deprecate and encourage users to rewrite their own code, and then wait long enough before dropping it (while other implementations can start without them).

If we have other places where we force users to rely on the data export, where are they?

[encukou]

I assume in the stable ABI

Ha! PyObject_TypeCheck is missing, but, it's defined as Py_IS_TYPE(ob, type) || PyType_IsSubtype(Py_TYPE(ob), type). We'll want to expose the function, now that Py_IS_TYPE is also an ABI function.

assume that they're all immortal [...] or add an API that doesn't incref (which we previously agreed we didn't want to add).

Maybe we should revisit the decision?
Like with Py_GetConstantBorrowed, the user-facing story would be that they're borrowed from the interpreter. That sounds okay for a limited set of built-in types?

Modifying PyLong_CheckExact() is not enough to be able to remove PyLong_Type from the stable ABI.

Yes. I imagine RustPython will, as the first step, implement a subset of the Stable ABI, and be able to load many (but not all) Stable ABI extensions.

The next step would be adding Py_GetBuiltinType_Borrow, and implementing PyLong_Type as a macro that calls it. (That'd break source compatibility -- PyLong_Type would no longer be a compile-time constant; on some platforms it now is.)
[edit] Or deprecating and dropping PyLong_Type, as Steve suggests.

Python implementations other than CPython can implement PyLong_CheckExact() differently

They can, but that won't affect extensions that are built with CPython and then loaded on that implementation.