Introspection: Introduce TypeHint struct #5438

Tpt · 2025-09-12T13:00:37Z

inspect::TypeHint is an AST of a Python type hint.

It is supposed to be created using a set of const constructors and used only in const context.

Introspection data generation is done using two associated methods. The serialized data is also an AST.

I make use of the AST to generate nice from X import Y as Z lines in the type stubs with code that auto-detect conflicts.

Miscellaneous changes:

Rename PyType{Info,Check}::TYPE_INFO into TYPE_HINT
Drop redundant PyClassImpl::TYPE_NAME

Tpt · 2025-09-12T13:25:57Z

src/inspect/mod.rs

+pub struct TypeHint {
+    /// The type hint annotation
+    #[doc(hidden)]
+    pub annotation: &'static str,


I would have preferred it to be an enum but sadly I don't see how to serialize it: const functions cannot concatenate and recursive macros are not possible

What would the enum definition be? I can see if I can come up with any crazy ideas if I know the preferred form.

Something like:

enum TypeHint {clashes BuiltIn { name: &'static str }, Member { module: &'static str, name: &'static str }, Union(&[TypeHint]), Subscript { value: TypeHint, parameters: &[TypeHint] }, Callable { arguments: &[TypeHint], return: TypeHint } // Special case because of the nested [] syntax }

This way it's easy to ensure the syntax is valid and to do manipulation like introducing aliases in case of name conflicts.

Hmm, I think if we split the const fn up into one which (recursively) calculates the final length, and a second which serializes to it with &mut [u8] similar to what I did in the concat.rs module, I think we could probably make the const fn work?

Unfortunately this would require MSRV 1.83. But this is currently an experimental feature, I would personally be ok with having it limited to MSRV 1.83.

Done!

This works fairly well. I have not introduced macros to build the TypeHint enum yet, I plan to do it in a follow up. My guess is that the real use case is for the custom type hint in the signature, most of INPUT_TYPE and OUTPUT_TYPE are likely fine with just the const constructors.

I now serialize the type hint as an AST, this allows to manipulate it when building the type stubs. As an example, I wrote some code that generates nice from X import Y imports and generate aliases in case of name conflicts.

src/inspect/mod.rs

davidhewitt

Thanks, this looks like the right direction for making the type hint definitions reliable wrt import resolution.

I think we can iterate a bit to find the best design for users, I wrote down some ideas.

davidhewitt · 2025-09-14T11:44:29Z

src/inspect/mod.rs

+    }};
+}
+
+/// Allows to build a [`TypeHint`] that is the subscripted


Sentence looks unfinished.

davidhewitt · 2025-09-14T11:55:06Z

src/inspect/mod.rs

+pub struct TypeHint {
+    /// The type hint annotation
+    #[doc(hidden)]
+    pub annotation: &'static str,


What would the enum definition be? I can see if I can come up with any crazy ideas if I know the preferred form.

src/inspect/mod.rs

davidhewitt · 2025-09-14T12:04:49Z

pyo3-introspection/src/introspection.rs

It looks like the annotation_stub stuff is still present, should we remove it at the same time?

The code to extract modules was in the stubs.rs module and got removed

inspect::TypeHint is composed of an "annotation" string and a list of "imports" ("from X import Y" kind) The type is expected to be built using the macros `type_hint!(module, name)`, `type_hint_union!(*args)` and `type_hint_subscript(main, *args)` that take care of maintaining the import list Introspection data generation is done using the hidden type_hint_json macro to avoid that the proc macros generate too much code Sadly, outside `type_hint` these macros can't be converted into const functions because they need to do some concatenation. I introduced `type_hint!` for consistency, happy to convert it to a const function. Miscellaneous changes: - Rename PyType{Info,Check}::TYPE_INFO into TYPE_HINT - Drop redundant PyClassImpl::TYPE_NAME

Tpt · 2025-09-23T14:30:08Z

pyo3-introspection/src/model.rs

+#[derive(Debug, Eq, PartialEq, Clone, Hash)]
+pub enum TypeHint {
+    Ast(TypeHintExpr),
+    Plain(String),


This is for custom type hint witten in the #[pyo3(signature = ] attribute. I plan a follow up to parse the value in the proc-macros and generate an AST. I tend to think it's fine to restrict the possible type hint grammar and expand it in the future if needs arise (the opposite of what we did by allowing anything inside of a string)

Cargo.toml

Tpt · 2025-09-24T09:41:21Z

src/type_object.rs

+    /// Provides the full python type as a type hint.
    #[cfg(feature = "experimental-inspect")]
-    const PYTHON_TYPE: &'static str = "typing.Any";
+    const TYPE_HINT: TypeHint = TypeHint::module_member("typing", "Any");


I think we can drop this and only have the field on PyTypeCheck. I will do it in a follow up except if you prefer it done in this MR

Doesn't the blanket impl PyTypeCheck for T: PyTypeInfo need this here?

I think we can write in this blanket impl:

const TYPE_HINT: TypeHint = if let Some(module) = Self::MODULE { TypeHint::module_member(module, Self::NAME) } else { TypeHint:builtins(Self::NAME) };

Tpt · 2025-10-03T12:16:00Z

@davidhewitt If you have time for an other review it would be amazing 🥺

davidhewitt

Sorry for the slow review, have been spinning many plates this week.

This is looking very cool, nicely done! It looks like it fixes a lot of the problems we had and makes the API a lot cleaner.

I have quite a few thoughts & questions on this one 😁

Cargo.toml

pyo3-introspection/Cargo.toml

davidhewitt · 2025-10-04T19:30:20Z

pyo3-introspection/src/introspection.rs

+fn deserialize_type_hint<'de, D: Deserializer<'de>>(
+    deserializer: D,
+) -> Result<Option<ChunkTypeHint>, D::Error> {


Can you please expand on why the ChunkTypeHint type needs to exist (seems very similar to TypeHint in model.rs)? Maybe add a doc comment to the type.

I have added a comment. I duplicated the type to make it clear this ChunkTypeHint type is designed for extraction of the introspected data and will have to deal with backward compatibility whereas the TypeHint type is for the user facing API. I guess they will diverge in the future. Is it fine with you?

davidhewitt · 2025-10-04T19:43:26Z

pyo3-introspection/src/stubs.rs

+                "from bat import A as A2",
+                "from foo import A as A3, B"


Does the aliasing introduced here cause IDEs to hint the type to the user as "A2" ?

If so, I wonder if we should go the other direction and have import bat and import foo here and use fully qualified names below?

I think I've seen the Python protobuf compiler handled this same problem, we might find it useful to see what they did for that implementation.

... I wonder if rather than A2 and A3, nicer aliases might be bat_A and foo_A?

Does the aliasing introduced here cause IDEs to hint the type to the user as "A2" ?

It is not the case in PyCharm (I properly get A). I have not tested in other IDEs.

If so, I wonder if we should go the other direction and have import bat and import foo here and use fully qualified names below?

We can, it might make regular type stubs a bit less readable (a lot of full paths). We might still have some case of naming conflicts (e.g. between a module and a function) but sounds less likely. Happy to do the change if you want.

I think I've seen the Python protobuf compiler handled this same problem, we might find it useful to see what they did for that implementation.

Thanks! Will have a look. However, I guess their lives is a bit easier than ours because they do not have to deal with arbitrary imports (they only generate code from protobuf whereas we have to deal with arbitrary type hint from user code that can use whatever Python type).

... I wonder if rather than A2 and A3, nicer aliases might be bat_A and foo_A?

Indeed. I have added a TODO. My feeling is that it's a polishing that can be far in the future. WDYT?

pyo3-macros-backend/src/frompyobject.rs

davidhewitt · 2025-10-04T20:01:15Z

src/type_object.rs

+    /// Provides the full python type as a type hint.
    #[cfg(feature = "experimental-inspect")]
-    const PYTHON_TYPE: &'static str = "typing.Any";
+    const TYPE_HINT: TypeHint = TypeHint::module_member("typing", "Any");


Doesn't the blanket impl PyTypeCheck for T: PyTypeInfo need this here?

src/inspect/mod.rs

davidhewitt · 2025-10-04T20:03:28Z

src/inspect/mod.rs

+    value.serialize_for_introspection(output);
+    output
+        .split_at_mut(value.serialized_len_for_introspection())


To avoid iterating through value twice it might make sense for serialize_for_introspection to return the number of bytes written?

Indeed. Done.

src/inspect/mod.rs

davidhewitt · 2025-10-04T20:11:22Z

src/inspect/mod.rs

+    /// const T: TypeHint = TypeHint::module_member("datetime", "time");
+    /// assert_eq!(T.to_string(), "datetime.time");
+    /// ```
+    pub const fn module_member(module: &'static str, attr: &'static str) -> Self {


How about module_attr? It's slighty shorter, matches the argument names and enum name, and it feels a bit more correct to me than module_member somehow 🤔

Also, does this work for nested classes?

e.g.

# a.py class A: class B: ...

... I guess the import would be from a import A, and the annotation would be A.B.

... I guess the answer is yes, this works, as long as in the import aliasing code we only import & alias the ident up the the first dot (i.e. if another "A" is in scope, this might be aliased as A2.B)?

How about module_attr?

Done

I guess the answer is yes, this works, as long as in the import aliasing code we only import & alias the ident up the the first dot (i.e. if another "A" is in scope, this might be aliased as A2.B)?

Exactly. I just added it in 53119ed (this was missing)

Tpt · 2025-10-06T07:49:44Z

I have quite a few thoughts & questions on this one 😁

Thank you for taking the time for the thoughtful review. I was not expected so much even if I expected a lot of comments.

Follow ups in 1f30ee7 and 53119ed

# Conflicts: # noxfile.py # src/types/weakref/anyref.rs # src/types/weakref/proxy.rs

Tpt changed the title ~~Introduce TypeHint struct~~ Introspection: Introduce TypeHint struct Sep 12, 2025

Tpt force-pushed the tpt/moduleandtype branch 3 times, most recently from 7d6b267 to 7bb5b56 Compare September 12, 2025 13:24

Tpt commented Sep 12, 2025

View reviewed changes

src/inspect/mod.rs Outdated Show resolved Hide resolved

Tpt force-pushed the tpt/moduleandtype branch 5 times, most recently from 47a4fc1 to 4ece557 Compare September 12, 2025 14:15

Tpt marked this pull request as ready for review September 12, 2025 15:01

davidhewitt reviewed Sep 14, 2025

View reviewed changes

davidhewitt mentioned this pull request Sep 14, 2025

fill PyTypeInfo::NAME with Pythonic values (e.g. BaseException) instead of C ones (e.g. PyBaseException) #5352

Draft

Tpt force-pushed the tpt/moduleandtype branch from 4ece557 to 02ce980 Compare September 23, 2025 14:23

Tpt commented Sep 23, 2025

View reviewed changes

Tpt force-pushed the tpt/moduleandtype branch 2 times, most recently from 1f56ea9 to 3fee134 Compare September 23, 2025 14:38

Tpt commented Sep 23, 2025

View reviewed changes

Cargo.toml Show resolved Hide resolved

Tpt commented Sep 24, 2025

View reviewed changes

Tpt force-pushed the tpt/moduleandtype branch from 3fee134 to a43f077 Compare September 25, 2025 14:39

Makes TypeHint an enum

751482e

Tpt force-pushed the tpt/moduleandtype branch from a43f077 to 751482e Compare September 25, 2025 14:42

davidhewitt mentioned this pull request Oct 4, 2025

Raise MSRV #5486

Open

davidhewitt reviewed Oct 4, 2025

View reviewed changes

Tpt added 2 commits October 6, 2025 09:53

Merge remote-tracking branch 'upstream/HEAD' into tpt/moduleandtype

beee6a8

# Conflicts: # noxfile.py # src/types/weakref/anyref.rs # src/types/weakref/proxy.rs

Code review feedbacks

1f30ee7

Tpt force-pushed the tpt/moduleandtype branch from 613f2a6 to 1f30ee7 Compare October 6, 2025 08:37

Handle nested modules imports

53119ed

Introspection: Introduce TypeHint struct #5438

Are you sure you want to change the base?

Introspection: Introduce TypeHint struct #5438

Uh oh!

Conversation

Tpt commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tpt Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tpt Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tpt Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tpt Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tpt commented Oct 3, 2025

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tpt Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tpt Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tpt commented Sep 12, 2025 •

edited

Loading

Tpt Sep 15, 2025 •

edited

Loading

Tpt Sep 23, 2025 •

edited

Loading

Tpt Sep 23, 2025 •

edited

Loading

Tpt Oct 6, 2025 •

edited

Loading

Tpt Oct 6, 2025 •

edited

Loading

Tpt Oct 6, 2025 •

edited

Loading

Tpt commented Oct 6, 2025 •

edited

Loading