Type checking for *args and **kwargs when passing them to another function

[t1m013y]

Problem:

I am writing a python application, and I made a function that creates an object and passes its additional *args and **kwargs to object's constructor and takes few its own arguments. I want to make *args and **kwargs typed exactly as inner function's arguments.

Description and examples:

A simple example with two functions:

def inner(a: int, b: int) -> None:
    ...


def wrapper(arg1: str, *args, **kwargs) -> None:
    inner(*args, **kwargs)  # It will automatically detect passing arguments to another function

In my case, inner is object constructor, I gave an example with functions to make it more simple.

Type checkers should see wrapper's signature like this:

def wrapper(arg1: str, a: int, b: int):
    ...

It may be enabled by default when passing *args and **kwargs to another function is detected or it may be enabled by adding a decorator to the function.

Detailed description:

Already suggested this to mypy, detailed description of this feature can be found here: python/mypy#19302.

[erictraut]

Here's how that would typically be done.

from typing import Callable, Concatenate

def inner(a: int, b: int) -> None: ...

def create_wrapper[**P, R](fn: Callable[P, R]) -> Callable[Concatenate[str, P], R]:
    def wrapper(arg1: str, *args: P.args, **kwargs: P.kwargs) -> R:
        return fn(*args, **kwargs)

    return wrapper

wrapper = create_wrapper(inner)

[t1m013y]

Here's how that would typically be done.

from typing import Callable, Concatenate

def inner(a: int, b: int) -> None: ...

def create_wrapper[**P, R](fn: Callable[P, R]) -> Callable[Concatenate[str, *P], R]:
def wrapper(arg1: str, *args: P.args, **kwargs: P.kwargs) -> R:
return fn(*args, **kwargs)

return wrapper

wrapper = create_wrapper(inner)

It's too difficult, need to create a single-use decorator with complex typing. My feature is about an easy way to do this (automatically detected passing arguments to another function or by just adding a decorator to a wrapper).

[t1m013y]

Here's how that would typically be done.

from typing import Callable, Concatenate

def inner(a: int, b: int) -> None: ...

def create_wrapper[**P, R](fn: Callable[P, R]) -> Callable[Concatenate[str, P], R]:
def wrapper(arg1: str, *args: P.args, **kwargs: P.kwargs) -> R:
return fn(*args, **kwargs)

return wrapper

wrapper = create_wrapper(inner)

Please see python/mypy#19302 for detailed information. It would be much more easy, flexible and readable if this feature existed in the form in which I suggested it (auto-detect or adding a decorator).

[wyattscarpenter]

I agree completely. Specifying these types is far too difficult for end users and the typechecker should deduce them, instead. It's a common pattern that should be supported automatically.

[t1m013y]

I agree completely. Specifying these types is far too difficult for end users and the typechecker should deduce them, instead. It's a common pattern that should be supported automatically.

Additionally, decorators were initially created to modify behavior of many functions, not for single-use like the way it's typically done for passing *args and **kwargs.

[wyattscarpenter]

I wonder if a not-single-use decorator to do this could be defined with the current typing rules. Something like

def inner(a: int, b: int) -> None:
    ...

@pass_through(inner)
def wrapper(arg1: str, *args, **kwargs) -> None:
    inner(*args, **kwargs)

I took at stab at this but wasn't really able to get anywhere, despite summiting the trippy levels of recursive indirection involved. I don't think the current typing spec has suitable machinery to manipulate type signatures in this way.

[wyattscarpenter]

If anyone comes here looking for a workaround, I did figure out this similar technique, using our old friend TypedDict and PEP 692 – Using TypedDict for more precise **kwargs typing. Doesn't accomplish quite the desired thing, but it does single-source the type signature of the kwargs. In a TypedDict.

from typing import TypedDict, Unpack, reveal_type

InnerKwargs = TypedDict('InnerKwargs', {'x': int, 'y': int})

def inner(**kwargs: Unpack[InnerKwargs]) -> None:
    ...

def wrapper(arg1: str, **kwargs: Unpack[InnerKwargs]) -> None:
    inner(**kwargs)

reveal_type(wrapper) # Type of "wrapper" is "(arg1: str, **kwargs: **InnerKwargs) -> None"

wrapper("high") #Complains about missing arguments unless you mark the Point2D as total=False
wrapper("high", x=1, y=2, z=4) #Complains about No parameter z even if closed=False

Of course, one of the biggest reasons to want to pass through kwargs (also also args) is because the inner function is something you don't control, as described in #1501 — this solution doesn't fix that, since you'd have to change the type signature of the inner function to use a TypedDict kwargs (or, you'd have to duplicate its arguments in your kwargs TypedDict, anyway).

[t1m013y]

I wonder if a not-single-use decorator to do this could be defined with the current typing rules. Something like

def inner(a: int, b: int) -> None:
...

@pass_through(inner)
def wrapper(arg1: str, *args, **kwargs) -> None:
inner(*args, **kwargs)

I took at stab at this but wasn't really able to get anywhere, despite summiting the trippy levels of recursive indirection involved. I don't think the current typing spec has suitable machinery to manipulate type signatures in this way.

Yes, I have already described the similar decorator in python/mypy#19302, but with few additional options (this is the decorator factory):

def pass_args(func: collections.abc.Callable | None, pass_args: bool = False, pass_kwargs: bool = False):
    ...

# Usage:
@pass_args(inner, pass_args=True, pass_kwargs=True)
def wrapper(arg1: int, *args, **kwargs):
   ...
   inner(*args, **kwargs)
   ...

If anyone comes here looking for a workaround, I did figure out this similar technique, using our old friend TypedDict and PEP 692 – Using TypedDict for more precise **kwargs typing. Doesn't accomplish quite the desired thing, but it does single-source the type signature of the kwargs. In a TypedDict.

from typing import TypedDict, Unpack, reveal_type

InnerKwargs = TypedDict('InnerKwargs', {'x': int, 'y': int})

def inner(**kwargs: Unpack[InnerKwargs]) -> None:
...

def wrapper(arg1: str, **kwargs: Unpack[InnerKwargs]) -> None:
inner(**kwargs)

reveal_type(wrapper) # Type of "wrapper" is "(arg1: str, **kwargs: **InnerKwargs) -> None"

wrapper("high") #Complains about missing arguments unless you mark the Point2D as total=False
wrapper("high", x=1, y=2, z=4) #Complains about No parameter z even if closed=False

Of course, one of the biggest reasons to want to pass through kwargs (also also args) is because the inner function is something you don't control, as described in #1501 — this solution doesn't fix that, since you'd have to change the type signature of the inner function to use a TypedDict kwargs (or, you'd have to duplicate its arguments in your kwargs TypedDict, anyway).

I originally needed this for the library function, so it isn't the solution.

[t1m013y]

In the original message in python/mypy#19302 I mentioned the decorator factory which modifies argument passing behavior syntax like this:

def pass_args(func: collections.abc.Callable | None, pass_args: bool = False, pass_kwargs: bool = False):
    ...

Now I think it would be more practical to make pass_args and pass_kwargs True by default.

[t1m013y]

Also I think it would be useful to add an option to pass func argument to the decorator both function itself and as a type (would be useful for typing another decorators) (see python/mypy#19302)