Subclasses are GC'ed even though it changes `__subclasses__()` results

[rentruewang]

Bug report

Bug description:

Let's say I have an abstract class / protocol.

However, some of them are automatically generated in anonymous scopes as syntax sugar (in below's example, function is easier to write than ExtendBaseClass).

I would have to access it through __subclasses__().

Since __subclasses__ behavior changes just by calling gc.collect(), I believe this to be a bug.

See code below:

class SomeBaseClass:
    def compute(self) -> int:
        return 1


def adapt_function(f):
    class ExtendBaseClass(SomeBaseClass):
        def compute(self):
            return f()

    return f


@adapt_function
def function():
    return 2


def find_all_subclass(cls: type, result: set):
    result.add(cls)

    for subcls in cls.__subclasses__():
        find_all_subclass(subcls, result)


def main():
    result = set()
    find_all_subclass(SomeBaseClass, result)
    print(result)


if __name__ == "__main__":
    print("Before GC")
    main()
    import gc

    gc.collect()
    print("After GC")
    main()

Output:

(venv) ➜  ~ python main.py
Before GC
{<class '__main__.adapt_function.<locals>.ExtendBaseClass'>, <class '__main__.SomeBaseClass'>}
After GC
{<class '__main__.SomeBaseClass'>}

CPython versions tested on:

3.11

Operating systems tested on:

Linux

[bswck]

I'm not sure it changes the behavior of __subclasses__(); it changes the result.

[rentruewang]

@bswck That's right. Your wording is more correct.

[ericvsmith]

Does this cause so practical problem? I think it's just the nature of the garbage collector.

[rentruewang]

@ericvsmith Yes this is caused by GC. What this means is that even though you are able to access subclasses through __subclasses__, it changes behavior when GC kicks in. I guess I'm surprised that this is this case, because GCs are not supposed to change the results of a program.

[ericvsmith]

@rentruewang : Just because you can observe the GC working doesn't mean it's a bug. The GC absolutely changes the results of programs. Does this particular behavior cause a problem in some real code?

[bswck]

I also think this behavior is desirable. If you need the class outside of the local scope, create a strong reference to that class that will outlive that scope. If you want to do that automatically to collect all subclasses ever created of your base class, use __init_subclass__ and append it to some registry.

[ericvsmith]

I'm going to mark this as pending and close it in a few days. If there's some practical problem here, we can reconsider.

[rentruewang]

@ericvsmith Yes it causes problem in some code that uses heavy meta-programming (example I provided is a simplified version).
I looked up the python references, and indeed it says that __subclasses__ are tracked with weak references, so this is not a bug per se, but quite a surprising behavior nontheless.

@bswck That is what I did, but it's hacky. Not sure why it's desirable, because this means that 2 __subclasses__ calls can yield different results (which is how I noticed this in the first place).

TLDR: I don't think this behavior is very intuitive, even though it is the natural thing that would occur per the reference (not a bug).

[bswck]

Is it really "different results" that is the problem? Why would that be supposed to always yield the same results?

s1 = set(object.__subclasses__())

from functools import partial

s2 = set(object.__subclasses__())

print(s2 - s1)  # 47 extra items after the import!

Maybe the problem you're seeing is that some items disappear from the results, not just that the results change?

[rentruewang]

@bswck Yes subclass missing after gc is the problem. Actually my use case was what my example showed, automatically generate adaptors for functions, and would be missing after gc.

To put it precisely, I dont think subclasses should be referenced by a weak pointer.