[Feature Request] `pytest-beartype 0.3.0`: The State of the QA Address

[leycec]

Thank you for joining me tonight, gentle(men|women), cats, and LLMs alike. A GitHubber could not ask for a more magical audience. There may not be many of us. Yet, we are legion. We are united in a common cause of wanting @leycec to actually release pytest-beartype 0.3.0 already.

I mean, really. It's been friggin' months. What's going in that cottage in the Canadian wilderness, anyway!? Is anyone over there even doing any work anymore? They are. This is the sad tale of one bald man in a cottage in the Canadian wilderness. But first...

You Are All Invited to Live Chat with The Bear Crew

Do you have veritable mountains of valuable free time to squander "invest" in chatting? Hah. Of course you do! You're like me. You prefer doing anything to coding yet another broken line. Let the LLM do all the hard coding tonight.

Just us (by which I mean me) live at:

• beartype's new Zulip chat! I'll be on most nights from 10:00—4:00 EST. If I'm not, it's because @beartype is currently burning down, I'm running around shrieking, and my wife and the cats are muttering: "This is fine. This is fine... WAIT. Is this fine?"

Zulip. It rhymes with "tupip". No idea why. It's vaguely fun. It's kinda open-source. It's reminiscent of BBSes, IRC, America Online, and decrepit protocols like telnet, gopher, and HyperCard that nobody alive should even remember.

I even made a #pytest-beartype channel for the world-shattering QA crew here to gab about pytest-beartype specifically. There are dozens of us! Dozens! Actually... there's not even that much, huh? It's mostly just two or three unshaven dudes in bath robes.

Come hang out with the QA kids. If you dare! And now, for the moment we've all been waiting for...

What's Up with Friggin' pytest-beartype 0.3.0 Already!? 😫

In a word: the third-party pytest-asyncio plugin.

The upcoming release of pytest-beartype ships support for automatically type-checking literally all fixtures in your entire pytest stack via new --beartype-fixtures flag. That's great... if literally all fixtures in your entire pytest stack are synchronous (e.g., generators, normal functions). But what if one or more of those fixtures are asynchronous (e.g., asynchronous generaters, coroutines) backed by either pytest-asyncio or a similar asynchronous pytest plugin?

The answer currently is: "Nobody friggin' knows, but it probably ain't good." Why don't I even know? Because I can't even test the bloody functionality. I wrote it, but I can't test it. Might work. Might not. Either way, uncertainty isn't a great place for a QA tool to be.

Uhm. Why Don't You Know If Stuff You Wrote Works, Again?

Let's start over from the beginning. pytest-beartype 0.3.0 explicitly supports asynchronous fixtures through a new internal beartype_fixture_async() decorator in the main pytest-beartype codebase, which you can find here. So far, so good.

pytest-beartype 0.3.0 then attempted to test this support through a new test-specific test_option_beartype_fixtures() integration test, which you can find here. That test currently uses an official (albeit very arcane and mostly undocumented) pytest plugin called pytester. pytester works great if you don't need any optional plugins. But... we do! We really do! We need that test (and only that test) to enable the third-party pytest-asyncio plugin by passing -p asyncio. But... that doesn't work! That's why line 158 of that test submodule is currently commented out.

Thus, I only managed to get synchronous fixtures tested. They work great. Asynchronous fixtures, on the other hand? No idea. So, here's the new idea (in order):

1. Drop pytester. Honestly, pytester suuucks. It just does not work. It kinda works in common use scenarios, but even then doesn't really work. It's undocumented. It's grotty. It's mucky. It's a huge real pain for very little real gain.
2. Replace pytester with our own _run_pytest_plugin_test() runner. Basically, the sibling test_option_beartype_packages test submodule defines a vastly superior alternative to pytester. I stupidly named it _run_pytest_plugin_test(), which you can find here. It actually works, because it's just brute-forcing the execution of a new child pytest subprocess within the currently running parent pytest process. Unlike pytester, _run_pytest_plugin_test() gives us total control over how we invoke child pytest subprocesses — including enabling pytest-asyncio in that child pytest subprocess by passing -p asyncio. You may now be thinking:
   • "Uhh. Why do we use _run_pytest_plugin_test() in one unit test and pytester in another? Shouldn't we just be using _run_pytest_plugin_test() or pytester in both?" Yes. Yes, we should. You're exactly right. We should be using _run_pytest_plugin_test() everywhere. As for why we're not... I can't remember. It's not my fault! I didn't do it! I blame Guido for everything.
3. Uncomment all of the currently commented out async tests. For example, this huge block of disabled async tests. There are probably others. No idea! I don't remember any of this anymore! It's probably just a dumb path dependency. An accident of history. The left hand wasn't talking to the right hand. Neither hand was talking to the gripping hand. Oh, Gods... this is all bitrotting into /dev/null again, isn't it? 😭

pytest-asyncio. That's why you can't have good things, @adamtheturtle. I'm so sorry for taking so long, @knyazer. Your work was amazing. This is all on me. We sigh collectively as one sad tired community. 😮‍💨

Unrelatedly...

Beartype Configuration: We Can Do That, Too

Lastly, I also figured out how to let pytest-beartype users configure @beartype at test-time by defining BeartypeConf configuration blocks from within pyproject.toml like so:

# In "pyproject.toml":
[tool.pytest.ini_options]
beartype_conf = 'BeartypeConf(claw_is_pep526=False, is_debug=True)'

Cool, huh? I'll never actually finish pytest-beartype 0.3.0, huh? You're not wrong. Feature request #21 isn't a hard-blocker for pytest-beartype 0.3.0, but... we've already taken so long. What's one more feature when you're already in Hell? Don't answer that.

pytest-asyncio Summons All Who Would Hear This Desperate QA Cry

@tusharsadhwani, @knyazer, @adamtheturtle, @oyarsa, @jaens.

I thought there were more of us. I hoped there were more of us. But... yeah. Pretty sure it's just us. We'll have to be enough! Now we just need to wait for someone to fall on their sword, sacrifice weeks of their life, and make this glorious release happen for all six of us!

If only typing exclamation points made things happen.

[adamtheturtle]

Thank you for the thorough description!

[leycec]

You're most welcome. And thank you for gently prodding me to do this and... well, just being so interested about pytest-beartype. It's single awesome users like you that light up the open-source world! Our immediate Five Week Plan is thus:

1. Release beartype 0.23.0 Release Candidate 0. I'd really wanted that to be this Friday. Naturally, @ilyapoz had other ideas. 😆 -> 😢
2. Finish testing (and possibly implementing) pytest-beartype 0.3.0. As soon as beartype 0.23.0 drops, I'd be delighted to finally put this endless release cycle to rest. I'm confident I can bang out a solution that integrates well with pytest-asyncio. But...

In all honesty, it's probably gonna be a few more weeks. The main @beartype codebase takes immediate priority. It's not quite on fire over there. But... it's kinda on fire over there.

^{the beartype 0.23.0 release mantra can be your release mantra too}