Fix infinite iter_chunks() on empty response
#11397
Conversation
When aiohttp returns an empty reader, the first time the iter_chunks() iterator stops after the first read as expected. However, on the second empty reader return, iter_chunks() loops forever. This is due to state sharing caused by the global EmptyStreamReader instance. It has been noticed during review in aio-libs#7645, but so far hasn't been addressed. This PR attempts to address the issue by returning a new instance.
for more information, see https://pre-commit.ci
|
This came up in Home Assistant Core PR home-assistant/core#147899. The PR now avoids reading empty responses as a work around. /cc @bdraco |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @@ -54,7 +54,7 @@ | |||
| TransferEncodingError, | |||
| ) | |||
| from .http_writer import HttpVersion, HttpVersion10 | |||
| from .streams import EMPTY_PAYLOAD, StreamReader | |||
| from .streams import EMPTY_PAYLOAD, StreamReader, empty_stream_reader | |||
Check notice
Code scanning / CodeQL
Unused import Note
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #11397 +/- ##
==========================================
- Coverage 98.76% 98.74% -0.02%
==========================================
Files 129 129
Lines 43416 43444 +28
Branches 2324 2328 +4
==========================================
+ Hits 42879 42899 +20
- Misses 383 388 +5
- Partials 154 157 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
CodSpeed Performance ReportMerging #11397 will not alter performanceComparing Summary
|
Dreamsorcerer
added
backport-3.12
| @@ -604,6 +605,18 @@ def read_nowait(self, n: int = -1) -> bytes: | |||
| EMPTY_PAYLOAD: Final[StreamReader] = EmptyStreamReader() | |||
|
|
|||
|
|
|||
| def empty_stream_reader() -> "EmptyStreamReader": | |||
There was a problem hiding this comment.
Is there a point to having a function here, and not just importing EmptyStreamReader?
| @@ -38,7 +38,11 @@ from .http_writer import ( | |||
| HttpVersion10 as _HttpVersion10, | |||
| HttpVersion11 as _HttpVersion11, | |||
| ) | |||
| from .streams import EMPTY_PAYLOAD as _EMPTY_PAYLOAD, StreamReader as _StreamReader | |||
| from .streams import ( | |||
| EMPTY_PAYLOAD as _EMPTY_PAYLOAD, | |||
There was a problem hiding this comment.
This one is presumably also unused now.
| @@ -604,6 +605,18 @@ def read_nowait(self, n: int = -1) -> bytes: | |||
| EMPTY_PAYLOAD: Final[StreamReader] = EmptyStreamReader() | |||
There was a problem hiding this comment.
We'll probably want a followup PR to remove EMPTY_PAYLOAD too, without backporting.
| async for chunk in reader1.iter_chunks(): | ||
| chunks.append(chunk) | ||
| if len(chunks) > 5: # Safety break | ||
| break |
There was a problem hiding this comment.
Coverage doesn't seem happy with these. Not sure if it's just confused, as it seems to suggest that the break line is reached without any of the above lines being reached...
| chunks = [] | ||
| async for chunk in reader1.iter_chunks(): | ||
| chunks.append(chunk) | ||
| if len(chunks) > 5: # Safety break | ||
| break | ||
|
|
||
| assert len(chunks) == 0 # Should terminate normally with no chunks |
There was a problem hiding this comment.
If the expected behaviour is that we don't iterate here, then we should do something like:
| chunks = [] | |
| async for chunk in reader1.iter_chunks(): | |
| chunks.append(chunk) | |
| if len(chunks) > 5: # Safety break | |
| break | |
| assert len(chunks) == 0 # Should terminate normally with no chunks | |
| with pytest.raises(StopIteration): | |
| await anext(reader1.iter_chunks()) |
What do these changes do?
When aiohttp returns an empty reader, the first time the
iter_chunks()iterator stops after the first read as expected. However, on the second empty reader return,iter_chunks()loops forever.This is due to state sharing caused by the global
EmptyStreamReaderinstance. It has been noticed during review in #7645, but so far hasn't been addressed.This PR attempts to address the issue by returning a new instance.
Are there changes in behavior for the user?
The change makes it safe to read content of empty responses using
iter_chunks().Is it a substantial burden for the maintainers to support this?
Related issue number
Checklist
CONTRIBUTORS.txtCHANGES/foldername it
<issue_or_pr_num>.<type>.rst(e.g.588.bugfix.rst)if you don't have an issue number, change it to the pull request
number after creating the PR
.bugfix: A bug fix for something the maintainers deemed animproper undesired behavior that got corrected to match
pre-agreed expectations.
.feature: A new behavior, public APIs. That sort of stuff..deprecation: A declaration of future API removals and breakingchanges in behavior.
.breaking: When something public is removed in a breaking way.Could be deprecated in an earlier release.
.doc: Notable updates to the documentation structure or buildprocess.
.packaging: Notes for downstreams about unobvious side effectsand tooling. Changes in the test invocation considerations and
runtime assumptions.
.contrib: Stuff that affects the contributor experience. e.g.Running tests, building the docs, setting up the development
environment.
.misc: Changes that are hard to assign to any of the abovecategories.
Make sure to use full sentences with correct case and punctuation,
for example:
Use the past tense or the present tense a non-imperative mood,
referring to what's changed compared to the last released version
of this project.