made QuerySet iteration respect chunk_size #88
Conversation
django_mongodb/compiler.py
Outdated
| chunk = [] | ||
| for i, row in enumerate(cursor): | ||
| if i % chunk_size == 0 and i > 0: | ||
| yield chunk |
There was a problem hiding this comment.
possible bug, always happens. Shouldn't chunk get re-initialized here?
There was a problem hiding this comment.
the other thing, you could move line 116 up above the if and you can remove the i > 0 check
There was a problem hiding this comment.
And, the last query. 😄 . you aren't using len(chunk) I think the function would be easy to red in the following way:
def cursor_iter(self, cursor, chunk_size, columns):
"""Generator to yield chunks from cursor."""
chunk = []
for row in cursor:
chunk.append(self._make_result(row, columns))
if len(chunk) == chunk_size:
yield chunk
chunk = []
yield chunk
timgraham
force-pushed
the
chunk_size
branch
2 times, most recently
from
b0a8296 to
6313c34
Compare
| if not chunked_fetch: | ||
| # If using non-chunked reads, read data into memory. | ||
| return list(result) | ||
| return result | ||
|
|
||
| def results_iter( |
There was a problem hiding this comment.
Sorry if this is a dumb question. How is this getting called in relation to _make_result or cursor_iter
There was a problem hiding this comment.
It's called afterward, except when results is None and this method invokes execute_sql() itself. You can look at ModelIterable.__iter__() in django/db/models/query.py and see compiler.execute_sql and compiler.results_iter
| chunk.append(self._make_result(row, columns)) | ||
| if len(chunk) == chunk_size: | ||
| yield chunk | ||
| chunk = [] |
There was a problem hiding this comment.
So "Chunk" here is the number of items in a list and not bounded by bytes? (confirming as that was the expectation)
There was a problem hiding this comment.
Right. Django's default for chunk_size is 100 (rows), or 2000 if using QuerySet.iterator().
No description provided.