Skip to content

Commit 8e103a5

Browse files
committed
Fix HFS limit and cursor handling
Obelisk HFS still returns a cursor to the rest of data, even after the limit is reached. This peculiar behaviour meant that the `limit` was never actually respected. The workaround implemented here, by client side limiting len(result_set), was suggested by Obelisk HFS developers.
1 parent 8266428 commit 8e103a5

File tree

1 file changed

+10
-0
lines changed

1 file changed

+10
-0
lines changed

src/obelisk/asynchronous/consumer.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,16 @@ async def query(self, datasets: List[str], metrics:Optional[List[str]] = None,
163163
result_set.extend(result.items)
164164
cursor = result.cursor
165165

166+
if limit and len(result_set) >= limit:
167+
"""On Obelisk HFS, limit is actually page size,
168+
so continuing to read the cursor will result in a larger than desired
169+
set of results.
170+
171+
On the other hand, if the limit is very large,
172+
we may need to iterate before we reach the desired limit after all.
173+
"""
174+
break
175+
166176
return result_set
167177

168178

0 commit comments

Comments
 (0)