Skip to content

Commit 3baae42

Browse files
authored
Loader optimization: Fix incorrect in-place sort of chunks by partition (#116)
* Fix incorrect in-place sort * Update CHANGELOG
1 parent 78e7ad0 commit 3baae42

File tree

2 files changed

+6
-1
lines changed

2 files changed

+6
-1
lines changed

CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# Changelog
22

3+
## Unreleased
4+
5+
- Fixed issue loader grouping an unordered iterable by partition, speeding up loads of items with mixed partitions [#116](https://github.com/stac-utils/pgstac/pull/116)
6+
37
## [v0.6.3]
48

59
### Fixed

pypgstac/pypgstac/load.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -552,7 +552,8 @@ def load_items(
552552
items = self.read_hydrated(file)
553553

554554
for chunk in chunked_iterable(items, chunksize):
555-
list(chunk).sort(key=lambda x: x["partition"])
555+
chunk = list(chunk)
556+
chunk.sort(key=lambda x: x["partition"])
556557
for k, g in itertools.groupby(chunk, lambda x: x["partition"]):
557558
self.load_partition(self._partition_cache[k], g, insert_mode)
558559

0 commit comments

Comments
 (0)