Skip to content

Commit ad7d52a

Browse files
Dataset parquet chunking fix (#349)
* adds return if entities in chunk = 0 * print statement for testing * adds logging info message
1 parent 56bdd78 commit ad7d52a

1 file changed

Lines changed: 7 additions & 0 deletions

File tree

digital_land/package/dataset_parquet.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,13 @@ def load_entities_range(
218218

219219
# distinct_fields - list of fields in the field in fact
220220
rows = self.conn.execute(query).fetchall()
221+
# if there are no entities in the entity range then we don't need to proceed
222+
if len(rows) == 0:
223+
logger.info(
224+
f"skipping entity numbers {entity_range[0]} to {entity_range[1]} as there are no entities"
225+
)
226+
return
227+
221228
distinct_fields = [row[0] for row in rows]
222229

223230
# json fields - list of fields which are present in the fact table which

0 commit comments

Comments
 (0)