Skip to content

Commit c9f2bdb

Browse files
committed
Re-insert missing countfile search lines
Signed-off-by: Davis Wertheimer <davis.wertheimer@ibm.com>
1 parent 0f667be commit c9f2bdb

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

fms_fsdp/utils/dataset_utils.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1129,6 +1129,10 @@ def setup(self):
11291129
min(max((end - tally) / shard_sizes[i], 0), 1),
11301130
]
11311131
tally += shard_sizes[i]
1132+
1133+
# Assemble length of each owned shard file
1134+
doc_counts = {}
1135+
if len(countfiles) > 0:
11321136
# Count file exists, use it
11331137
with open(countpath, "r") as csvfile:
11341138
reader = csv.DictReader(csvfile)

0 commit comments

Comments
 (0)