Skip to content

Commit 392b15f

Browse files
committed
Apply fix for tiny documents
1 parent b9dd73f commit 392b15f

File tree

1 file changed

+6
-0
lines changed
  • app/indexer/core/src/main/java/com/microsoft/openai/samples/indexer/parser

1 file changed

+6
-0
lines changed

app/indexer/core/src/main/java/com/microsoft/openai/samples/indexer/parser/TextSplitter.java

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,12 @@ public List<SplitPage> splitPages(List<Page> pages) {
5656
int length = allText.length();
5757
int start = 0;
5858
int end = length;
59+
60+
if (length <= maxSectionLength) {
61+
splitPages.add(new SplitPage(findPage(start, pages), allText.toString()));
62+
return splitPages;
63+
}
64+
5965
while (start + sectionOverlap < length) {
6066
int lastWord = -1;
6167
end = start + maxSectionLength;

0 commit comments

Comments
 (0)