Skip to content

Commit 1a2546a

Browse files
replaced split sentence default
1 parent 815300f commit 1a2546a

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

guardrails/validator_base.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ def split_sentence_str(chunk: str):
4444
return [fragments[0] + ".", ".".join(fragments[1:])]
4545

4646

47-
def split_sentence_str_v2(chunk: str):
47+
def split_sentence_word_tokenizers_jl(chunk: str):
4848
"""
4949
Use a sentence tokenizer to detect if at least one sentence is present in the chunk.
5050
We return the first sentence and the remaining chunks without the first sentence.
@@ -303,7 +303,7 @@ def _chunking_function(self, chunk: str) -> List[str]:
303303
Returns:
304304
list[str]: The text chunked into some subset.
305305
"""
306-
return split_sentence_str(chunk)
306+
return split_sentence_word_tokenizers_jl(chunk)
307307

308308
def validate_stream(
309309
self, chunk: Any, metadata: Dict[str, Any], **kwargs

0 commit comments

Comments
 (0)