Skip to content

Commit 42eb356

Browse files
[OpenAI]: Encoding Model (#31402)
- **Description:** Small Fix for when getting the encoder in case of KeyError and using the correct encoder for newer models - **Issue:** #31390
1 parent 40bd71c commit 42eb356

File tree

1 file changed

+6
-2
lines changed
  • libs/partners/openai/langchain_openai/chat_models

1 file changed

+6
-2
lines changed

libs/partners/openai/langchain_openai/chat_models/base.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1259,8 +1259,12 @@ def _get_encoding_model(self) -> tuple[str, tiktoken.Encoding]:
12591259
try:
12601260
encoding = tiktoken.encoding_for_model(model)
12611261
except KeyError:
1262-
model = "cl100k_base"
1263-
encoding = tiktoken.get_encoding(model)
1262+
encoder = "cl100k_base"
1263+
if self.model_name.startswith("gpt-4o") or self.model_name.startswith(
1264+
"gpt-4.1"
1265+
):
1266+
encoder = "o200k_base"
1267+
encoding = tiktoken.get_encoding(encoder)
12641268
return model, encoding
12651269

12661270
def get_token_ids(self, text: str) -> list[int]:

0 commit comments

Comments
 (0)