FEATURE: Update OpenAI tokenizer to GPT-4o and later one#1467

xfalcox merged 2 commits intomaindiscourse/discourse-ai:mainfrom

openai-new-tokenizerdiscourse/discourse-ai:openai-new-tokenizerCopy head branch name to clipboard