Incorporate new harmony tiktoken vocabulary #7491
Description
The recently released OSS openai models come with a new vocabulary. The TiktokenTokenizer should be updated accordingly.
cc: @tarekgh
{{ message }}
The recently released OSS openai models come with a new vocabulary. The TiktokenTokenizer should be updated accordingly.
cc: @tarekgh