Throwing OutOfMemoryException while getting encoding model

[CodePlacer]

Discussed in #28

^{Originally posted by CodePlacer May 10, 2024}
When calling below method to get the encoding for model gpt-4, am getting out of memory exception. Exception is not throwing all the time. Its throwing in some times and not able to understand the exact cause. Can you please help?

Version Used
1.1.3

Code

string model = "gpt-4";
var encoding = Tiktoken.Encoding.ForModel(model);

Error Stack

Exception of type System.OutOfMemoryException was thrown.","stackTrace":" at System.Collections.Generic.Dictionary\u00602.Initialize(Int32 capacity)\r\n at System.Collections.Generic.Dictionary\u00602..ctor(Int32 capacity, IEqualityComparer\u00601 comparer)\r\n at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable\u00601 source, Func\u00602 keySelector, Func\u00602 elementSelector, IEqualityComparer\u00601 comparer)\r\n at Tiktoken.CoreBpe..ctor(IReadOnlyDictionary\u00602 encoder, IReadOnlyDictionary\u00602 specialTokensEncoder, String pattern) in //src/libs/Tiktoken/CoreBPE.cs:line 44\r\n at Tiktoken.Encoding..ctor(EncodingSettingModel setting) in //src/libs/Tiktoken/Encoding.cs:line 80\r\n at Tiktoken.Encoding.Get(String encodingName) in /_/src/libs/Tiktoken/Encoding.cs:line 48