[retry] Use pytorch-labs/tokenizers and remove tokenizer/ (#1401) #1443
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1443
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 1 PendingAs of commit 2075fd7 with merge base 019f76f ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
| cmake -S . -B ./cmake-out -DCMAKE_PREFIX_PATH=`python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DLINK_TORCHAO_OPS="${LINK_TORCHAO_OPS}" -DET_USE_ADAPTIVE_THREADS=ON -DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=1" -G Ninja | ||
| else | ||
| cmake -S . -B ./cmake-out -DCMAKE_PREFIX_PATH=`python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DLINK_TORCHAO_OPS="${LINK_TORCHAO_OPS}" -DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" -G Ninja | ||
| cmake -S . -B ./cmake-out -DCMAKE_PREFIX_PATH=`python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DLINK_TORCHAO_OPS="${LINK_TORCHAO_OPS}" -DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=1" -G Ninja |
There was a problem hiding this comment.
Getting "model file path should not be empty." from sentencepiece processor.
According to chatgpt:
The _GLIBCXX_USE_CXX11_ABI macro is related to the Application Binary Interface (ABI) used by the GNU C++ Standard Library.
In 2011, the C++11 standard introduced a new ABI for C++ objects, which changed the way that standard library objects, including std::string, are laid out in memory. This new ABI is enabled by default in GCC 5 and later versions.
However, when _GLIBCXX_USE_CXX11_ABI=0 is defined, it tells the compiler to use the older, pre-C++11 ABI instead. This can cause issues with std::string objects because the older ABI uses a different memory layout for these objects.
When you pass a const char* pointer to a function that expects a std::string object, the compiler needs to create a temporary std::string object from the pointer. However, if the ABI is set to the older version, the compiler may not be able to correctly create this temporary object, resulting in an empty string being passed to the function.
This issue is likely due to the fact that the older ABI uses a different memory layout for std::string objects, which includes a pointer to the string data, a length field, and a capacity field. When the compiler creates a temporary std::string object from a const char* pointer, it needs to initialize these fields correctly. However, if the ABI is set to the older version, the compiler may not be able to correctly initialize these fields, resulting in an empty string.
Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
|
Thanks for the fix!! xrf: #1440 |
) * [retry] Use pytorch-labs/tokenizers and remove tokenizer/ (#1401) Summary: Retry of #1401 Test Plan: Re-run the repro command in #1413: ``` python3 torchchat.py generate llama3.2-1b-base --prompt "write me a story about a boy and his bear" ``` Reviewers: Subscribers: Tasks: Tags: * Use latest commit Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: * Try enabling _GLIBCXX_USE_CXX11_ABI Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: * Fix PUBLIC issue Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:
Summary: Retry of #1401
Test Plan: Re-run the repro command in #1413:
Reviewers:
Subscribers:
Tasks:
Tags: