Skip to content
This repository was archived by the owner on Sep 10, 2025. It is now read-only.

Wording sentencepiece.cpp #1435

Closed
wants to merge 11 commits into from
Closed

Wording sentencepiece.cpp #1435

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6cac9ea
Wording sentencepiece.cpp
mikekgfb Dec 20, 2024
ad5ba1e
Merge branch 'pytorch:main' into patch-32
mikekgfb Dec 22, 2024
8195034
List tokenizer file in sentencepiece.cpp
mikekgfb Dec 22, 2024
f7b3df6
Update sentencepiece.cpp
mikekgfb Dec 22, 2024
e5635fd
Update sentencepiece.cpp
mikekgfb Dec 23, 2024
fb24e0b
Merge branch 'pytorch:main' into patch-32
mikekgfb Dec 23, 2024
871a874
Update sentencepiece.cpp
mikekgfb Dec 23, 2024
9f6b198
Update sentencepiece.cpp
mikekgfb Dec 23, 2024
ece7b45
Update sentencepiece.cpp
mikekgfb Dec 23, 2024
55c48b5
Merge branch 'pytorch:main' into patch-32
mikekgfb Dec 29, 2024
8455d58
Update build_native.sh
mikekgfb Dec 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion tokenizer/sentencepiece.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
#include <tokenizer.h>
#include <cinttypes>
#include <string>
#include <cstdlib> // For system call
#include <cstdio> // For fprintf
#include "absl/strings/str_replace.h"

const char kSpaceSymbol[] = "\xe2\x96\x81";
Expand All @@ -38,7 +40,15 @@ void SPTokenizer::load(const std::string& tokenizer_path) {
// read in the file
const auto status = _processor->Load(tokenizer_path);
if (!status.ok()) {
fprintf(stderr, "couldn't load %s\n. If this tokenizer artifact is for llama3, please pass `-l 3`.", tokenizer_path.c_str());
// Execute 'ls -al' on the tokenizer path
Copy link
Contributor

@Jack-Khuu Jack-Khuu Dec 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the print, great for debugging

Looks like the ls is spitting out the root torchchat directory instead of tokenize path which is curious

Copy link
Contributor Author

@mikekgfb mikekgfb Dec 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would explain why the tokenizer can't be loaded, and the AOTI tests keep failing. #1429

Copy link
Contributor Author

@mikekgfb mikekgfb Dec 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added set -x to the command to echo the ls to make absolutely sure the path is not getting corrupted somehow (not sure how it would, but belts and suspenders)

Copy link
Contributor

@Jack-Khuu Jack-Khuu Dec 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neutral signal- looks like the arg is not being picked up by ls (which would explain why it just shows PWD)

Copy link
Contributor Author

@mikekgfb mikekgfb Dec 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is sooooo weird! Want to add a print of command and rerun? Maybe there’s some magic character that causes indigestion for the shell running La; and the tokenizer model load?

Copy link
Contributor Author

@mikekgfb mikekgfb Dec 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added print of command before execution

Copy link
Contributor Author

@mikekgfb mikekgfb Dec 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split C style strong conversion and c++ const as convert followed by append

std::string command = "set -x ; ls -al ";
command += tokenizer_path;
fprintf(stderr, "Command: '%s'.\nlen: %d\n", command.c_str(), strlen(command.c_str()));
int ret = system(command.c_str());
if (ret != 0) {
fprintf(stderr, "Failed to execute 'ls -al' on path: %s\n", tokenizer_path.c_str());
}
fprintf(stderr, "Could not load `%s`.\n If this tokenizer artifact is for llama3, please pass `-l 3`.", tokenizer_path.c_str());
exit(EXIT_FAILURE);
}
// load vocab_size, bos_tok, eos_tok
Expand Down
2 changes: 1 addition & 1 deletion torchchat/utils/scripts/build_native.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ popd
if [[ "$TARGET" == "et" ]]; then
cmake -S . -B ./cmake-out -DCMAKE_PREFIX_PATH=`python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DLINK_TORCHAO_OPS="${LINK_TORCHAO_OPS}" -DET_USE_ADAPTIVE_THREADS=ON -DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=1" -G Ninja
else
cmake -S . -B ./cmake-out -DCMAKE_PREFIX_PATH=`python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DLINK_TORCHAO_OPS="${LINK_TORCHAO_OPS}" -DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" -G Ninja
cmake -S . -B ./cmake-out -DCMAKE_PREFIX_PATH=`python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DLINK_TORCHAO_OPS="${LINK_TORCHAO_OPS}" -DCMAKE_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=1" -G Ninja
fi
cmake --build ./cmake-out --target "${TARGET}"_run

Expand Down
Loading