Port newmm tokenizer from PyThaiNLP to C

[Copilot]

Implements newmm (New Maximum Matching) word segmentation algorithm from PyThaiNLP in C with similar API design.

Implementation

• Trie data structure (src/trie.c) - Prefix matching with UTF-8 support for dictionary lookups
• Thai Character Cluster (src/tcc.c) - TCC boundary detection following Theeramunkong et al. 2000 rules
• newmm algorithm (src/newmm.c) - Maximal matching constrained by TCC boundaries, handles Thai/English/numeric text

API

// Segment Thai text with optional custom dictionary
char** newmm_segment(const char* text, const char* dict_path, int* token_count);
void newmm_free_result(char** tokens, int token_count);

Usage mirrors PyThaiNLP:

// C
int count;
char** tokens = newmm_segment("ฉันไปโรงเรียน", "dict.txt", &count);
// ['ฉัน', 'ไป', 'โรงเรียน']
newmm_free_result(tokens, count);

# Python (PyThaiNLP)
tokens = word_tokenize("ฉันไปโรงเรียน", engine="newmm")
# ['ฉัน', 'ไป', 'โรงเรียน']

Testing

• Test suite with 8 cases covering Thai text, mixed content, edge cases
• Example program demonstrating usage
• Sample dictionary included in data/

CI/CD

• GitHub Actions workflow (.github/workflows/test.yml) - Automated testing on Ubuntu (gcc) and macOS (clang)
• Runs on push and pull requests to ensure code quality
• CI badge added to README for build status visibility

Original prompt

Port newmm fro pythainlp to C that API should like pythainlp

GitHub: https://github.com/PyThaiNLP/pythainlp

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[wannaphong]

@copilot Add Github action for testing

[Copilot]

@copilot Add Github action for testing

Added GitHub Actions workflow in commit dcba1f6. The CI pipeline now:

• Tests on Ubuntu (gcc) and macOS (clang)
• Runs on push and pull requests
• Executes full test suite and example programs
• Uploads artifacts on failure for debugging

CI badge added to README.