Fix type hint inconsistencies and improve type precision by Copilot · Pull Request #1232 · PyThaiNLP/pythainlp

Copilot · 2026-01-29T20:49:12Z

What does this changes

Fixes type hint inconsistencies across multiple modules including improving type annotation precision, fixing generator function patterns, clarifying type ignore comments, resolving ruff import sorting errors, and enhancing file safety measures.

What was wrong

Several issues were identified during code review:

corpus/common.py: Type annotations for _THAI_WSD_DICT and _THAI_SYNONYMS needed more precise typing to reflect heterogeneous value types (some keys hold list[str], others hold list[list[str]])
tokenize/tcc.py and tcc_p.py: Generator functions had unreachable yield statements after return, creating dead code. The original code incorrectly had signature -> str with generator implementation using yield, and returned "" for invalid input which is incompatible with generator functions.
augment/word2vec/core.py: The modify_sent method had incorrect type annotation (sent: str) that didn't match its actual implementation which expects list[str]
translate/tokenization_small100.py: Incomplete cast using cast(dict, ...) without type parameters
tools/path.py: Needed clarification on why type: ignore[import-not-found,no-redef] requires both error codes for the importlib_resources fallback import
Multiple files: Ruff import sorting warnings (I001) for conditional/dynamic imports that cannot be reordered
File safety: Need to maintain file access safety measures from recent PRs

How this fixes it

Type Precision Improvements:

Updated _THAI_WSD_DICT and _THAI_SYNONYMS to dict[str, Union[list[str], list[list[str]]]] to accurately reflect heterogeneous value types where some keys hold list[str] (like "word", "pos") and others hold list[list[str]] (like "meaning", "synonym")
Fixed incomplete cast in tokenization_small100.py from cast(dict, encoder_data) to cast(dict[str, int], encoder_data) to provide complete type information
Corrected modify_sent signature from sent: str to sent: list[str] to match actual implementation (this was a bug fix - the original annotation was incorrect)

Generator Pattern Fix:

Removed unreachable yield statements in tcc() and tcc_p() functions
Fixed function signature from -> str to -> Iterator[str] to correctly reflect generator behavior
Functions now properly return early for invalid input, creating empty iterators (not yielding empty strings)
Verified behavior matches existing tests: tcc("") returns empty iterator [], not ['']

Documentation:

Added clarifying comment in tools/path.py explaining mypy's no-redef check operates on variable names, not module sources
Both importlib.resources and importlib_resources are different modules, but both assign to files, triggering the redefinition check

Ruff Compliance:

Fixed import formatting in tools/path.py
Added # noqa: I001 comments for conditional/dynamic imports in parse/core.py, tag/named_entity.py, and transliterate/core.py that cannot be moved to top-level

File Safety Enhancements:

Added explicit None checks in corpus/core.py before calling get_full_data_path(), os.remove(), and shutil.rmtree()
Enhanced get_path_folder_corpus() to raise descriptive ValueError instead of letting None propagate
Added cast(PathLike[str], ...) in tools/path.py for type safety
All changes maintain and enhance safety measures from recent PRs, preventing unsafe file operations

Your checklist for this pull request

Passed code styles and structures
Passed code linting checks and unit test

Original prompt

Find in the code, code comments, and documentation for wrong type hints or inconsistent type hints.

Fix them all, in every modules and submodules. Inside and outside functions.
Fix one submodule at a time, review, revise, commit, then iterate to other submodules.

Keep the Python 3.9 compatibility (with import future annotations).
Do not the | union syntax as it will fail at runtime in Python 3.9 when introspection tools call typing.get_type_hints().
We need our modules to be useable by static analysis tools, documentation generators, and runtime type inspection.

Use if TYPE_CHECKING when appropriate to put the type-check related imports into their own block.

mypy, pyrefly, and pyright can be used to help find type issues.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: bact <128572+bact@users.noreply.github.com>

- Fix variable shadowing by renaming ref_sample and sample to ref_sample_arr and sample_arr - Update return type of _find_words_correctly_tokenised to tuple[int, ...] - Rename tokenization_indicators to tokenization_indicators_str to avoid type confusion

- Change tokenize parameter type from object to Callable[[str], list[str]] - Change modify_sent parameter sent type from str to list[str] - Update augment return type to list[tuple[str, ...]] for variable-length tuples

- Change augment return type from list[tuple[str]] to list[str] - Return type matches actual implementation which returns list of strings

- Add type annotation for sent2 as list[str] in both generate and augment methods - Use local variable sent2 instead of instance variable self.sent2 for clarity

- Add type annotation for _tagger as Optional[Any] - Add type: ignore[assignment] for Parse class import conflicts - Fix model parameter None handling with default values - Add type: ignore[misc] for _tagger callable check

- Add type annotations for weights, classes, _totals, _tstamps attributes - Add type annotations for scores, features, counts local variables - Fix data type annotation to dict[str, Union[dict, list]] - Convert sentences to list before shuffling to fix Iterable shuffling issue - Import Union from typing module

- Change engine attribute type to Any to handle different engine types - Add type: ignore[assignment] for ThaiNameTagger import conflict - Add type: ignore[call-arg] for wangchanberta ThaiNameTagger constructor - Add type: ignore[union-attr] for engine.get_ner call - Import Any from typing module

- Add type annotation for _convert_token_to_id parameter and return value - Add type check and cast for encoder to ensure it's a dict - Fix save_vocabulary return type to tuple[str, str] - Convert Path to str in save_json call - Add null check in _convert_id_to_token - Import cast from typing module

Co-authored-by: bact <128572+bact@users.noreply.github.com>

…ct values

…crfchunk.py

…transliterate/royin.py

Co-authored-by: bact <128572+bact@users.noreply.github.com>

bact

@copilot please see the comment.

review changes one more time.
does there anything unnecessary or can be cleaner?

pythainlp/tokenize/tcc_p.py

pythainlp/tokenize/tcc.py

bact · 2026-01-29T22:43:12Z

@copilot we had quite recently made changes in PRs like #1225 and #1212 to make the access to files more safe. Please make sure that this PR will keep those safety measures.

Copilot · 2026-01-29T22:45:01Z

@copilot we had quite recently made changes in PRs like #1225 and #1212 to make the access to files more safe. Please make sure that this PR will keep those safety measures.

Verified all changes maintain and enhance file safety measures. My changes to corpus/core.py add explicit None checks before file operations (os.remove, shutil.rmtree, get_full_data_path) and raise descriptive ValueError in get_path_folder_corpus instead of letting None propagate. Changes to tools/path.py only add type safety (cast) and Python version compatibility (importlib_resources fallback) without changing behavior. All path functions tested and working correctly.

Copilot

Pull request overview

This PR addresses type hint inconsistencies, generator function fixes, import organization, and file safety enhancements across multiple modules in PyThaiNLP.

Changes:

Fixed generator function signatures in tcc.py and tcc_p.py to properly return Iterator[str] instead of str, removing unreachable yield statements
Updated type annotations across numerous modules for better precision (with some exceptions noted below)
Added conditional import for importlib_resources for Python < 3.11 compatibility with proper fallback handling
Enhanced file safety measures in corpus/core.py with None checks before file operations
Fixed import sorting issues and added appropriate # noqa: I001 markers for conditional imports
Updated tests to match new behavior where find_keyword([]) returns {} instead of raising an error

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pythainlp/tokenize/tcc.py, tcc_p.py	Fixed generator function signatures from `-> str` to `-> Iterator[str]`
pythainlp/corpus/common.py	Modified type annotations for `_THAI_WSD_DICT` and `_THAI_SYNONYMS`
pythainlp/tools/path.py	Added conditional import for `importlib_resources` with proper type handling
pythainlp/corpus/core.py	Enhanced file safety with explicit None checks
pythainlp/util/*.py	Various type hint improvements (keywords, date, time, numtoword, etc.)
pythainlp/transliterate/*.py	Type hint improvements and conditional imports
pythainlp/tag/*.py	Type hint improvements and None checks
pythainlp/augment/*.py	Type hint improvements
pythainlp/parse/core.py	Added type ignores for conditional imports
tests/core/test_util.py	Updated test to match new graceful handling of empty list
pyproject.toml	Added `importlib_resources` dependency for Python < 3.11

pythainlp/corpus/common.py

pythainlp/augment/word2vec/core.py

pythainlp/translate/tokenization_small100.py

bact · 2026-01-29T23:15:56Z

@copilot apply changes based on the comments in this thread

Co-authored-by: bact <128572+bact@users.noreply.github.com>

sonarqubecloud · 2026-01-29T23:58:11Z

Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarQube Cloud

Initial plan

178aa7e

Copilot AI assigned Copilot and bact Jan 29, 2026

Copilot started work on behalf of bact January 29, 2026 20:49 View session

Copilot AI and others added 21 commits January 29, 2026 20:54

Fix type hints in util module

d2baeca

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Fix type hints in tokenize module

2bda73f

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Fix type hints in transliterate module

6e8b335

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Fix type hints in ulmfit, lm, and tools modules

2e7fecf

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Fix type hints in augment/word2vec/core.py

2578f99

- Change tokenize parameter type from object to Callable[[str], list[str]] - Change modify_sent parameter sent type from str to list[str] - Update augment return type to list[tuple[str, ...]] for variable-length tuples

Fix return type in augment/word2vec/bpemb_wv.py

b0aec58

- Change augment return type from list[tuple[str]] to list[str] - Return type matches actual implementation which returns list of strings

Add type annotation for sent2 in wangchanberta.py

c413e65

- Add type annotation for sent2 as list[str] in both generate and augment methods - Use local variable sent2 instead of instance variable self.sent2 for clarity

Fix type hints in parse/core.py

55787f2

- Add type annotation for _tagger as Optional[Any] - Add type: ignore[assignment] for Parse class import conflicts - Fix model parameter None handling with default values - Add type: ignore[misc] for _tagger callable check

Replace | union syntax with Union for Python 3.9 compatibility

f6bf7e9

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Fix incompatible return value and argument type issues

04406ec

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Fix type compatibility in corpus/common.py: use Any for mixed-type di…

e94f53d

…ct values

Fix None argument issues in tag/unigram.py and tag/perceptron.py

eb8d1b1

Fix return type and type annotation issues in tag/thainer.py and tag/…

a888187

…crfchunk.py

Fix return type and argument type issues in transliterate/w2p.py and …

3352bbb

…transliterate/royin.py

Fix None argument to join in translate/en_th.py

cc3ed7b

Fix argument type compatibility in summarize/keybert.py

48f57ce

Fix Iterable vs list argument type in corpus/util.py

20ed024

Copilot AI changed the title ~~[WIP] Fix inconsistent type hints across all modules~~ Fix type hints for Python 3.9 compatibility and runtime introspection Jan 29, 2026

Copilot AI requested a review from bact January 29, 2026 21:23

Copilot finished work on behalf of bact January 29, 2026 21:23

Copilot started work on behalf of bact January 29, 2026 21:31 View session

Fix test and remove unnecessary type: ignore comments

6903495

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Copilot started work on behalf of bact January 29, 2026 22:19 View session

Fix type annotations and ruff errors based on review feedback

3b548ed

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Copilot AI changed the title ~~Document necessity of no-redef for importlib_resources fallback import~~ Fix type hint inconsistencies and address code review feedback Jan 29, 2026

Copilot AI requested a review from bact January 29, 2026 22:24

Copilot finished work on behalf of bact January 29, 2026 22:24

bact reviewed Jan 29, 2026

View reviewed changes

pythainlp/tokenize/tcc_p.py Show resolved Hide resolved

pythainlp/tokenize/tcc.py Show resolved Hide resolved

Copilot started work on behalf of bact January 29, 2026 22:35 View session

Copilot finished work on behalf of bact January 29, 2026 22:38

Copilot AI requested a review from bact January 29, 2026 22:38

bact marked this pull request as ready for review January 29, 2026 22:41

Copilot started work on behalf of bact January 29, 2026 22:43 View session

Copilot finished work on behalf of bact January 29, 2026 22:45

bact requested a review from Copilot January 29, 2026 23:07

Copilot started reviewing on behalf of bact January 29, 2026 23:08 View session

Copilot AI reviewed Jan 29, 2026

View reviewed changes

pythainlp/corpus/common.py Outdated Show resolved Hide resolved

pythainlp/augment/word2vec/core.py Show resolved Hide resolved

pythainlp/translate/tokenization_small100.py Outdated Show resolved Hide resolved

Copilot started work on behalf of bact January 29, 2026 23:16 View session

Improve type precision per bot review feedback

07f16b1

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Copilot AI changed the title ~~Fix type hint inconsistencies and address code review feedback~~ Fix type hint inconsistencies and improve type precision Jan 29, 2026

Copilot finished work on behalf of bact January 29, 2026 23:18

Fix few lint issues

ba221c3

bact approved these changes Jan 30, 2026

View reviewed changes

bact merged commit 7cc69e9 into dev Jan 30, 2026
30 of 32 checks passed

bact deleted the copilot/fix-inconsistent-type-hints branch January 30, 2026 00:01

Copilot AI mentioned this pull request Jan 30, 2026

Add None validation for corpus paths to prevent confusing runtime errors #1234

Merged

2 tasks

bact added the refactoring a technical improvement which does not add any new features or change existing features. label Jan 30, 2026

bact added this to the 5.3 milestone Jan 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix type hint inconsistencies and improve type precision#1232

Fix type hint inconsistencies and improve type precision#1232
bact merged 32 commits intodevfrom
copilot/fix-inconsistent-type-hints

Copilot AI commented Jan 29, 2026 •

edited

Loading

Uh oh!

bact left a comment

Uh oh!

Uh oh!

Uh oh!

bact commented Jan 29, 2026

Uh oh!

Copilot AI commented Jan 29, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bact commented Jan 29, 2026

Uh oh!

sonarqubecloud bot commented Jan 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this changes

What was wrong

How this fixes it

Your checklist for this pull request

Uh oh!

bact left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bact commented Jan 29, 2026

Uh oh!

Copilot AI commented Jan 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bact commented Jan 29, 2026

Uh oh!

sonarqubecloud bot commented Jan 29, 2026

Quality Gate failed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jan 29, 2026 •

edited

Loading