feat: parse padding settings from tokenizer_config.json

[besaleli]

No description provided.

[codspeed-hq]

CodSpeed Performance Report

Merging #183 will not alter performance

_{Comparing 179-optionally-embed-tokenizer_configjson-2 (a1e255a) with main (05b9c67)}

Summary

✅ 20 untouched
⏩ 20 skipped¹

Footnotes

1. 20 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[codecov-commenter]

Codecov Report

❌ Patch coverage is 83.33333% with 46 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
encoderfile/src/tokenizer.rs	79.41%	42 Missing ⚠️
encoderfile-core/src/runtime/tokenizer.rs	50.00%	2 Missing ⚠️
encoderfile-core/src/factory.rs	0.00%	1 Missing ⚠️
encoderfile/src/config.rs	98.14%	1 Missing ⚠️

Files with missing lines	Coverage Δ
encoderfile-core/src/dev_utils/mod.rs	100.00% <100.00%> (ø)
encoderfile/src/transforms/validation/embedding.rs	84.00% <100.00%> (+0.21%)	⬆️
encoderfile/src/transforms/validation/mod.rs	88.39% <100.00%> (+0.31%)	⬆️
...le/src/transforms/validation/sentence_embedding.rs	84.81% <100.00%> (+0.19%)	⬆️
...c/transforms/validation/sequence_classification.rs	86.84% <100.00%> (+0.17%)	⬆️
.../src/transforms/validation/token_classification.rs	86.84% <100.00%> (+0.17%)	⬆️
encoderfile-core/src/factory.rs	0.00% <0.00%> (ø)
encoderfile/src/config.rs	89.55% <98.14%> (+1.76%)	⬆️
encoderfile-core/src/runtime/tokenizer.rs	57.14% <50.00%> (-16.55%)	⬇️
encoderfile/src/tokenizer.rs	79.41% <79.41%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.