Update dependencies to support python 3.10 to 3.13

[gsaint]

Add Python 3.10 to 3.13 support by introducing version-conditional dependencies for packages that have breaking changes or lack support across Python versions

Changes

• Replace pycld3 with pycld2 for language detection on Python >= 3.10
  • Very close API and compatibility
  • Reuse pycld3 language ID mapping
• Add spacy 3.x support
  • Add explicit lemmatizer component initialization for blank language models, which is an API change starting at 3.0
  • Use spacy 3.X for python >= 3.10
• Fix pandas 2.0 compatibility: iteritems() → items()
• Fix NumPy 2.0 compatibility: np.NaN → np.nan
• Testing
  • Expand unit test coverage to make sure we keep backward compatibility
  • Add tox configuration to run unit tests across Python 3.6-3.13

Verification

Run unit tests on all supported python versions

tox

Execute unit tests on a single python version

tox py312

On MacOS this is only working on the intel architecture.

• Language detection
  • pycld3 and pycld2 are only used for text longer than 140 chars, otherwise existing code using langid is executed
• Spell checking and text cleaning
  • Verify spaCy tokenization and lemmatization is working on p39 and from p310