Remove external untokenize package. Use tokenize from the Python standard library.

[tdenewiler]

No description provided.

[tdenewiler]

This is required for Python 3.14. The untokenize package used by docformatter has not received any new commits in 6 years.

Tested with

docker run -it python:3.14 bash
cd /tmp/
git clone https://github.com/tdenewiler/docformatter.git
cd docformatter
git checkout untokenize-std-lib
pip install tox
tox -e py314

All tests are passing. If you stay on master branch and do not checkout the new branch then those same tests fail.

[tdenewiler]

I ran the tox tests on Python 3.10, 3.11, 3.12, 3.13, and 3.14.

[adamtheturtle]

The key insight here is that untokenize isn't even used. It is an unnecessary dependency.

FWIW I recommend using deptry to find unused dependencies.

[jonathan343]

+1 for removing this unused dependency

[adamtheturtle]

@weibullguy Hopefully this is a very clear "merge", and a release to fix 3.14 would be welcome