Making difflib more maintainable with dividing it into smaller files

[jurgenwigg]

Feature or enhancement

Proposal:

difflib is hardly maintainable due to it's size (over 2k loc). My proposal is to divide it into smaller files. It also means converting it from module to the package but with backward compatibility.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• [WIP] gh-132067 Divide difflib into smaller files for better maintenance #132066

[eendebakpt]

There are several other files in Lib/ even larger and I am not sure moving around the code for smaller sizes is worth the potential downsides (churn, backwards incompatibility). I suggest to open a topic on Python discourse in the ideas or core development section to see whether there is any desire for these code refactorings.

[hugovk]

What sort of maintenance problems are we facing due to size?

A lot of it is comments.

❯ wc -l Lib/difflib.py
    2062 Lib/difflib.py

❯ cloc Lib/difflib.py
       1 text file.
       1 unique file.
       0 files ignored.

github.com/AlDanial/cloc v 2.04  T=0.01 s (77.7 files/s, 160255.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1            290           1001            771
-------------------------------------------------------------------------------

Here are some of the bigger ones by wc -l:

    2062 Lib/difflib.py
    2092 Lib/xml/etree/ElementTree.py
    2121 Lib/http/cookiejar.py
    2137 Lib/urllib/request.py
    2177 Lib/enum.py
    2221 Lib/subprocess.py
    2223 Lib/mailbox.py
    2324 Lib/logging/__init__.py
    2370 Lib/zipfile/__init__.py
    2408 Lib/idlelib/configdialog.py
    2408 Lib/ipaddress.py
    2513 Lib/html/entities.py
    2650 Lib/pdb.py
    2679 Lib/argparse.py
    2730 Lib/_pydatetime.py
    2742 Lib/_pyio.py
    2857 Lib/pydoc.py
    2885 Lib/pickletools.py
    2916 Lib/doctest.py
    2971 Lib/tarfile.py
    3055 Lib/test/support/__init__.py
    3095 Lib/email/_header_value_parser.py
    3188 Lib/unittest/mock.py
    3374 Lib/inspect.py
    3754 Lib/typing.py
    4268 Lib/turtle.py
    4986 Lib/tkinter/__init__.py
    5125 Lib/test/pickletester.py
    6354 Lib/_pydecimal.py
    7281 Lib/test/datetimetester.py
   12859 Lib/pydoc_data/topics.py

[jurgenwigg]

Maybe it's just me, but for me source code file shouldn't be longer than 1k loc. Why? I don't have to scroll hundreds of lines between the changes. Small, functionally-oriented files could be the goal. It also makes changes easier and prevents from accidentally changing, e.g. Differ instead of SequenceMather simply because they are separate files. Keeping tests as a part of documentation is a nice feature but can lead to over documented code for me. Argument, that there are bigger files is not an argument, because we can always find bigger file.

[ZeroIntensity]

It also makes changes easier and prevents from accidentally changing, e.g. Differ instead of SequenceMather simply because they are separate files.

That's what reviews are for 😃.

I'm not sure this is worth doing, even if we determine that difflib is "too big;" it gets very little maintenance anyway.

[jurgenwigg]

I would like to make some changes in that library due to the bug that causes waiting forever for the computation in ndiff,find_longest_match. Maybe I'll just fix that without any additional refactoring but such approach doesn't sound good. Leaving code a little better when you're making changes is always a good thing for me. So for the starting point for me was to make it more readable, maintainable with smaller and more feature oriented files.

[tomasr8]

I think in general we tend to prefer incremental improvements over large-scale changes. Also keep in mind that large refactors have other downsides such as git blame being less useful. Also, people familiar with the module coming back to it after the refactor now have to spend extra time re-learning how the module works.

[jurgenwigg]

Good point - I'm closing this issue. I'll start with fixing bugs and slowly improves code quality.