fix: Improve HTML entity sanitization for invalid XML characters (#348)

[Rithboss]

Fixes #348

Summary

This PR enhances the HTML sanitization logic to properly handle invalid XML character entities, preventing when parsing PACER dockets that contain escape sequences or other invalid XML characters.

Problem

PACER dockets sometimes contain invalid XML characters (like - ESC character) that cause parsing failures with error:

lxml.etree.XMLSyntaxError: PCDATA invalid Char value 27

The existing clean_html() function had limited support for removing invalid character entities, only handling � through �.

Solution

• Enhanced the regex pattern in clean_html() to remove ALL invalid XML character entities
• Handles decimal entities: � through �, �, , � through �
• Handles hexadecimal entities: � through � (excluding valid ones)
• Preserves valid XML characters: tab (0x09), LF (0x0A), CR (0x0D)
• The existing Unicode character filtering already handles raw invalid characters

Testing

Added comprehensive test suite (tests/local/test_xml_character_sanitization.py) with 6 test cases:

• ✅ ESC character (\x1b) removal
• ✅ Various invalid XML characters (NULL, SOH, STX, BS, VT, FF, SO, ESC, US)
• ✅ Valid XML character preservation (tab, newline, carriage return)
• ✅ HTML entity handling (�, etc.)
• ✅ Integration with strip_bad_html_tags_insecure()
• ✅ Real-world docket text scenarios

All tests pass (6/6).

Changes

• Modified: juriscraper/lib/html_utils.py
• Modified: CHANGES.md
• Added: tests/local/test_xml_character_sanitization.py

Backward Compatibility

This change is fully backward compatible. It only removes invalid characters that would have caused parsing errors anyway.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}