Better documentation

[gsakkis]

As much as this project is an awesome piece of work, I find it hard to navigate and rely on without a lot of trial and error, mostly due to lacking or (worse) misleading documentation. A few recent examples:

• The ExtractionConfig docs:
  • Mention The force_ocr parameter has been deprecated in favor of the new ocr configuration object.
    This is not true, at least as of the latest 4.2.9 release few hours ago. force_ocr is still required to always call tesseract.
  • Lists 7 fields instead of 16 that exist as of now
• html_options (mentioned only in the migration guide) is a dict instead of a dataclass, making typos in keys/values undetectable.
• While looking around in the code I stumbled on HtmlConversionOptions in _internal_bindings but this doesn't exist at runtime:
  AttributeError: module 'kreuzberg._internal_bindings' has no attribute 'HtmlConversionOptions'
• I needed to remove inline images from html; again by a combination of digging in the code and trial and error I discovered an undocumented convert_as_inline option that seems to do the work, though i'm not sure if it's by design or by accident.

Perhaps that's the price of massive AI-generated code but any improvement in the overall documentation & consistency would be amazing!

[Goldziher]

Hi there,

Reasonable indeed 🤣. Ok, I didn't realize this is soo bad. I'll do a pass through the docs.

As for convert with inline images, it's part of the documented API for html-to-markdown - also in this GitHub org.

[gsakkis]

Where is the documented API for html-to-markdown? If github search is to be trusted, convert_as_inline doesn't appear in any markdown file.

[Goldziher]

Hi there,

Reasonable indeed 🤣. Ok, I didn't realize this is soo bad. I'll do a pass through the docs.

As for convert with inline images, it's part of the documented API for html-to-markdown - also in this GitHub org.

@gsakkis please give me as comprehensive a review as you can

You are welcome to raise PRS as well.