Feature: Extend Support for Multiple File Types in OCR/Extraction Service #15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit implements two major enhancements to the Unsiloed-chunker: 1. Extended File Type Support: - Added support for DOC files using docx2txt - Added support for Excel formats (XLSX, XLS) using openpyxl and xlrd - Added support for OpenDocument formats (ODT, ODS, ODP) using odfpy - Added support for plain text (TXT) and rich text (RTF) using striprtf - Added support for e-book format (EPUB) using ebooklib and BeautifulSoup4 - Updated file type detection in __init__.py and chunking_routes.py - Updated process_document_chunking to handle all new file types - Added necessary dependencies to requirements.txt and setup.py 2. Multiple OCR/LLM Model Support: - Verified support for multiple model providers beyond OpenAI - Confirmed implementation of Anthropic (Claude) models - Confirmed implementation of Hugging Face models (including Mistral) - Confirmed implementation of local models via llama.cpp - Ensured proper configuration and model selection 3. Documentation: - Updated README.md to reflect new supported file types - Updated documentation for model provider options - Added details about new dependencies These changes significantly enhance the flexibility and utility of the Unsiloed-chunker by supporting a wider range of file formats and model providers.
5 tasks
Kunal-Darekar
force-pushed
the
feature/extended-file-types-and-models
branch
2 times, most recently
from
bb82dc9 to
968393a
Compare
Contributor
|
@Kunal-Darekar please attach a video |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This pull request implements the enhancements outlined in issue #3: "Extend Support for Multiple File Types in OCR/Extraction Service". It significantly expands the file type support beyond the current PDF, DOCX, and PPTX formats.
Changes
Added Support for New File Types:
Microsoft Office Formats:
.DOC(older Word format) using docx2txt.XLSX(Excel) using openpyxl.XLS(older Excel format) using xlrdOpenDocument Formats:
.ODT(OpenDocument Text) using odfpy.ODS(OpenDocument Spreadsheet) using odfpy.ODP(OpenDocument Presentation) using odfpyText Formats:
.TXTfiles.RTFfiles using striprtfE-book Format:
.EPUBformat using ebooklib and BeautifulSoup4Implementation Details:
New Extraction Functions:
extract_text_from_doc()for DOC filesextract_text_from_xlsx()andextract_text_from_xls()for Excel filesextract_text_from_odt(),extract_text_from_ods(), andextract_text_from_odp()for OpenDocument formatsextract_text_from_txt()andextract_text_from_rtf()for text formatsextract_text_from_epub()for EPUB filesFile Type Detection:
__init__.pyfor both URL and local file pathschunking_routes.pyfor API uploadsProcessing Logic:
process_document_chunking()inservices/chunking.pyto handle all new file typesDependencies:
requirements.txt:setup.pywith the same dependenciesDocumentation:
Testing
All new file type extraction functions have been implemented with proper error handling and logging. The implementation follows the same patterns as the existing extraction functions for PDF, DOCX, and PPTX.
Resolves
This PR resolves issue #3: "Extend Support for Multiple File Types in OCR/Extraction Service"
/claim #3