Add Markdown/Text Parser (#1381)

* add markdown parser

* add example file in markdown

* fix formatting and linting errors

* rename company in markdown example file

* rename markdownparser to textparser

* fix typo

* remove empty space from test case

* add txt support + fix import

* Apply suggestions from code review

---------

Co-authored-by: Pamela Fox <[email protected]>

Author: yuvalyaron

data/Contoso_Electronics_Company_Overview.md

@@ -0,0 +1,48 @@
+# Contoso Electronics
+
+*Disclaimer: This content is generated by AI and may not accurately represent factual information about any real entity. Use this information with caution and verify details from reliable sources.*
+
+## History
+
+Contoso Electronics, a pioneering force in the tech industry, was founded in 1985 by visionary entrepreneurs with a passion for innovation. Over the years, the company has played a pivotal role in shaping the landscape of consumer electronics.
+
+| Year | Milestone |
+|------|-----------|
+| 1985 | Company founded with a focus on cutting-edge technology |
+| 1990 | Launched the first-ever handheld personal computer |
+| 2000 | Introduced groundbreaking advancements in AI and robotics |
+| 2015 | Expansion into sustainable and eco-friendly product lines |
+
+## Company Overview
+
+At Contoso Electronics, we take pride in fostering a dynamic and inclusive workplace. Our dedicated team of experts collaborates to create innovative solutions that empower and connect people globally.
+
+### Core Values
+
+- **Innovation:** Constantly pushing the boundaries of technology.
+- **Diversity:** Embracing different perspectives for creative excellence.
+- **Sustainability:** Committed to eco-friendly practices in our products.
+
+## Vacation Perks
+
+We believe in work-life balance and understand the importance of well-deserved breaks. Our vacation perks are designed to help our employees recharge and return with renewed enthusiasm.
+
+| Vacation Tier | Duration | Additional Benefits |
+|---------------|----------|---------------------|
+| Standard      | 2 weeks  | Health and wellness stipend |
+| Senior        | 4 weeks  | Travel vouchers for a dream destination |
+| Executive     | 6 weeks  | Luxury resort getaway with family |
+
+## Employee Recognition
+
+Recognizing the hard work and dedication of our employees is at the core of our culture. Here are some ways we celebrate achievements:
+
+- Monthly "Innovator of the Month" awards
+- Annual gala with awards for outstanding contributions
+- Team-building retreats for high-performing departments
+
+## Join Us!
+
+Contoso Electronics is always on the lookout for talented individuals who share our passion for innovation. If you're ready to be part of a dynamic team shaping the future of technology, check out our [careers page](http://www.contoso.com) for exciting opportunities.
+
+[Learn more about Contoso Electronics!](http://www.contoso.com)

scripts/prepdocs.py

@@ -29,6 +29,7 @@
 from prepdocslib.parser import Parser
 from prepdocslib.pdfparser import DocumentAnalysisParser, LocalPdfParser
 from prepdocslib.strategy import DocumentAction, SearchInfo, Strategy
+from prepdocslib.textparser import TextParser
 from prepdocslib.textsplitter import SentenceTextSplitter, SimpleTextSplitter
 
 
@@ -87,6 +88,8 @@ async def setup_file_strategy(credential: AsyncTokenCredential, args: Any) -> St
         ".tiff": FileProcessor(doc_int_parser, sentence_text_splitter),
         ".bmp": FileProcessor(doc_int_parser, sentence_text_splitter),
         ".heic": FileProcessor(doc_int_parser, sentence_text_splitter),
+        ".md": FileProcessor(TextParser(), sentence_text_splitter),
+        ".txt": FileProcessor(TextParser(), sentence_text_splitter),
     }
     use_vectors = not args.novectors
     embeddings: Optional[OpenAIEmbeddings] = None

scripts/prepdocslib/textparser.py

@@ -0,0 +1,30 @@
+import re
+from typing import IO, AsyncGenerator
+
+from .page import Page
+from .parser import Parser
+
+
+def cleanup_data(data: str) -> str:
+    """Cleans up the given content using regexes
+    Args:
+        data: (str): The data to clean up.
+    Returns:
+        str: The cleaned up data.
+    """
+    # match two or more newlines and replace them with one new line
+    output = re.sub(r"\n{2,}", "\n", data)
+    # match two or more spaces that are not newlines and replace them with one space
+    output = re.sub(r"[^\S\n]{2,}", " ", output)
+
+    return output.strip()
+
+
+class TextParser(Parser):
+    """Parses simple text into a Page object."""
+
+    async def parse(self, content: IO) -> AsyncGenerator[Page, None]:
+        data = content.read()
+        decoded_data = data.decode("utf-8")
+        text = cleanup_data(decoded_data)
+        yield Page(0, 0, text=text)

tests/test_textparser.py

@@ -0,0 +1,60 @@
+import io
+
+import pytest
+
+from scripts.prepdocslib.textparser import TextParser
+
+
+@pytest.mark.asyncio
+async def test_textparser_remove_new_lines():
+    file = io.BytesIO(
+        b"""
+        # Text Example with multiple empty lines
+        this is paragraph 1
+
+
+
+        and this is paragraph 2
+        """
+    )
+    parser = TextParser()
+    pages = [page async for page in parser.parse(file)]
+    assert len(pages) == 1
+    assert pages[0].page_num == 0
+    assert pages[0].offset == 0
+    assert pages[0].text == "# Text Example with multiple empty lines\n this is paragraph 1\n and this is paragraph 2"
+
+
+@pytest.mark.asyncio
+async def test_textparser_remove_white_spaces():
+    file = io.BytesIO(b"              Test multiple  white spaces                  ")
+    parser = TextParser()
+    pages = [page async for page in parser.parse(file)]
+    assert pages[0].text == "Test multiple white spaces"
+
+
+@pytest.mark.asyncio
+async def test_textparser_full():
+    file = io.BytesIO(
+        b"""
+        # Text  Example
+        Some short text here, with bullets:
+        * write code
+        * test code
+        * merge code
+
+
+        ## Subheading
+        Some more text here with a link to Azure.  Here's a the link to [Azure](https://azure.microsoft.com/).
+        """
+    )
+    file.name = "test.md"
+    parser = TextParser()
+    pages = [page async for page in parser.parse(file)]
+    assert len(pages) == 1
+    assert pages[0].page_num == 0
+    assert pages[0].offset == 0
+    assert (
+        pages[0].text
+        == "# Text Example\n Some short text here, with bullets:\n * write code\n * test code\n * merge code\n ## Subheading\n Some more text here with a link to Azure. Here's a the link to [Azure](https://azure.microsoft.com/)."
+    )