Custom ingestion. Html tables generated too wordy and inefficient.

[evan2k]

It seems that after calling Document Intelligence, python code would convert json tables returned to html tables. However, this seems inefficient when sent to the LLM, as html is too wordy, not only costing more in terms of tokens send, but also many medium sized tables are sent "broken" to the LLM due to chunking.
Should the app be modified to use something more efficient like csv or markup instead of html? The viewing experience of course will be affected.
Your thoughts? Thank you.

[pamelafox]

We did look into this a bit, here's some relevant research on HTML vs plaintext vs markdown:
https://arxiv.org/abs/2411.02959
https://arxiv.org/abs/2406.08100

The reason that we're currently picking HTML for tables is that HTML can convey more complex table structure (like row spans and cell spans) and the research seems to support using HTML for richer information.

I think we could do a better job with the chunking, however. We currently have logic that will start a new chunk if it knows that it had to break a chunk, which means we do ultimately store the full table, but the broken chunks are awkward. I think we'd have similar brokenness with markdown tables, too, by the way.

I think of a few approaches:

• Allow chunks with tables to exceed the token size. This increase costs and risks exceeding context windows, depending how much it exceeds.
• Implement smarter table breaking, only after a tr, with a note that says "table has been truncated"

It'd be good to know what other chunking frameworks are doing, like Langchain.

[evan2k]

Thank you for your comments. I guess ultimately it depends on the nature of your data, if you have many medium to large grid/excel like tabular data you may consider changing to a less wordy structure.
I was experimenting with table with 20-30 rows that listed type of materials used for a product. When asking the LLM to list all materials used for the product, it listed only the materials contained in one chunk, ignoring the rest, but did state that there may be other materials not listed

[pamelafox]

Yeah, makes sense. This is the method that would need changing:

DocumentAnalysisParser.table_to_html() in pdfparser.py

You could put a table_to_csv() in there and try that instead.

If your table are still getting split, then you can modify the splitting code, look for
"last_figure_start = section_text.rfind("<figure")" in textsplitter.py

We could add it as an option to the main repo if you make it configurable via an environment variable.

[ms-johnalex]

Yeah, makes sense. This is the method that would need changing:

> DocumentAnalysisParser.table_to_html() in pdfparser.py

You could put a table_to_csv() in there and try that instead.

If your table are still getting split, then you can modify the splitting code, look for "last_figure_start = section_text.rfind("<figure")" in textsplitter.py

We could add it as an option to the main repo if you make it configurable via an environment variable.

How does it do with complex pdf tables?

[evan2k]

I only tested with simple grid like long but simple tables so far.

[ms-johnalex]

I'm playing around with https://github.com/DS4SD/docling. I'll let you know my results

[javith2dj]

We had the same issue with Azure Document Intelligence, so we used pandas to convert the table to markdown. And we created a function that:

1. Identifies Table Headers: Recognizes the header row and the separator row in Markdown tables.

2. Splits Tables into Chunks: Divides the table into smaller parts based on a specified number of rows.

3. Ensures Headers are Included in Each Chunk: Prepends the header to each chunk to maintain table integrity.

We tested it on table with more than 300 rows of tabular data, the column values were mostly text so the llm summarization improved a lot.

[cloudhunnter]

We had the same issue with Azure Document Intelligence, so we used pandas to convert the table to markdown. And we created a function that:

1. Identifies Table Headers: Recognizes the header row and the separator row in Markdown tables.
2. Splits Tables into Chunks: Divides the table into smaller parts based on a specified number of rows.
3. Ensures Headers are Included in Each Chunk: Prepends the header to each chunk to maintain table integrity.

We tested it on table with more than 300 rows of tabular data, the column values were mostly text so the llm summarization improved a lot.

Sounds interesting. Any chance of show your code changes? I do have big problem with rowspans and colspans, where the extraction somehow combines cells. Would like to test your solution.

[javith2dj]

We had the same issue with Azure Document Intelligence, so we used pandas to convert the table to markdown. And we created a function that:

1. Identifies Table Headers: Recognizes the header row and the separator row in Markdown tables.
2. Splits Tables into Chunks: Divides the table into smaller parts based on a specified number of rows.
3. Ensures Headers are Included in Each Chunk: Prepends the header to each chunk to maintain table integrity.

We tested it on table with more than 300 rows of tabular data, the column values were mostly text so the llm summarization improved a lot.

Sounds interesting. Any chance of show your code changes? I do have big problem with rowspans and colspans, where the extraction somehow combines cells. Would like to test your solution.

Sure, these are the functions. I hope it helps.

import re
import pandas as pd
from tabulate import tabulate
from typing import List

def split_markdown_tables_with_headers(markdown_text: str, max_rows: int = 50) -> List[str]:
  """
  Splits Markdown tables into chunks with a maximum number of rows each,
  ensuring that each chunk includes the table header.

  Parameters:
  - markdown_text: The full Markdown text containing one or more tables.
  - max_rows: Maximum number of data rows per chunk (excluding header).

  Returns:
  - A list of Markdown strings, each representing a chunked table.
  """

  # Regular expression to identify Markdown tables
  table_pattern = re.compile(
      r'(\|.*\|\n\|[-| :]+\|\n(?:\|.*\|\n?)*)',
      re.MULTILINE
  )
  
  tables = table_pattern.findall(markdown_text)
  chunks = []

  for table in tables:
      lines = table.strip().split('\n')
      if len(lines) < 2:
          continue  # Not a valid table
      
      header = lines[0]
      separator = lines[1]
      data_rows = lines[2:]

      # Split data_rows into chunks
      for i in range(0, len(data_rows), max_rows):
          chunk_rows = data_rows[i:i + max_rows]
          chunk = '\n'.join([header, separator] + chunk_rows)
          chunks.append(chunk)
  
  return chunks

def excel_to_markdown_with_chunking(excel_path: str, sheet_name=0, max_rows=50) -> List[str]:
  """
  Reads an Excel file, converts it to Markdown, and splits the Markdown into chunks.

  Parameters:
  - excel_path: Path to the Excel file (.xlsx).
  - sheet_name: Sheet name or index to extract.
  - max_rows: Maximum number of data rows per Markdown chunk.

  Returns:
  - A list of Markdown table chunks.
  """

  # Load Excel file into a Pandas DataFrame
  df = pd.read_excel(excel_path, sheet_name=sheet_name)
  
  # Convert to Markdown
  markdown_text = tabulate(df, headers='keys', tablefmt='pipe', showindex=False)
  
  # Split into chunks while preserving headers
  markdown_chunks = split_markdown_tables_with_headers(markdown_text, max_rows=max_rows)
  
  return markdown_chunks