feat: add evaluation metric for table extraction (#216)

## Summary

This PR resolves
[CORE-1930](https://unstructured-ai.atlassian.net/browse/CORE-1930). It
adds helpers to compute token ratios between two tables' (actual vs.
prediction). This score goes from 0 (no match at all) to 100 (fully
matched).

## metric sketch

1. represent tables as rows and columns data, with cells spanning
multiple columns or rows occupying the left upper most position while
leaving the rest of the positions empty
2. join the table together as a long string of text by `tab_token`
(often `\t` ) in a row then `row_break_token` (often `\n`) between rows
3. compare the string representations’
`rapidfuzz.fuzz.partial_token_ratio` with tokenizer that tokenizes words
and each number is its own token
4. do 2-3 again but this time join text along the columns instead of
rows (in practice just take the transpose of both pred and actual
tables)

e.g., 

This table:

![table-multi-row-column-cells](https://github.com/Unstructured-IO/unstructured-inference/assets/647930/4e5bc0c9-9cc9-4cc3-a64b-6a229dd33931)

Would be represented as table

```python

Disability Category  Participants  Ballots Completed  Ballots Incomplete/Terminated                 Results                           None
               None          None               None                           None                Accuracy               Time to complete
              Blind             5                  1                              4              34.5%, n=1                  1199 sec, n=1
         Low Vision             5                  2                              3  98.3% n=2 (97.7%, n=3)  1716 sec, n=3 (1934 sec, n=2)
          Dexterity             5                  4                              1              98.3%, n=4                1672.1 sec, n=4
           Mobility             3                  3                              0              95.4%, n=3                  1416 sec, n=3
```

then as string by row

```python
('Disability Category\tParticipants\tBallots Completed\tBallots '
 'Incomplete/Terminated\tResults\t\n'
 '\t\t\t\tAccuracy\tTime to complete\n'
 'Blind\t5\t1\t4\t34.5%, n=1\t1199 sec, n=1\n'
 'Low Vision\t5\t2\t3\t98.3% n=2 (97.7%, n=3)\t1716 sec, n=3 (1934 sec, n=2)\n'
 'Dexterity\t5\t4\t1\t98.3%, n=4\t1672.1 sec, n=4\n'
 'Mobility\t3\t3\t0\t95.4%, n=3\t1416 sec, n=3')
```

string by column

```python
('Disability Category\t\tBlind\tLow Vision\tDexterity\tMobility\n'
 'Participants\t\t5\t5\t5\t3\n'
 'Ballots Completed\t\t1\t2\t4\t3\n'
 'Ballots Incomplete/Terminated\t\t4\t3\t1\t0\n'
 'Results\tAccuracy\t34.5%, n=1\t98.3% n=2 (97.7%, n=3)\t98.3%, n=4\t95.4%, '
 'n=3\n'
 '\tTime to complete\t1199 sec, n=1\t1716 sec, n=3 (1934 sec, n=2)\t1672.1 '
 'sec, n=4\t1416 sec, n=3')
```

## rational

1. This metric choose to represent tables as long strings. This is
intentional because we intent to run follow up tasks that treats tables
as text (RAG with LLM)
2. we use token ration instead of ratio so that numbers are compared as
a whole instead of comparing digits. This also ensures that if a word is
split by either OCR or table structure detection it is considered in the
metric as negative

Use the example above, we examine a few predictions and their scores

- predict 1 [pre surge]: lots of structural errors and mistakes in OCR:
**row: 58, column: 61**

![Screenshot 2023-09-19 at 8 46 28
AM](https://github.com/Unstructured-IO/unstructured-inference/assets/647930/381cf7a7-625d-40ea-9e25-ec6911b633a5)

- predict 2 [post surge]: perfect structural extraction but mistakes in
OCR: **row: 92, column: 85**

![Screenshot 2023-09-20 at 9 01 56
AM](https://github.com/Unstructured-IO/unstructured-inference/assets/647930/20498ca8-c3b6-457e-bdea-f131efc867fc)

- predict 3 [chipper]: incorrect structure on multi-row cells and
multi-line cells, but perfect OCR: **row: 84, column: 86**

![Untitled](https://github.com/Unstructured-IO/unstructured-inference/assets/647930/78042969-01a9-4e9a-840c-1741f7ff71a2)



[CORE-1930]:
https://unstructured-ai.atlassian.net/browse/CORE-1930?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Author: badGarnet

CHANGELOG.md

@@ -1,3 +1,7 @@
+## 0.5.30
+
+* add an evaluation metric for table comparison based on token similarity
+
 ## 0.5.29-dev0
 
 * fix paddle unit tests where `make test` fails since paddle doesn't work on M1/M2 chip locally

requirements/base.in

@@ -6,3 +6,4 @@ onnx==1.14.1
 onnxruntime
 # NOTE(alan): Pinned because this is when the most recent module we import appeared
 transformers>=4.25.1
+rapidfuzz

requirements/base.txt

@@ -16,15 +16,15 @@ charset-normalizer==3.2.0
     #   requests
 coloredlogs==15.0.1
     # via onnxruntime
-contourpy==1.1.0
+contourpy==1.1.1
     # via matplotlib
-cryptography==41.0.3
+cryptography==41.0.4
     # via pdfminer-six
 cycler==0.11.0
     # via matplotlib
 effdet==0.4.1
     # via layoutparser
-filelock==3.12.3
+filelock==3.12.4
     # via
     #   huggingface-hub
     #   torch
@@ -33,9 +33,9 @@ flatbuffers==23.5.26
     # via onnxruntime
 fonttools==4.42.1
     # via matplotlib
-fsspec==2023.6.0
+fsspec==2023.9.1
     # via huggingface-hub
-huggingface-hub==0.16.4
+huggingface-hub==0.17.2
     # via
     #   -r requirements/base.in
     #   timm
@@ -56,7 +56,7 @@ layoutparser[layoutmodels,tesseract]==0.3.4
     # via -r requirements/base.in
 markupsafe==2.1.3
     # via jinja2
-matplotlib==3.7.2
+matplotlib==3.7.3
     # via pycocotools
 mpmath==1.3.0
     # via sympy
@@ -79,7 +79,7 @@ omegaconf==2.3.0
     # via effdet
 onnx==1.14.1
     # via -r requirements/base.in
-onnxruntime==1.15.1
+onnxruntime==1.16.0
     # via -r requirements/base.in
 opencv-python==4.8.0.76
     # via
@@ -100,27 +100,27 @@ pdfminer-six==20221105
     # via pdfplumber
 pdfplumber==0.10.2
     # via layoutparser
-pillow==10.0.0
+pillow==10.0.1
     # via
     #   layoutparser
     #   matplotlib
     #   pdf2image
     #   pdfplumber
     #   pytesseract
     #   torchvision
-portalocker==2.7.0
+portalocker==2.8.2
     # via iopath
-protobuf==4.24.2
+protobuf==4.24.3
     # via
     #   onnx
     #   onnxruntime
 pycocotools==2.0.7
     # via effdet
 pycparser==2.21
     # via cffi
-pyparsing==3.0.9
+pyparsing==3.1.1
     # via matplotlib
-pypdfium2==4.19.0
+pypdfium2==4.20.0
     # via pdfplumber
 pytesseract==0.3.10
     # via layoutparser
@@ -130,7 +130,7 @@ python-dateutil==2.8.2
     #   pandas
 python-multipart==0.0.6
     # via -r requirements/base.in
-pytz==2023.3
+pytz==2023.3.post1
     # via pandas
 pyyaml==6.0.1
     # via
@@ -139,6 +139,8 @@ pyyaml==6.0.1
     #   omegaconf
     #   timm
     #   transformers
+rapidfuzz==3.3.0
+    # via -r requirements/base.in
 regex==2023.8.8
     # via transformers
 requests==2.31.0
@@ -158,7 +160,7 @@ sympy==1.12
     # via
     #   onnxruntime
     #   torch
-timm==0.9.6
+timm==0.9.7
     # via effdet
 tokenizers==0.13.3
     # via transformers
@@ -178,18 +180,17 @@ tqdm==4.66.1
     #   huggingface-hub
     #   iopath
     #   transformers
-transformers==4.32.1
+transformers==4.33.2
     # via -r requirements/base.in
-typing-extensions==4.7.1
+typing-extensions==4.8.0
     # via
-    #   filelock
     #   huggingface-hub
     #   iopath
     #   onnx
     #   torch
 tzdata==2023.3
     # via pandas
-urllib3==2.0.4
+urllib3==2.0.5
     # via requests
-zipp==3.16.2
+zipp==3.17.0
     # via importlib-resources

requirements/dev.txt

@@ -18,7 +18,7 @@ argon2-cffi-bindings==21.2.0
     # via argon2-cffi
 arrow==1.2.3
     # via isoduration
-asttokens==2.2.1
+asttokens==2.4.0
     # via stack-data
 async-lru==2.0.4
     # via jupyterlab
@@ -34,7 +34,7 @@ beautifulsoup4==4.12.2
     # via nbconvert
 bleach==6.0.0
     # via nbconvert
-build==0.10.0
+build==1.0.3
     # via pip-tools
 certifi==2023.7.22
     # via
@@ -58,15 +58,15 @@ comm==0.1.4
     # via
     #   ipykernel
     #   ipywidgets
-contourpy==1.1.0
+contourpy==1.1.1
     # via
     #   -c requirements/base.txt
     #   matplotlib
 cycler==0.11.0
     # via
     #   -c requirements/base.txt
     #   matplotlib
-debugpy==1.6.7.post1
+debugpy==1.8.0
     # via ipykernel
 decorator==5.1.1
     # via ipython
@@ -95,6 +95,7 @@ idna==3.4
     #   requests
 importlib-metadata==6.8.0
     # via
+    #   build
     #   jupyter-client
     #   jupyter-lsp
     #   jupyterlab
@@ -107,7 +108,7 @@ importlib-resources==6.0.1
     #   jsonschema-specifications
     #   jupyterlab
     #   matplotlib
-ipykernel==6.25.1
+ipykernel==6.25.2
     # via
     #   jupyter
     #   jupyter-console
@@ -121,7 +122,7 @@ ipython==8.12.2
     #   jupyter-console
 ipython-genutils==0.2.0
     # via qtconsole
-ipywidgets==8.1.0
+ipywidgets==8.1.1
     # via jupyter
 isoduration==20.11.0
     # via jsonschema
@@ -180,15 +181,15 @@ jupyter-server==2.7.3
     #   notebook-shim
 jupyter-server-terminals==0.4.4
     # via jupyter-server
-jupyterlab==4.0.5
+jupyterlab==4.0.6
     # via notebook
 jupyterlab-pygments==0.2.2
     # via nbconvert
-jupyterlab-server==2.24.0
+jupyterlab-server==2.25.0
     # via
     #   jupyterlab
     #   notebook
-jupyterlab-widgets==3.0.8
+jupyterlab-widgets==3.0.9
     # via ipywidgets
 kiwisolver==1.4.5
     # via
@@ -199,7 +200,7 @@ markupsafe==2.1.3
     #   -c requirements/base.txt
     #   jinja2
     #   nbconvert
-matplotlib==3.7.2
+matplotlib==3.7.3
     # via
     #   -c requirements/base.txt
     #   -r requirements/dev.in
@@ -220,9 +221,9 @@ nbformat==5.9.2
     #   jupyter-server
     #   nbclient
     #   nbconvert
-nest-asyncio==1.5.7
+nest-asyncio==1.5.8
     # via ipykernel
-notebook==7.0.3
+notebook==7.0.4
     # via jupyter
 notebook-shim==0.2.3
     # via
@@ -256,7 +257,7 @@ pexpect==4.8.0
     # via ipython
 pickleshare==0.7.5
     # via ipython
-pillow==10.0.0
+pillow==10.0.1
     # via
     #   -c requirements/base.txt
     #   -c requirements/test.txt
@@ -293,7 +294,7 @@ pygments==2.16.1
     #   jupyter-console
     #   nbconvert
     #   qtconsole
-pyparsing==3.0.9
+pyparsing==3.1.1
     # via
     #   -c requirements/base.txt
     #   matplotlib
@@ -307,7 +308,7 @@ python-dateutil==2.8.2
     #   matplotlib
 python-json-logger==2.0.7
     # via jupyter-events
-pytz==2023.3
+pytz==2023.3.post1
     # via
     #   -c requirements/base.txt
     #   babel
@@ -323,7 +324,7 @@ pyzmq==25.1.1
     #   jupyter-console
     #   jupyter-server
     #   qtconsole
-qtconsole==5.4.3
+qtconsole==5.4.4
     # via jupyter
 qtpy==2.4.0
     # via qtconsole
@@ -345,7 +346,7 @@ rfc3986-validator==0.1.1
     # via
     #   jsonschema
     #   jupyter-events
-rpds-py==0.10.0
+rpds-py==0.10.3
     # via
     #   jsonschema
     #   referencing
@@ -362,7 +363,7 @@ sniffio==1.3.0
     # via
     #   -c requirements/test.txt
     #   anyio
-soupsieve==2.4.1
+soupsieve==2.5
     # via beautifulsoup4
 stack-data==0.6.2
     # via ipython
@@ -387,7 +388,7 @@ tornado==6.3.3
     #   jupyterlab
     #   notebook
     #   terminado
-traitlets==5.9.0
+traitlets==5.10.0
     # via
     #   comm
     #   ipykernel
@@ -404,15 +405,15 @@ traitlets==5.9.0
     #   nbconvert
     #   nbformat
     #   qtconsole
-typing-extensions==4.7.1
+typing-extensions==4.8.0
     # via
     #   -c requirements/base.txt
     #   -c requirements/test.txt
     #   async-lru
     #   ipython
 uri-template==1.3.0
     # via jsonschema
-urllib3==2.0.4
+urllib3==2.0.5
     # via
     #   -c requirements/base.txt
     #   -c requirements/test.txt
@@ -425,13 +426,13 @@ webencodings==0.5.1
     # via
     #   bleach
     #   tinycss2
-websocket-client==1.6.2
+websocket-client==1.6.3
     # via jupyter-server
 wheel==0.41.2
     # via pip-tools
-widgetsnbextension==4.0.8
+widgetsnbextension==4.0.9
     # via ipywidgets
-zipp==3.16.2
+zipp==3.17.0
     # via
     #   -c requirements/base.txt
     #   importlib-metadata