feat: Enable remote chunking via unstructured-ingest (#2905)

Update: The cli shell script works when sending documents to the free
api, but the paid api is down, so waiting to test against it.

- The first commit adds docstrings and fixes type hints.
- The second commit reorganizes `test_unstructured_ingest` so it matches
the structure of `unstructured/ingest`.
- The third commit contains the primary changes for this PR.
- The `.chunk()` method responsible for sending elements to the correct
method is moved from `ChunkingConfig` to `Chunker` so that
`ChunkingConfig` acts as a config object instead of containing
implementation logic. `Chunker.chunk()` also now takes a json file
instead of a list of elements. This is done to avoid redundant
serialization if the file is to be sent to the api for chunking.

---------

Co-authored-by: Ahmet Melek <[email protected]>

Author: Coniferish

CHANGELOG.md

@@ -1,7 +1,13 @@
-## 0.13.4-dev1
+## 0.13.4-dev2
 
 ### Enhancements
-* **Unique and deterministic hash IDs for elements** Element IDs produced by any partitioning function are now deterministic and unique at the document level by default. Before, hashes were based only on text; however, they now also take into account the element's sequence number on a page, the page's number in the document, and the document's file name.
+* **Unique and deterministic hash IDs for elements** Element IDs produced by any partitioning
+  function are now deterministic and unique at the document level by default. Before, hashes were
+  based only on text; however, they now also take into account the element's sequence number on a
+  page, the page's number in the document, and the document's file name.
+* **Enable remote chunking via unstructured-ingest** Chunking using unstructured-ingest was
+  previously limited to local chunking using the strategies `basic` and `by_title`. Remote chunking
+  options via the API are now accessible.
 
 ### Features
 

docs/source/ingest/source_connectors.rst

@@ -2,7 +2,7 @@ Source Connectors
 =================
 
 Connect to your favorite data storage platforms for effortless batch processing of your files.
-We are constantly adding new data connectors and if you don'table see your favorite platform let us know
+We are constantly adding new data connectors and if you don't see your favorite platform let us know
 in our community `Slack. <https://short.unstructured.io/pzw05l7>`_
 
 .. toctree::

test_unstructured_ingest/src/against-api.sh

@@ -34,6 +34,8 @@ PYTHONPATH=${PYTHONPATH:-.} "$RUN_SCRIPT" \
   --metadata-exclude coordinates,metadata.last_modified,metadata.detection_class_prob,metadata.parent_id,metadata.category_depth \
   --partition-by-api \
   --strategy hi_res \
+  --chunking-strategy by_page \
+  --chunk-max-characters 10000 \
   --pdf-infer-table-structure \
   --reprocess \
   --output-dir "$OUTPUT_DIR" \