feat: refactor ingest (#3009)

### Description
This refactors the current ingest CLI process to support better
granularity in how the steps are ran
* Both multiprocessing and async now supported. Given that a lot of the
steps are IO-bound, such as downloading and uploading content, we can
achieve better parallelization by using async here
* Destination step broken up into a stager step and an upload step. This
will allow for steps that require manipulation of the data between
formats, such as converting the elements json into a csv format to
upload for tabular destinations, to be pulled out of the step that does
the actual upload.
* The process of writing the content to a local destination was now
pulled out as it's own dedicated destination connector, meaning you no
longer need to persist the content locally once the process is done if
the content was uploaded elsewhere.
* Quick update to the chunker/partition step to use the python client.
* Move the uncompress suppport as a pipeline step since this can
arbitrarily apply to any concrete files that have been downloaded,
regardless of where they came from.
* Leverage last modified date to mark files to be reprocessed, even if
the file already exists locally.

### Callouts
Retry configs haven't been moved over yet. This is an open question
because the intent was for it to wrap potential connection errors but
now any of the other steps that leverage an API might run into network
connection issues. Should those be isolated in each of the steps and
wrapped with the same retry configs? Or do we need to expose a unique
retry config for each step? This would bloat the input params even more.

### Testing
* If you want to run the new code as an SDK, there's an example file
that was added to highlight how to do that:
[example.py](https://github.com/Unstructured-IO/unstructured/blob/roman/refactor-ingest/unstructured/ingest/v2/example.py)
* If you want to run the new code as an isolated CLI:
```shell
PYTHONPATH=. python unstructured/ingest/v2/main.py --help
```
* If you want to see which commands have been migrated to the new
version, there's now a `v2` short help text next to those commands when
running the current cli:
```shell
PYTHONPATH=. python unstructured/ingest/main.py --help
Usage: main.py [OPTIONS] COMMAND [ARGS]...main.py --help   

Options:
  --help  Show this message and exit.

Commands:
  airtable
  azure
  biomed
  box
  confluence
  delta-table
  discord
  dropbox
  elasticsearch
  fsspec
  gcs
  github
  gitlab
  google-drive
  hubspot
  jira
  local          v2
  mongodb
  notion
  onedrive
  opensearch
  outlook
  reddit
  s3             v2
  salesforce
  sftp
  sharepoint
  slack
  wikipedia
```

You can run any of the local or s3 specific ingest tests and these
should now work.

---------

Co-authored-by: ryannikolaidis <[email protected]>
Co-authored-by: rbiseck3 <[email protected]>

Author: rbiseck3

CHANGELOG.md

@@ -1,8 +1,15 @@
-## 0.14.1-dev0
+## 0.14.1-dev1
 
 * **Add support for Python 3.12**. `unstructured` now works with Python 3.12!
 
 ### Features
+* **Large improvements to the ingest process:**
+  * Support for multiprocessing and async, with limits for both.
+  * Streamlined to process when mapping CLI invocations to the underlying code
+  * More granular steps introduced to give better control over process (i.e. dedicated step to uncompress files already in the local filesystem, new optional staging step before upload)
+  * Use the python client when calling the unstructured api for partitioning or chunking
+  * Saving the final content is now a dedicated destination connector (local) set as the default if none are provided. Avoids adding new files locally if uploading elsewhere.
+  * Leverage last modified date when deciding if new files should be downloaded and reprocessed.
 
 ### Fixes
 

examples/ingest/chroma/ingest.sh

@@ -16,7 +16,7 @@ PYTHONPATH=. ./unstructured/ingest/main.py \
   --input-path example-docs/book-war-and-peace-1p.txt \
   --output-dir local-to-chroma \
   --strategy fast \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --embedding-provider "<an unstructured embedding provider, ie. langchain-huggingface>" \
   --num-processes 2 \
   --verbose \

examples/ingest/clarifai/ingest.sh

@@ -10,7 +10,7 @@ PYTHONPATH=. ./unstructured/ingest/main.py \
   --input-path example-docs/book-war-and-peace-1225p.txt \
   --output-dir local-output-to-clarifai \
   --strategy fast \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --num-processes 2 \
   --verbose \
   clarifai \

examples/ingest/elasticsearch/destination.sh

@@ -15,7 +15,7 @@ PYTHONPATH=. ./unstructured/ingest/main.py \
   --input-path example-docs/book-war-and-peace-1225p.txt \
   --output-dir local-to-elasticsearch \
   --strategy fast \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --embedding-provider "<an unstructured embedding provider, ie. langchain-huggingface>" \
   --num-processes 2 \
   --verbose \

examples/ingest/mongodb/destination.sh

@@ -15,7 +15,7 @@ PYTHONPATH=. ./unstructured/ingest/main.py \
   --input-path example-docs/book-war-and-peace-1225p.txt \
   --output-dir local-to-mongodb \
   --strategy fast \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --embedding-provider "<an unstructured embedding provider, ie. langchain-huggingface>" \
   --num-processes 2 \
   --verbose \

examples/ingest/opensearch/destination.sh

@@ -15,7 +15,7 @@ PYTHONPATH=. ./unstructured/ingest/main.py \
   --input-path example-docs/book-war-and-peace-1225p.txt \
   --output-dir local-to-opensearch \
   --strategy fast \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --embedding-provider "<an unstructured embedding provider, ie. langchain-huggingface>" \
   --num-processes 2 \
   --verbose \

examples/ingest/pinecone/ingest.sh

@@ -16,7 +16,7 @@ PYTHONPATH=. ./unstructured/ingest/main.py \
   --input-path example-docs/book-war-and-peace-1225p.txt \
   --output-dir local-to-pinecone \
   --strategy fast \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --embedding-provider "<an unstructured embedding provider, ie. langchain-huggingface>" \
   --num-processes 2 \
   --verbose \

examples/ingest/qdrant/ingest.sh

@@ -12,7 +12,7 @@ unstructured-ingest \
   --input-path example-docs/book-war-and-peace-1225p.txt \
   --output-dir local-output-to-qdrant \
   --strategy fast \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --embedding-provider "$EMBEDDING_PROVIDER" \
   --num-processes 2 \
   --verbose \

examples/ingest/sql/ingest.sh

@@ -12,7 +12,7 @@ PYTHONPATH=. ./unstructured/ingest/main.py \
   --input-path example-docs/book-war-and-peace-1225p.txt \
   --output-dir local-to-pinecone \
   --strategy fast \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --embedding-provider "<an unstructured embedding provider, ie. langchain-huggingface>" \
   --num-processes 2 \
   --verbose \

examples/ingest/weaviate/ingest.sh

@@ -16,7 +16,7 @@ PYTHONPATH=. ./unstructured/ingest/main.py \
   --reprocess \
   --input-path example-docs/book-war-and-peace-1225p.txt \
   --work-dir weaviate-work-dir \
-  --chunk-elements \
+  --chunking-strategy by_title \
   --chunk-new-after-n-chars 2500 --chunk-multipage-sections \
   --embedding-provider "langchain-huggingface" \
   weaviate \