feat/support uploading data rather than file (#301)

* add new methods to all uploaders

* bump changelog

* move chroma e2e test to int test

* fix stage pipeline step to maintain extension

* fix df name in duckdb

* fix df name in motherduckdb

Author: rbiseck3

.github/workflows/e2e.yml

@@ -65,6 +65,8 @@ jobs:
           TOGETHERAI_API_KEY: ${{ secrets.TOGETHERAI_API_KEY }}
           VOYAGEAI_API_KEY: ${{ secrets.VOYAGEAI_API_KEY }}
           VERTEXAI_API_KEY: ${{ secrets.VERTEXAI_API_KEY }}
+          AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
+          AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
         run: |
           make install-base
           make install-all-embedders

CHANGELOG.md

@@ -1,9 +1,10 @@
-## 0.3.9-dev3
+## 0.3.9-dev4
 
 ### Enhancements
 
 * **Support ndjson files in stagers**
 * **Add Neo4j destination connector**
+* **Support passing data in for uploaders**
 
 ### Fixes
 

requirements/test.in

@@ -11,6 +11,7 @@ pytest-json-report
 faker
 docker
 universal_pathlib
+deepdiff
 
 # Connector specific deps
 cryptography

requirements/test.txt

@@ -14,6 +14,8 @@ coverage[toml]==7.6.9
     # via pytest-cov
 cryptography==44.0.0
     # via -r test.in
+deepdiff==8.0.1
+    # via -r test.in
 docker==7.1.0
     # via -r test.in
 docstring-parser==0.16
@@ -86,6 +88,8 @@ numpy==1.26.4
     # via
     #   -c ./common/constraints.txt
     #   shapely
+orderly-set==5.2.2
+    # via deepdiff
 packaging==24.2
     # via
     #   google-cloud-aiplatform