Skip to content

Commit 553e7a9

Browse files
Add support to execute ObjectApi ingestion entirely using TaskGraphs (#257)
This allows the cloud execution of the ingestion driver in a separate TaskGraph enabling: - Triggering jobs without having access permissions on the client (setting `access_credential_name`) - Triggering jobs without having all the required Python dependencies for readers and embedding functions. Tested in Cloud - [Driver TaskGraph](https://cloud.tiledb.com/compute/logs/taskgraphs/TileDB-Inc/a3f03968-d0fe-491b-afff-80dcb336a769) - [Embedding generation TaskGraph](https://cloud.tiledb.com/compute/logs/taskgraphs/TileDB-Inc/6ae250e9-0353-442a-9ec2-b93278cb160e)
1 parent ed9bf68 commit 553e7a9

File tree

9 files changed

+448
-305
lines changed

9 files changed

+448
-305
lines changed

apis/python/examples/object_api/image_search_from_directory.ipynb

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,8 @@
9696
" files_per_partition=800,\n",
9797
" max_tasks_per_stage=5,\n",
9898
" workers=5,\n",
99-
")"
99+
")\n",
100+
"index = object_index.ObjectIndex(uri=index_uri)"
100101
]
101102
},
102103
{

apis/python/examples/object_api/image_search_from_tiledb.ipynb

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,8 @@
144144
"index.update_index(\n",
145145
" max_tasks_per_stage=5,\n",
146146
" workers=5,\n",
147-
")"
147+
")\n",
148+
"index = object_index.ObjectIndex(uri=index_uri)"
148149
]
149150
},
150151
{

apis/python/examples/object_api/soma_cell_similarity_search.ipynb

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,8 @@
6464
"index.update_index(\n",
6565
" max_tasks_per_stage=5,\n",
6666
" workers=5,\n",
67-
")"
67+
")\n",
68+
"index = object_index.ObjectIndex(uri=index_uri)"
6869
]
6970
},
7071
{

apis/python/examples/object_api/text_search_documents.ipynb

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,8 @@
9797
"index.update_index(\n",
9898
" files_per_partition=100,\n",
9999
" config=config,\n",
100-
")"
100+
")\n",
101+
"index = object_index.ObjectIndex(uri=index_uri)"
101102
]
102103
},
103104
{

apis/python/examples/object_api/text_search_wikipedia.ipynb

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,8 @@
114114
" workers=2,\n",
115115
" max_tasks_per_stage=2,\n",
116116
" partition_tile_size=5000,\n",
117-
")"
117+
")\n",
118+
"index = object_index.ObjectIndex(uri=index_uri)"
118119
]
119120
},
120121
{
Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
1+
from .embeddings_ingestion import ingest_embeddings_with_driver
12
from .object_index import ObjectIndex
2-
from .embeddings_ingestion import ingest_embeddings
33

4-
__all__ = [
5-
"ObjectIndex",
6-
"ingest_embeddings",
7-
]
4+
__all__ = ["ObjectIndex", "ingest_embeddings_with_driver"]

0 commit comments

Comments
 (0)