Skip to content

Commit 7a8c8fc

Browse files
rbiseck3rbiseck3
authored
feat/improve file data batch content (#302)
* update file data to support explicit batch, leverage in ES connector * update sqlite * update the other sql connectors * update astra connectors * update couchbase * bump changelog * fix mongodb connector as well * fix partitioner * fix astra * update fixtures * give additional metadata models better names * fix kafka int tests * update confluence fixtures * leverage from file util for all file data * update fixtures for couchbase
1 parent 3c84f96 commit 7a8c8fc

File tree

221 files changed

+1654
-1591
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

221 files changed

+1654
-1591
lines changed

‎CHANGELOG.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
## 0.3.10-dev0
2+
3+
### Enhancements
4+
5+
* **Support more concrete FileData content for batch support**
6+
17
## 0.3.9
28

39
### Enhancements

‎test/integration/connectors/expected_results/astradb/directory_structure.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"directory_structure": [
3+
"25b75f1d-a2ea-4c97-b75f-1da2eadc97f7.csv",
4+
"60297eea-73d7-4fca-a97e-ea73d7cfca62.csv",
5+
"641d99e3-9941-4c18-9d99-e399414c183d.csv",
6+
"762c0093-2277-4f3e-ac00-932277af3e0e.csv",
7+
"ae40df94-0b3a-4f89-80df-940b3a6f8966.csv"
8+
]
9+
}

‎test/integration/connectors/expected_results/astradb/downloads/25b75f1d-a2ea-4c97-b75f-1da2eadc97f7.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
_id,title,reviewid,creationdate,criticname,originalscore,reviewstate,reviewtext
2+
25b75f1d-a2ea-4c97-b75f-1da2eadc97f7,City Hunter: Shinjuku Private Eyes,2558908,2019-02-14,Matt Schley,2.5/5,rotten,The film's out-of-touch attempts at humor may find them hunting for the reason the franchise was so popular in the first place.

‎test/integration/connectors/expected_results/astradb/downloads/60297eea-73d7-4fca-a97e-ea73d7cfca62.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
_id,title,reviewid,creationdate,criticname,originalscore,reviewstate,reviewtext
2+
60297eea-73d7-4fca-a97e-ea73d7cfca62,City Hunter: Shinjuku Private Eyes,2590987,2019-05-28,Reuben Baron,,fresh,"The choreography is so precise and lifelike at points one might wonder whether the movie was rotoscoped, but no live-action reference footage was used. The quality is due to the skill of the animators and Kodama's love for professional wrestling."

‎test/integration/connectors/expected_results/astradb/downloads/641d99e3-9941-4c18-9d99-e399414c183d.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
_id,title,reviewid,creationdate,criticname,originalscore,reviewstate,reviewtext
2+
641d99e3-9941-4c18-9d99-e399414c183d,Beavers,1145982,2003-05-23,Ivan M. Lincoln,3.5/4,fresh,"Timed to be just long enough for most youngsters' brief attention spans -- and it's packed with plenty of interesting activity, both on land and under the water."

‎test/integration/connectors/expected_results/astradb/downloads/762c0093-2277-4f3e-ac00-932277af3e0e.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
_id,title,reviewid,creationdate,criticname,originalscore,reviewstate,reviewtext
2+
762c0093-2277-4f3e-ac00-932277af3e0e,Blood Mask,1636744,2007-06-02,The Foywonder,1/5,rotten,"It doesn't matter if a movie costs 300 million or only 300 dollars; good is good and bad is bad, and Bloodmask: The Possession of Nicole Lameroux is just plain bad."

‎test/integration/connectors/expected_results/astradb/downloads/ae40df94-0b3a-4f89-80df-940b3a6f8966.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
_id,title,reviewid,creationdate,criticname,originalscore,reviewstate,reviewtext
2+
ae40df94-0b3a-4f89-80df-940b3a6f8966,Dangerous Men,2504681,2018-08-29,Pat Padua,,fresh,Its clumsy determination is endearing and sometimes wildly entertaining

‎test/integration/connectors/expected_results/astradb/file_data/25b75f1d-a2ea-4c97-b75f-1da2eadc97f7.csv.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"identifier": "25b75f1d-a2ea-4c97-b75f-1da2eadc97f7.csv",
3+
"connector_type": "astradb",
4+
"source_identifiers": null,
5+
"metadata": {
6+
"url": null,
7+
"version": null,
8+
"record_locator": {
9+
"document_id": "25b75f1d-a2ea-4c97-b75f-1da2eadc97f7"
10+
},
11+
"date_created": null,
12+
"date_modified": null,
13+
"date_processed": "1734108082.652201",
14+
"permissions_data": null,
15+
"filesize_bytes": null
16+
},
17+
"additional_metadata": {
18+
"collection_name": "ingest_test_src",
19+
"keyspace": null
20+
},
21+
"reprocess": false,
22+
"local_download_path": "/private/var/folders/n8/rps3wl195pj4p_0vyxqj5jrw0000gn/T/pytest-of-romanisecke/pytest-75/test_astra_search_source0/25b75f1d-a2ea-4c97-b75f-1da2eadc97f7.csv",
23+
"display_name": null
24+
}

‎test/integration/connectors/expected_results/astradb/file_data/43d02113-723f-5ec1-acbb-c8da8d3650dc.json‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
{
2+
"identifier": "43d02113-723f-5ec1-acbb-c8da8d3650dc",
3+
"connector_type": "astradb",
4+
"source_identifiers": null,
5+
"metadata": {
6+
"url": null,
7+
"version": null,
8+
"record_locator": null,
9+
"date_created": null,
10+
"date_modified": null,
11+
"date_processed": "1734108082.497229",
12+
"permissions_data": null,
13+
"filesize_bytes": null
14+
},
15+
"additional_metadata": {
16+
"collection_name": "ingest_test_src",
17+
"keyspace": null
18+
},
19+
"reprocess": false,
20+
"local_download_path": null,
21+
"display_name": null,
22+
"batch_items": [
23+
{
24+
"identifier": "25b75f1d-a2ea-4c97-b75f-1da2eadc97f7",
25+
"version": null
26+
},
27+
{
28+
"identifier": "60297eea-73d7-4fca-a97e-ea73d7cfca62",
29+
"version": null
30+
},
31+
{
32+
"identifier": "641d99e3-9941-4c18-9d99-e399414c183d",
33+
"version": null
34+
},
35+
{
36+
"identifier": "762c0093-2277-4f3e-ac00-932277af3e0e",
37+
"version": null
38+
},
39+
{
40+
"identifier": "ae40df94-0b3a-4f89-80df-940b3a6f8966",
41+
"version": null
42+
}
43+
]
44+
}

‎test/integration/connectors/expected_results/astradb/file_data/60297eea-73d7-4fca-a97e-ea73d7cfca62.csv.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"identifier": "60297eea-73d7-4fca-a97e-ea73d7cfca62.csv",
3+
"connector_type": "astradb",
4+
"source_identifiers": null,
5+
"metadata": {
6+
"url": null,
7+
"version": null,
8+
"record_locator": {
9+
"document_id": "60297eea-73d7-4fca-a97e-ea73d7cfca62"
10+
},
11+
"date_created": null,
12+
"date_modified": null,
13+
"date_processed": "1734108082.6517608",
14+
"permissions_data": null,
15+
"filesize_bytes": null
16+
},
17+
"additional_metadata": {
18+
"collection_name": "ingest_test_src",
19+
"keyspace": null
20+
},
21+
"reprocess": false,
22+
"local_download_path": "/private/var/folders/n8/rps3wl195pj4p_0vyxqj5jrw0000gn/T/pytest-of-romanisecke/pytest-75/test_astra_search_source0/60297eea-73d7-4fca-a97e-ea73d7cfca62.csv",
23+
"display_name": null
24+
}

0 commit comments

Comments
 (0)