improve orig_elements handling in astra and neo4j (#389)

* improve orig_elements handling in astra and neo4j

* .

* fix json

* fix

* more fixes

* add test

Author: vangheem

CHANGELOG.md

@@ -1,3 +1,9 @@
+## 0.5.5
+
+### Enhancements
+
+* **Improve orig_elements handling in astra and neo4j connectors**
+
 ## 0.5.4
 
 ### Enhancements

test/integration/connectors/expected_results/astradb/stager/DA-1p-with-duplicate-pages.pdf.json

@@ -0,0 +1,26 @@
+import base64
+import json
+import zlib
+
+from unstructured_ingest.v2.processes.connectors.utils import format_and_truncate_orig_elements
+
+
+def test_format_and_truncate_orig_elements():
+    original_elements = [
+        {
+            "text": "Hello, world!",
+            "metadata": {
+                "image_base64": "iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAABwUlEQVR42mNk",
+                "text_as_html": "<p>Hello, world!</p>",
+                "page": 1,
+            },
+        }
+    ]
+    json_bytes = json.dumps(original_elements, sort_keys=True).encode("utf-8")
+    deflated_bytes = zlib.compress(json_bytes)
+    b64_deflated_bytes = base64.b64encode(deflated_bytes)
+    b64_deflated_bytes.decode("utf-8")
+
+    assert format_and_truncate_orig_elements(
+        {"text": "Hello, world!", "metadata": {"orig_elements": b64_deflated_bytes.decode("utf-8")}}
+    ) == [{"metadata": {"page": 1}}]

test/integration/connectors/expected_results/astradb/stager/DA-1p-with-duplicate-pages.pdf.ndjson

@@ -359,33 +359,9 @@
       }
     }
   },
-  {
-    "type": "UncategorizedText",
-    "element_id": "4f2dbe3656a9ebc60c7e3426ad3cb3e3",
-    "text": "_____________________________________________________________________________________________",
-    "metadata": {
-      "filetype": "application/pdf",
-      "languages": [
-        "eng"
-      ],
-      "page_number": 2,
-      "data_source": {
-        "url": "abfs://container1/Core-Skills-for-Biomedical-Data-Scientists-2-pages.pdf",
-        "version": "0x8DB214A673DD8D8",
-        "record_locator": {
-          "protocol": "abfs",
-          "remote_file_path": "abfs://container1/"
-        },
-        "date_created": "1678440764.0",
-        "date_modified": "1678440764.0",
-        "permissions_data": null,
-        "filesize_bytes": 41164
-      }
-    }
-  },
   {
     "type": "NarrativeText",
-    "element_id": "cd359ae8c49885ead47318021438eead",
+    "element_id": "c8fdefac1ae82fa42caeceff04853415",
     "text": "this commitment, a recent report to the NLM Director recommended working across NIH to identify and develop core skills required of a biomedical data scientist to consistency across the cohort of NIH-trained data scientists. This report provides a set of recommended core skills based on analysis of current BD2K-funded training programs, biomedical data science job ads, and practicing members of the current data science workforce.",
     "metadata": {
       "filetype": "application/pdf",
@@ -409,7 +385,7 @@
   },
   {
     "type": "Title",
-    "element_id": "bf8321a34edb7103ec4209f3e4a8a8da",
+    "element_id": "b5b7392d0a946f5016bfa8ad0c248a9b",
     "text": "Methodology",
     "metadata": {
       "filetype": "application/pdf",
@@ -433,7 +409,7 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "1e1d3d1a5c1397fc588393568d829bc8",
+    "element_id": "d9d8e38d221ae621c0ddbcabaa4a28b4",
     "text": "The Workforce Excellence team took a three-pronged approach to identifying core skills required of a biomedical data scientist (BDS), drawing from:",
     "metadata": {
       "filetype": "application/pdf",
@@ -457,7 +433,7 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "45d7ff56632d66a2ab2d4dd2716d4d2e",
+    "element_id": "ba70aa3bc3ad0dec6a62939c94c5a20c",
     "text": "a) Responses to a 2017 Kaggle1 survey2 of over 16,000 self-identified data scientists working across many industries. Analysis of the Kaggle survey responses from the current data science workforce provided insights into the current generation of data scientists, including how they were trained and what programming and analysis skills they use.",
     "metadata": {
       "filetype": "application/pdf",
@@ -481,7 +457,7 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "bf452aac5123fcedda30dd6ed179f41c",
+    "element_id": "24724b1f0d20a6575f2782fd525c562f",
     "text": "b) Data science skills taught in BD2K-funded training programs. A qualitative content analysis was applied to the descriptions of required courses offered under the 12 BD2K-funded training programs. Each course was coded using qualitative data analysis software, with each skill that was present in the description counted once. The coding schema of data science-related skills was inductively developed and was organized into four major categories: (1) statistics and math skills; (2) computer science; (3) subject knowledge; (4) general skills, like communication and teamwork. The coding schema is detailed in Appendix A.",
     "metadata": {
       "filetype": "application/pdf",
@@ -505,7 +481,7 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "ca176cbef532792b1f11830ff7520587",
+    "element_id": "5e6c73154a1e5f74780c69afbc9bc084",
     "text": "c) Desired skills identified from data science-related job ads. 59 job ads from government (8.5%), academia (42.4%), industry (33.9%), and the nonprofit sector (15.3%) were sampled from websites like Glassdoor, Linkedin, and Ziprecruiter. The content analysis methodology and coding schema utilized in analyzing the training programs were applied to the job descriptions. Because many job ads mentioned the same skill more than once, each occurrence of the skill was coded, therefore weighting important skills that were mentioned multiple times in a single ad.",
     "metadata": {
       "filetype": "application/pdf",
@@ -529,7 +505,7 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "11b170fedd889c3b895bbd28acd811ca",
+    "element_id": "249f6c76b2c99dadbefb8b8811b0d4cd",
     "text": "Analysis of the above data provided insights into the current state of biomedical data science training, as well as a view into data science-related skills likely to be needed to prepare the BDS workforce to succeed in the future. Together, these analyses informed recommendations for core skills necessary for a competitive biomedical data scientist.",
     "metadata": {
       "filetype": "application/pdf",
@@ -553,7 +529,7 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "2665aadf75bca259f1f5b4c91a53a301",
+    "element_id": "f4b34fe2b03c12e48a89276dca673bfb",
     "text": "1 Kaggle is an online community for data scientists, serving as a platform for collaboration, competition, and learning: http://kaggle.com",
     "metadata": {
       "filetype": "application/pdf",
@@ -577,7 +553,7 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "8bbfe1c3e6bca9a33226d20d69b2297a",
+    "element_id": "75e0008cfdfecc18fb8c43490c53d6d4",
     "text": "2 In August 2017, Kaggle conducted an industry-wide survey to gain a clearer picture of the state of data science and machine learning. A standard set of questions were asked of all respondents, with more specific questions related to work for employed data scientists and questions related to learning for data scientists in training. Methodology and results: https://www.kaggle.com/kaggle/kaggle-survey-2017",
     "metadata": {
       "filetype": "application/pdf",
@@ -600,8 +576,8 @@
     }
   },
   {
-    "type": "UncategorizedText",
-    "element_id": "dd4a661e1a3c898a5cf6328ba56b924d",
+    "type": "PageNumber",
+    "element_id": "e5d48e29d989341ba281611d4eb9311a",
     "text": "2",
     "metadata": {
       "filetype": "application/pdf",

test/unit/processes/connectors/test_utils.py

@@ -1033,8 +1033,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "0753dc21ca9fa94cb607cdb49bef3eed",
-    "text": "Item 6, page 2.—The term “gross receipts” includes total sales (net of returns and allowances) and all amounts received for services. In addition, gross receipts include any income from investments and from incidental or outside sources (e.g., interest, dividends, rents, royalties, and annuities). However, if you are a resaler of personal property, exclude from gross receipts any amounts not derived in the ordinary course of a trade or business. Gross receipts do not include amounts received for sales taxes if, under the applicable state or local law, the tax is legally imposed on the purchaser of the good or service, and the taxpayer merely collects and remits the tax to the taxing authority.",
+    "element_id": "5dbefdf9c5729f9d9a74a4b9c40bfb03",
+    "text": "Item 6, page 2.—The term “gross receipts” includes total sales (net of returns and allowances) and all amounts received for services. In addition, gross receipts include any income from investments and from incidental or outside sources (e.g., interest, dividends, rents, royalties, and annuities). However, if you are a resaler of personal property, exclude from gross recepts any amounts not derived in the ordinary course of a trae or business. Gross receipts do not include amounts received for sales taxes if, under the applicable state or local law, the tax is legally imposed on the purchaser of the good or service, and the taxpayer merely collects and remits the tax to the taxing authority.",
     "metadata": {
       "filetype": "application/pdf",
       "languages": [
@@ -1081,8 +1081,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "a34b5c633b40ae532a293aa5ece41ff6",
-    "text": "(manufacturing, retailer, wholesaler, etc.), employer identification number, overall method of accounting, and whether, in the last 6 years, that business has changed its accounting method, or is also changing its accounting method as part of this request or as a separate request.",
+    "element_id": "1fe93da70cb3544175c812a8fb231a93",
+    "text": "(manufacturing, retailer, wholesaler, etc.), employer identification number, overall method of accounting, and whether, in the last 6 years, that business has changed its accounting method, or s also changing its accounting method as part of this request or as a separate request.",
     "metadata": {
       "filetype": "application/pdf",
       "languages": [
@@ -1273,8 +1273,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "375f471287a32d216212e71c83efac13",
-    "text": "Item 1b, page 2.—Include any amounts reported as income In a prior year although the income had not been accrued (earned) or received In the prior year; for example, discount on instaliment loans reported as income for the year in which the loans were made instead of for the year or years in which the income was received or earned. Advance payments under Rev. Proc. 71-21 or Regulations section 1.451-5 must be fully explained and all pertinent information must be submitted with this application.",
+    "element_id": "4402466124ef06237b1c582818097ef5",
+    "text": "Item 1b, page 2.—Include any amounts reported as income In a prior year although the income had not been accrued (earned) or received In the prior year; for example, discount on instalment loans reported as income for the year in which the loans were made instead of for the year or years in which the income was received or earned. Advance payments under Rev. Proc. 71-21 or Regulations section 1.451-5 must be fully explained and all pertinent information must be submitted with this application.",
     "metadata": {
       "filetype": "application/pdf",
       "languages": [
@@ -1753,8 +1753,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "3c85924610889b0aea2755f36f32d151",
-    "text": "Section 460(f) provides that the term “long-term contract” means any contract for the manufacturing, building, installation, or construction of property that is not completed within the tax year in which it is entered into. However, a manufacturing contract will not qualify as a long-term contract unless the contract involves the manufacture of: (1) a unique item not normally included in your finished goods inventory, or (2) any item that normally requires more than 12 calendar months to complete.",
+    "element_id": "6641f860f560189f2a4f70a11bcb18a0",
+    "text": "Section 460(f) provides that the term “long-term contract” means any contract for the manufacturing, building, installation, or construction of property that is not completed within the tax year in which it is entered into. However, a manufacturing contract will not qualify as a long-term contract unless the contract involves the manufacture of: (1) a unique item not normally included n your finished goods inventory, or (2) any item that normally requires more than 12 calendar months to complete.",
     "metadata": {
       "filetype": "application/pdf",
       "languages": [
@@ -1849,8 +1849,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "211a99a9043cb86883fccf0990dca05a",
-    "text": "This section is to be used only to request a change in a method of accounting for depreciation under section 167.",
+    "element_id": "7a531ff803f3d44ea7844acad139239f",
+    "text": "This section s to be used only to request a change n a method of accounting for depreciation under section 167.",
     "metadata": {
       "filetype": "application/pdf",
       "languages": [
@@ -1945,8 +1945,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "7fa50d465048201b8120b4c992461622",
-    "text": "Generally, this section should be used for requesting changes in a method of accounting for which provision has not been made elsewhere on this form. Attach additional pages if more space 1s needed for a full explanation of the present method used and the proposed change requested.",
+    "element_id": "f6c9da2dd8c289bbfd9803b594cd42d4",
+    "text": "Generally, this section should be used for requesting changes n a method of accounting for which provision has not been made elsewhere on this form. Attach additional pages if more space 1s needed for a full explanation of the present method used and the proposed change requested.",
     "metadata": {
       "filetype": "application/pdf",
       "languages": [

test_e2e/expected-structured-output/azure/Core-Skills-for-Biomedical-Data-Scientists-2-pages.pdf.json

@@ -337,8 +337,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "7906b3444dddda0127174fa5c39eeaaf",
-    "text": "Generally, applicants must complete Section A. In addition, complete the appropriate sections (B-1 through H) for which a change 1s desired.",
+    "element_id": "fddbf541191e6288740825d1ba2ad699",
+    "text": "Generally, applicants must complete Section A. In addition, complete the appropriate sections (B-1 through H) for which a change s desired.",
     "metadata": {
       "filetype": "image/png",
       "languages": [
@@ -433,8 +433,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "10e96c0c96cbb0372d9aa0c53e4e22fd",
-    "text": "Uniform capitalization rules and limitation on cash method.—If you are required to change your method of accounting under sectiort 263A (relating to the capitalization and inclusion in inventory costs of certain expenses) or 448 (limiting the use of the cash method of accounting by certain taxpayers) as added by the Tax Reform Act of 1986 (“Act”), the change 1s treated as initiated by the taxpayer, approved by the Commissioner, and the period for taking the adjustments under section 481(a) into account will not exceed 4 years. (Hospitals required to change from the cash method under section 448 have 10 years to take the adjustments into account.) Complete Section A and the appropriate sections (B-1 or C and D) for which the change is required.",
+    "element_id": "db1340e51c6f16b0d09662ef99b6a95c",
+    "text": "Uniform capitalization rules and limitation on cash method.—If you are required to change your method of accounting under sectior 263A (relating to the capitalization and inclusion in inventory costs of certain expenses) or 448 (limiting the use of the cash method of accounting by certain taxpayers) as added by the Tax Reform Act of 1986 (“Act”), the change 1s treated as initiated by the taxpayer, approved by the Commissioner, and the period for taking the adjustments under section 481(a) nto account will not exceed 4 years. (Hospitals required to change from the cash method under section 448 have 10 years to take the adjustments into account.) Complete Section A and the appropriate sections (B-1 or C and D) for which the change is required.",
     "metadata": {
       "filetype": "image/png",
       "languages": [
@@ -457,8 +457,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "b708681dc3b4972238aa4adb68ce9d58",
-    "text": "Disregard the instructions under Time and Place for Filing and Late Applications. Instead, attach Form 3115 to your income tax return for the year of change; do not file it separately. Also include on a separate statement accompanying the Form 3115 the period over which the section 481(a) adjustment will be taken into account and the basis for that conclusion. Identify the automatic change being made at the top of page 1of Form 3115 (e “Automatic Change to Accrual Method— e tion 448\"). See Temporary Regulations sections 1.263A-1T and 1.448-1T for additional information.",
+    "element_id": "9acc565afdfca753f8670453ee0bcbc9",
+    "text": "Disregard the instructions under Time and Place for Filing and Late Applications. Instead, attach Form 3115 to your income tax return for the year of change; do not file it separately. Also include on a separate statement accompanying the Form 3115 the period over which the section 481(a) adjustment will be taken into account and the basis for that conclusion. Identify the automatic change being made at the top of page 1of Form 3115 (e “Automatic Change to Accrual Method— tion 448\"). See Temporary Regulations sections 1.263A-1T and 1.448-1T for additional information.",
     "metadata": {
       "filetype": "image/png",
       "languages": [
@@ -505,8 +505,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "032a665732e8c3d746c4b9c1a008d806",
-    "text": "Generally, applicants must file this form within the first 180 days of the tax year in which it is desired to make the change.",
+    "element_id": "9f6e038f1b56fc0e2b240959e8e26380",
+    "text": "Generally, applicants must file this form within the first 180 days of the tax year in which it is desire to make the change.",
     "metadata": {
       "filetype": "image/png",
       "languages": [
@@ -529,8 +529,8 @@
   },
   {
     "type": "NarrativeText",
-    "element_id": "e801588e099fc459b0de871907003b6b",
-    "text": "Taxpayers, other than exempt organizations, should file Form 3115 with the Commissioner of Internal Revenue, Attention: CC:C:4, 1111 Constitution Avenue, NW, Washington, DC 20224, Exempt organizations should file with the Assistant Commissioner (Employee Plans and Exempt Organizations), 1111 Constitution Avenue, NW, Washington, DC 20224.",
+    "element_id": "52a2bedfff51dcae79b58b1bf03791f3",
+    "text": "Taxpayers, other than exempt organizations, should file Form 3115 with the Commssioner of Internal Revenue, Attention: CC:C:4, 1111 Constitution Avenue, NW, Washington, DC 20224, Exempt organizations should file with the Assistant Commissioner (Employee Plans and Exempt Organizations), 1111 Constitution Avenue, NW, Washington, DC 20224.",
     "metadata": {
       "filetype": "image/png",
       "languages": [