feat/add support for ndjson files (#426)

* add support for ndjson files

* attempt to fix CI

* chunk using api as well in e2e test

* fix couchbase dest test

* fix couchbase dest test

* fix couchbase dest test

* remove redundant line

* update changelog

Author: rbiseck3

.github/workflows/ingest-test-fixtures-update-pr.yml

@@ -85,6 +85,7 @@ jobs:
           CI: "true"
         run: |
           source .venv/bin/activate
+          pip install deepdiff
           sudo make install-docker-compose
           docker compose version
           pip freeze

CHANGELOG.md

@@ -1,9 +1,13 @@
-## 0.5.15-dev0
+## 0.5.15
 
 ### Fixes 
 
 * **Separate password and api_token for Confluence connector**
 
+### Features 
+
+* Support NDJSON for data between pipeline steps for data streaming**
+
 ## 0.5.14
 
 ### Fixes

debugging/chunk/24fc120e9adc.json

@@ -0,0 +1,107 @@
+[
+  {
+    "type": "CompositeElement",
+    "element_id": "c4155c9bd30b69dbc5562b63bb649e70",
+    "text": "CHAPTER I\n\n\"Well, Prince, so Genoa and Lucca are now just family estates of the Buonapartes. But I warn you, if you don't tell me that this means war, if you still try to defend the infamies and horrors perpetrated by that Antichrist--I really believe he is Antichrist--I will have nothing more to do with you and you are no longer my friend, no longer my 'faithful slave,' as you call yourself! But how do you do? I see I have frightened you--sit down and tell me all the news.\"",
+    "metadata": {
+      "filename": "8598c4e156b7.json",
+      "filetype": "application/json",
+      "languages": [
+        "eng"
+      ],
+      "orig_elements": "eJztU8FuGyEQ/ZUpl1y8Xq/X9np7qdKqai1VbRQl6iGJLAyDl4aFFbBxrMj/3mETR2nUHnrroRdg4PHemxm4emBosEUb11qyt8BQiAUKJSWf1NVc1sV8tkCpCiFnkwqXCzYC1mLkkkdO+AeWFuvgei/wGONaeKRpICyq6XQ6r6tiNq5n5bKYFoliQLVOaqV/A5sVVbl8hnXeCQzhiJsVdTUp5+W4WNRVWSdYh77VIWhnw/rJ2NUDI/pkqSyL5fJwQzCPwnm5Nk7w6PzgtuOxSbT5ZUAfcu9abnVAcYv5pQ3R9yL2HuXqW96/CDNttxhijve87Qxm0omQb5y7zXbcZ9zKrEMuMCu6cbyP7HAgcaUNWt4mR+zP0Cdg3HcDMOJ9zDvDtU0nhtttz0k55cfQbtlNYk6gBP7w+fTs4uM5rBL2yHCho0FGsNeNrvlmKhZKlEqphUAplrQSOKVBLKvJ/H+j/6lGp+S5/4t/+vJlXLPvaMwIzry2AkcQHHxC6ziQLfjSC0Erj2DdDn70IYLirTZ7oMypLQGcgtggvO+d5WSCtsYURFgBpWZh7/oRaJVmkM6eRIikBi3SLU5BowMF3IYEf0aGqAkU/R6iA4kKyUpS0Tapk2ry1jjvnQ9Afe8w+vTUYLN/5D21UYvG6xCzbAX0Dg1Z3qDReIeQiMIryC4JNvwuJUqm7BZaR1kneUeHsRl8JdlhHgoCxtEL8NDuQXlNHke/bp4oThdVbyAYYh6dAA/DdUF20sIHNOrNUK6GyktKj2V6R9ULiDQOjoh820S0OIhnWdCRQDs72DmWM1GmElnchfE1e/nNv3JP1dF3eJFafrj5CZex6oE="
+    }
+  },
+  {
+    "type": "CompositeElement",
+    "element_id": "a01983a67e9b804bee051afc4e618321",
+    "text": "It was in July, 1805, and the speaker was the well-known Anna Pavlovna Scherer, maid of honor and favorite of the Empress Marya Fedorovna. With these words she greeted Prince Vasili Kuragin, a man of high rank and importance, who was the first to arrive at her reception. Anna Pavlovna had had a cough for some days. She was, as she said, suffering from la grippe; grippe being then a new word in St. Petersburg, used only by the elite.",
+    "metadata": {
+      "filename": "8598c4e156b7.json",
+      "filetype": "application/json",
+      "languages": [
+        "eng"
+      ],
+      "orig_elements": "eJx1kk+PmzAQxb/KiHNC4kACtKceWmlbtV1pu9vDahUN9gBWwEa2STZa5bt3zP5pVbUHZIwfb37PM/dPCfU0kAl7rZJ3kIiqUTuZF0KoWuyyfLctJYp6W27qut6Um2QByUABFQZk/VMSX/beTk7S65720hEvz4bFZrPZVoXI0yrPSrER0WJWDVbpRv9DlosiK99ko7OSvH/V5aIq1tk2S8WuKrIqykZyg/ZeW+P3L2D3TwnbR6QsE2V5eWCZI2md2vdWYrBuph0xdNF2devJ+ZWzAxrtSR5odWt8cJMMkyN19X01/bFdatOSDyt6xGHsaams9Kva2sPyhG6JRi1HQklLMabhMSSXCxdvdE8Gh0iU/F/6IgzncRYGegyrsUdt4kmPpp2QK8d8CZk2ialGdL/bR1LuSDZK4boqtqoS23xHqhFS5euCyl0SUaJrFF8FOKEHbeDz1J8XIMr1dgHMBKEj8Ax2IDdL4v5Efb88GHsy8MEYhGs89vbILzeyI0duAQNqBbaBzhrrZp8Gj9bpQPFr9Pg4jI5bCV/RnRE+kbIuWqTwU4cuKjzX4R558KxuHRFPEVw7bSTBHXrda/gyOWy1YVAuaOZ6uu3AoTnMNfUwWheQ/1jAqbNv/I12PkCwgM7pIwEGYG7goaAx8Oikf8XqUM0PgrQTF2g4k7cDgcKzT+EmXgl6xniG9Rx+AX5qGmLcFhqeJeiRQ+hxpPcvK9QUD5nHsLGh0xw3tuAmpHDNcZ2vJ9cuYOKBB2v6M9TnmZ96vsg0zsHrdHzjJBg4y4/Y0MvDL3KnQiw="
+    }
+  },
+  {
+    "type": "CompositeElement",
+    "element_id": "f84e69662a92e2f5f3469fee3ff0f70e",
+    "text": "All her invitations without exception, written in French, and delivered by a scarlet-liveried footman that morning, ran as follows:\n\n\"If you have nothing better to do, Count (or Prince), and if the prospect of spending an evening with a poor invalid is not too terrible, I shall be very charmed to see you tonight between 7 and 10--Annette Scherer.\"",
+    "metadata": {
+      "filename": "8598c4e156b7.json",
+      "filetype": "application/json",
+      "languages": [
+        "eng"
+      ],
+      "orig_elements": "eJztk9tu1DAQhl9llCuQNpv1JptD7yokpN4AEvSqrVYTe7KxmtiR7exBVd+dcWgBIeABEFfx4c/MN55/7p4SGmgkE/ZaJVeQNLhtqCi7vGhoV9ayJVlIkasSZalyEskKkpECKgzI+qckLvbezk7S65720hF/loCi2m63u6YSxbop8lpslxCLarRKd/o3skJUef1dNjkryftXXSGaapPv8rUomypvomwiN2rvtTV+/wJ295Rw+IiU56Kunx9Y5khap/aDlRisW2gnDH0Mm916cj5zdkSjPclHym6ND26WYXakbj5m80/bVJsD+ZDRGcdpoFRZ6bPW2sf0hC5Fo9KJUFIqpnU4h+T5mZN3eiCDYyRK/ix9EYbLtAgDnUM2DahNvBnQHGbkzLG+hMwhiVVN6H60j6QsSXZK4aapdqoRu6Ik1Qmpik1FdZlElBg1iq+HAXpyoM1RBwzx+eCkQ2/nAHSWNMWjFZycDoEMy+A9p5L9CpgbFA36SPwa0F4AwUt0DJ4uh9xU6KwN/JoQegwwWmf40Vbg+AQ9Xw6DPfmrWNZrsR/QOaY40pfIx6C/epPyqquQe8+OrCshhGpbVZWSzYD1rqj/e/Pf8eZ9ctPBxc7Q45HA2NBzXdASO9FBsKDsCt7Z2QR4Yx18ctpIevvNmLpjzxFwa/xEMoDtgBdGxQDsPjpStOLidPbtZO0yATho/tXHVBzfAudxuh1oBTfge+RRaQnY2heQPbqRDc4UnmiBDNboQx8i34l4VKoFRGzS9NqYyAyfJU8aufV98hfLP3wFMQrOeg=="
+    }
+  },
+  {
+    "type": "CompositeElement",
+    "element_id": "afa7dc06b83ed02316bd1a932d6deacb",
+    "text": "\"Heavens! what a virulent attack!\" replied the prince, not in the least disconcerted by this reception. He had just entered, wearing an embroidered court uniform, knee breeches, and shoes, and had stars on his breast and a serene expression on his flat face. He spoke in that refined French in which our grandfathers not only spoke but thought, and with the gentle, patronizing intonation natural to a man of importance who had grown old in society and at court. He went up to Anna Pavlovna, kissed",
+    "metadata": {
+      "filename": "8598c4e156b7.json",
+      "filetype": "application/json",
+      "languages": [
+        "eng"
+      ],
+      "orig_elements": "eJx1U7ty2zAQ/JUza70oSqKYLk3GaZIUcWV7NCfgQCIiAQ4ASnI8+vfcUVKSIqkIHhaL3b3D83tGLXXk0s7q7ANkxqwLqqigpVrgaqG25dYUeV5qg0T5krIJZB0l1JiQ8e+ZLHbRD0HR/Z92KhB/RsK8XC6X66rMV7NqVWzzZS4UI6rz2hr7D9gqL4vtb1gfvKIY77hVXpWLYl3M8k1VFpXAegqdjdF6F3c3Yc/vGdOLpKLIt9vLK8MCKR/0rvUKkw+j2h5TI7Tzp0ghzoPv0NlI6kDzJxdTGFQaAunPX+fDX79T62qKaU5n7PqWptqrON97f5ieMEzR6WlPqGia97N0Ttnlwpcb25LDThRl/4fegOmtH4GJzmnet2id7LTo6gH5ZvGXkaszcdVj+NM+UmpDymiNi6pc6ypfrzakTa70alHSdpOJFGEV8Ev2SHgkFx/g1GAChKMNA8vkZUqoDg8vGQTqW+4RpIagD9YpmoDzCawbSy1hTKBtVJ63Ajcd9m+8YyOfVNQnbsoMHgka1PBjYCzTE2c4gRMh89WADqjbB2+11EHxLCUYnDU+dBM4OCLYByLVUJwwWENs/H0prDFhiOAdyKWMFEGyh8A9JUdA5z7QOB53lGnZruHcR2mx9we6GuJyIGMd6/jEZ1Uj5VNjecGyoA5MbHhmeFjGFLxr327n90NiAj/UTbpqO9nUjBnVbLnl2HjYgnf2p5i2LnmHkg7wZwjYQvIsmecPvAHb9T4k5Ej5dj/arIM/8V6rRVL0yvKUXH2ma2ajl5N0b+iF7KNzCN/w2PqjQw7SyhsCls50jvOXUBgtahg91iVAbHkvKmmTvudtnaAawltFeZ57FIzYH986H+4itUYiFtPRG5zJ1N5n+QuGwIaP9F3G7/L6C8K9hOs="
+    }
+  },
+  {
+    "type": "CompositeElement",
+    "element_id": "65b57edb5b8b5456487acbfe2a81a34c",
+    "text": "her hand, presenting to her his bald, scented, and shining head, and complacently seated himself on the sofa.",
+    "metadata": {
+      "filename": "8598c4e156b7.json",
+      "filetype": "application/json",
+      "languages": [
+        "eng"
+      ],
+      "orig_elements": "eJx1U7ty2zAQ/JUza70oSqKYLk3GaZIUcWV7NCfgQCIiAQ4ASnI8+vfcUVKSIqkIHhaL3b3D83tGLXXk0s7q7ANkxqwLqqigpVrgaqG25dYUeV5qg0T5krIJZB0l1JiQ8e+ZLHbRD0HR/Z92KhB/RsK8XC6X66rMV7NqVWzzZS4UI6rz2hr7D9gqL4vtb1gfvKIY77hVXpWLYl3M8k1VFpXAegqdjdF6F3c3Yc/vGdOLpKLIt9vLK8MCKR/0rvUKkw+j2h5TI7Tzp0ghzoPv0NlI6kDzJxdTGFQaAunPX+fDX79T62qKaU5n7PqWptqrON97f5ieMEzR6WlPqGia97N0Ttnlwpcb25LDThRl/4fegOmtH4GJzmnet2id7LTo6gH5ZvGXkaszcdVj+NM+UmpDymiNi6pc6ypfrzakTa70alHSdpOJFGEV8Ev2SHgkFx/g1GAChKMNA8vkZUqoDg8vGQTqW+4RpIagD9YpmoDzCawbSy1hTKBtVJ63Ajcd9m+8YyOfVNQnbsoMHgka1PBjYCzTE2c4gRMh89WADqjbB2+11EHxLCUYnDU+dBM4OCLYByLVUJwwWENs/H0prDFhiOAdyKWMFEGyh8A9JUdA5z7QOB53lGnZruHcR2mx9we6GuJyIGMd6/jEZ1Uj5VNjecGyoA5MbHhmeFjGFLxr327n90NiAj/UTbpqO9nUjBnVbLnl2HjYgnf2p5i2LnmHkg7wZwjYQvIsmecPvAHb9T4k5Ej5dj/arIM/8V6rRVL0yvKUXH2ma2ajl5N0b+iF7KNzCN/w2PqjQw7SyhsCls50jvOXUBgtahg91iVAbHkvKmmTvudtnaAawltFeZ57FIzYH986H+4itUYiFtPRG5zJ1N5n+QuGwIaP9F3G7/L6C8K9hOs=",
+      "is_continuation": true
+    }
+  },
+  {
+    "type": "CompositeElement",
+    "element_id": "5efffc7ade628ceaf94ab81acb24eeba",
+    "text": "\"First of all, dear friend, tell me how you are. Set your friend's mind at rest,\" said he without altering his tone, beneath the politeness and affected sympathy of which indifference and even irony could be discerned.\n\n\"Can one be well while suffering morally? Can one be calm in times like these if one has any feeling?\" said Anna Pavlovna. \"You are staying the whole evening, I hope?\"",
+    "metadata": {
+      "filename": "8598c4e156b7.json",
+      "filetype": "application/json",
+      "languages": [
+        "eng"
+      ],
+      "orig_elements": "eJztk8tq3TAQhl9l8Kabc/Pd7iaUQiGbttBmUZpwmCONYhFZMpKcExPy7h2d5rSBXh6gdGWP9PPPRd98fczI0Eg27rXMXkPWV5Uqy1ztqFdF2zR5QUVZHMpGFd2hVk22gmykiBIjsv4xSz/74GYv6BzTXnjiz8kwb4uiqPs2rzZ9VXZ5kSeLk2p0Uiv9G1mVt2X3QzZ5JyiEs67K+3ZX1uUmb/q27JNsIj/qELSzYf9c2NfHjO1TSdxN1z3dsMyTcF7ujRMYnT9VO2Ecku32KpAPW+9GtDqQuKPtlQ3RzyLOnuTlh+38Ilxre0shbukBx8nQWjoRtgfn7tZH9Gu0cj0RClrn0yY+xOzpiZMrbcjimCrK/ix9FsZlOgkjPcTtZFDbdGPQ3s7ImVN/GdnbLHU1of/5fCREQ0JJibu+rWWf11VDUuVCVruWuiZLpSTXJL7O3mkfIjgFaMwKJKEH5TVZuYJIxsBIMLgjLG4GzrKBTxRTcFa9CjBqKwEjeJ7H6jqDgFrCQHDUcXBzZONInscFgw4QnaUVHMgyHQNElk3O6MhxCIDJSCkSDA6EZUxPs6TajoMWA3Aezbfcq6CTlu7JgvbOLiDcbCT7gtRBkLckN2le5ym+R+8x6nv6nBrnCfwCvVK57FRLfSnropE7KpFBwzrv6h2J7j/0/xL0b9ECg5h4OSbImS9DEOZEVyJ1dJ7XYbmAF0KBZmQEIeqRAhh9RwnfQKDVSTJgAngBRWTY4+K8CW+sRfiI98bdW9zAdfbl+ypBiLikZGkJjoPjAhLPfLKCS965idjiLwzffANrJduy"
+    }
+  },
+  {
+    "type": "CompositeElement",
+    "element_id": "01864f9d6a9351da4fad804c0a237577",
+    "text": "\"And the fete at the English ambassador's? Today is Wednesday. I must put in an appearance there,\" said the prince. \"My daughter is coming for me to take me there.\"\n\n\"I thought today's fete had been canceled. I confess all these festivities and fireworks are becoming wearisome.\"\n\n\"If they had known that you wished it, the entertainment would have been put off,\" said the prince, who, like a wound-up clock, by force of habit said things he did not even wish to be believed.",
+    "metadata": {
+      "filename": "8598c4e156b7.json",
+      "filetype": "application/json",
+      "languages": [
+        "eng"
+      ],
+      "orig_elements": "eJztlE1v2zAMhv8K4Usv+fC37F2GHXboYdulxQ5tEdASnQixJUOSmwZF//uotNmGoRuwa1EkQET71UtSesKbx4QGGsmEjVbJB0jaXEqJomzrllSeCpl3RS1Jdhl/aiGSBSQjBVQYkPWPSVxsvJ2dpHNMG+mIf06GmcjzvGpFVq7asmiyPIsWJ9Vole71K7IyE0XzUzY5K8n7s67MWpEWVbHK6lYUbZRN5EbtvbbGb14Ku3lM2D6WVBRZ0zzdscyRtE5tBisxWHeqdsKwi7bra0/Or50d0WhPck/ra+ODm2WYHanLb+v5t3CpzZZ8WNMDjtNAS2WlX3fW7pcHdEs0ajkRSlpm0yo8hOTpiZP3eiCDY6wo+bv0RRiO00kY6CGspwG1iW8GNNsZOXPsLyGzTWJXE7pf10dS1iR7pTBtRaXarCprUn0mVZkKauoklhJdo/g2+WQUhB1BT4EAw2n92WwH7XeAY4feo7Luwn+EK6vwCNrDd1KGPAcruIRx9gGmOYA2gPyduBmHRlJ0crS4TcCjfs4xOc0vVnCbfDmCwnm7C+Sio7Qjnyf01sHIGy0E3NNpGT1Wt0ns/XwiX9E5DPqermIT3M2fAPcya1pM01Y2vSyrVHSqzFVbp1mnuqqu3wF+SwBfMiQ2osTcMJMX/hnlHSroiAzICONAKsIqren5GgCHIaLlI/aeUdJBEz/lv0KvHR2s23PkiA1eyDww1Nrb8b9ZFEWXNqIUVdN3oqyKirBuMtHWBVWpktU7i2+KxT5idTzBtzf2YDjkmXq0Mxx4oJICHRanUcgpyAUuJLICBzsPinfd0zOzcZ7avn9leC7gsLMLGDTPR4z7uON5AsmXsF9Ad4wzlGev7dmt0+G8nQ/YA5sojowNQPecJZYUh20Xsw6an6nVP+C++wFFZ6MV"
+    }
+  },
+  {
+    "type": "CompositeElement",
+    "element_id": "359eeb2d900f33412afa99c4dc3e1672",
+    "text": "\"Don't tease! Well, and what has been decided about Novosiltsev's dispatch? You know everything.\"\n\n\"What can one say about it?\" replied the prince in a cold, listless tone. \"What has been decided? They have decided that Buonaparte has burnt his boats, and I believe that we are ready to burn ours.\"",
+    "metadata": {
+      "filename": "8598c4e156b7.json",
+      "filetype": "application/json",
+      "languages": [
+        "eng"
+      ],
+      "orig_elements": "eJztk8tu2zAQRX9lyk02sWU9rEc3AYpuskk3CYIiCYwRObKISKRAjuwYQf69pOOkRdD2A4quREpn7twhr+6eBQ00kuGNVuIziLytqSq7rEXsMC/qjFTb1GVWyrJTFaI4BzESo0LGwD+LuNh4OztJb3vaSEfhcRRMqyzL1k2VFsumyOs0S6PEkRqt0p3+DVakVV6/Y5Ozkrx/44q0qVb5Ol+mZVPlTcQmcqP2XlvjNydjd88iyEdLeZ7W9ctDwBxJ69RmsBLZuqPbCbmPssmNJ+cTZ0c02pN8pOTGeHaz5NmRuvyWzL9sF9psyXNCTzhOAy2UlT5prX1c7NEt0KjFRChpkU5LfmLx8hKad3ogg2N0JP6MnkA+TEeQ6YmTaUBt4pcBzXbG0DnOJ8hsRZxqQvfz+kjKkmSnFK6aaq2adF2UpLpUqmJVUV2KaCWqRvhefLXmjIEJPX2CWxqGcwiWYN8jQ48eWiIDiqRWpABbOzNc2Z31emBPuzMPSvtwhLK/gO92hkdj90A7cgfuwxEt70W0/TbMFTqHrHd0HfsHIx+z17UyLWSL+apNlcxltmrVqpVtm5eqUrT+n71/KXu3MWQSDVhD4PFwypfmi3sBjqYh3A5wTzA5bSSBNoAg7aDOYdCeh3AtwKF2CSetj4G9gOueDuH1jt4zzBH8MluDwTrTa83sTKjWYWWR/esvcBmkgoNQeizZE4RRgy1Uh9D1WAMheP7vGX/4AVMJvdg="
+    }
+  }
+]