feat: Kafka source and destination connector (#3176)

Thanks to @tullytim we have a new Kafka source and destination
connector. It also works with hosted Kafka via Confluent.

Documentation will be added to the Docs repo.

Author: potter-potter

CHANGELOG.md

@@ -1,4 +1,4 @@
-## 0.14.8-dev3
+## 0.14.8-dev4
 
 ### Enhancements
 
@@ -21,6 +21,8 @@
 
 * **Expose conversion functions for tables** Adds public functions to convert tables from HTML to the Deckerd format and back
 
+* **Adds Kafka Source and Destination** New source and destination connector added to all CLI ingest commands to support reading from and writing to Kafka streams. Also supports Confluent Kafka.
+
 ### Fixes
 
 * **Fix an error publishing docker images.** Update user in docker-smoke-test to reflect changes made by the amd64 image pull from the "unstructured" "wolfi-base" image.

MANIFEST.in

@@ -38,6 +38,7 @@ include requirements/ingest/gitlab.in
 include requirements/ingest/google-drive.in
 include requirements/ingest/hubspot.in
 include requirements/ingest/jira.in
+include requirements/ingest/kafka.in
 include requirements/ingest/mongodb.in
 include requirements/ingest/notion.in
 include requirements/ingest/onedrive.in

Makefile

@@ -169,6 +169,10 @@ install-ingest-reddit:
 install-ingest-slack:
 	pip install -r requirements/ingest/slack.txt
 
+.PHONY: install-ingest-kafka
+install-ingest-kafka:
+	python3 -m pip install -r requirements/ingest/kafka.txt
+
 .PHONY: install-ingest-wikipedia
 install-ingest-wikipedia:
 	python3 -m pip install -r requirements/ingest/wikipedia.txt

docs/source/ingest/destination_connectors/kafka.rst

@@ -0,0 +1,6 @@
+Unstructured Documentation
+==========================
+
+The Unstructured documentation page has moved! Check out our new and improved docs page at
+`https://docs.unstructured.io <https://docs.unstructured.io>`_ to learn more about our
+products and tools.

docs/source/ingest/source_connectors/kafka.rst

@@ -0,0 +1,6 @@
+Unstructured Documentation
+==========================
+
+The Unstructured documentation page has moved! Check out our new and improved docs page at
+`https://docs.unstructured.io <https://docs.unstructured.io>`_ to learn more about our
+products and tools.

examples/ingest/kafka/ingest.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+
+# Processes the pdf specified in the input path
+# processes the document, and writes to results to a Confluent topic.
+
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
+cd "$SCRIPT_DIR"/../../.. || exit 1
+
+PYTHONPATH=. ./unstructured/ingest/main.py \
+  local \
+  --input-path="<path to the file to be processed/partitioned>" \
+  kafka \
+  --bootstrap-server="<bootstrap server fully qualified hostname>" \
+  --port "<port, likely 9092>" \
+  --topic "<destination topic in confluent>" \
+  --kafka-api-key="<confluent api key>" \
+  --secret="<confluent secret>" \
+  --num-processes="<number of processes to be used>"

requirements/ingest/kafka.in

@@ -0,0 +1,3 @@
+-c ../deps/constraints.txt
+-c ../base.txt
+confluent-kafka

requirements/ingest/kafka.txt

@@ -0,0 +1,8 @@
+#
+# This file is autogenerated by pip-compile with Python 3.9
+# by the following command:
+#
+#    pip-compile ./ingest/kafka.in
+#
+confluent-kafka==2.4.0
+    # via -r ./ingest/kafka.in

scripts/kafka-test-helpers/create-kafka-instance.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+set -e
+
+SCRIPT_DIR=$(dirname "$(realpath "$0")")
+
+# Create the Weaviate instance
+docker-compose version
+docker-compose -f "$SCRIPT_DIR"/docker-compose.yml up --wait
+docker-compose -f "$SCRIPT_DIR"/docker-compose.yml ps
+
+echo "Instance is live."

scripts/kafka-test-helpers/docker-compose.yml

@@ -0,0 +1,22 @@
+services:
+  zookeeper:
+    image: confluentinc/cp-zookeeper:latest
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_TICK_TIME: 2000
+    ports:
+      - 22181:2181
+
+  kafka:
+    image: confluentinc/cp-kafka:latest
+    depends_on:
+      - zookeeper
+    ports:
+      - 29092:29092
+    environment:
+      KAFKA_BROKER_ID: 1
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1