Architectural Guidance for neo4j-streams (Antora docs) (#337)

* carry-over of architectural guidance doc for neo4j-streams

Author: moxious

doc/docs/modules/ROOT/images/graph-etl.png

@@ -54,6 +54,16 @@
 
 * xref::cloud.adoc[Confluent Cloud]
 
+* xref::architecture.adoc[Architectural Guidance]
+** xref::architecture/kafkatopics.adoc[Kafka Topics Overview]
+** xref::architecture/pluginvsconnect.adoc[Kafka Connect vs. Plugin]
+** xref::architecture/sinkconsume.adoc[Sink: Consume Records]
+** xref::architecture/sourceproduce.adoc[Source: Produce Records]
+** xref::architecture/retries.adoc[Retries & Error Handling]
+** xref::architecture/throughput.adoc[Factors which Affect Throughput]
+** xref::architecture/transformations.adoc[Streams Transformations]
+** xref::architecture/optimize.adoc[Optimizing Kafka]
+
 * xref::examples.adoc[Examples with Confluent Platform and Kafka Connect Datagen]
 ** xref::examples.adoc#examples_binary_format[Confluent and Neo4j in binary format]
 ** xref::examples.adoc#confluent_docker_example[Confluent with Docker, Neo4j in binary format]

doc/docs/modules/ROOT/images/kafka-partitions.png

@@ -0,0 +1,21 @@
+[[architecture]]
+== Architectural Guidance
+
+The purpose of this section is to:
+
+* Describe how this integration works
+* Discuss the architectural issues that are at play
+* Provide a basis for advice on how to build systems with Neo4j & Kafka
+
+[NOTE]
+Customers should use the information in this document to design the best possible pipelines to connect graphs and streams.
+
+== The Challenge:  Graph ETL
+
+Data coming from Kafka isn't a graph.  When we load JSON documents or Avro into a Graph, what we're actually doing is extracting relevant bits of data from some input message.  We're then transforming those relevant bits into a "Graph Snippet" and then loading it into Neo4j.
+
+image::graph-etl.png[align="center"]
+
+Using neo4j-streams is a form of graph ETL.   And it pays to separate out these two pieces (the extraction and the transformation) and handle them separately if we want to do this in a performant and easy to maintain manner.   If we are producing records from Neo4j back out to Kafka, it's still the same challenge, just in the opposite direction.
+
+https://www.confluent.io/blog/building-real-time-streaming-etl-pipeline-20-minutes/[Streaming ETL is nothing new for Kafka] -- it is one of the platform's core use cases.   A big complicating factor for Neo4j-streams is that not many people have done it for graph before neo4j-streams.

doc/docs/modules/ROOT/images/transaction-event-handler.png

@@ -0,0 +1,24 @@
+= Kafka Topics Overview
+
+image::kafka-partitions.png[align="center"]
+
+Each topic (message queue) is broken into any number of partitions, which can accept independent reads and writes.   Producers push messages onto the end of the queue, and readers advance through the partition offsets.
+
+== Partitions and Offsets
+
+Partitions in Kafka are generally read by a client in "round robin" fashion.  Partition offsets describe where a given reader is in a given partition.  In a simple example, given a single partition, Bob may be currently reading at offset 3 while Sarah is at offset 11.   Either reader can selectively "rewind" or "replay" any part of the message queue if they like.
+
+== Compaction & Retention
+
+Kafka topics can be configured with various "compaction" and "retention" settings that control how long the Kafka cluster keeps parts of the topic.  If all history on the topic is retained, then in theory you can reconstruct all of history by playing it from the beginning.
+
+== Database Replication
+
+All databases in Kafka land can be thought of as generic data buckets that can emit messages (producers) or can consume messages (consumers).  A common technique in Kafka is to set up one database to publish all its ongoing flow of changes (CDC) to a Kafka topic.  Generally, the Debezium format is used for this, but not exclusively.
+
+**This is an important concept for database replication**, which is common.  If you are ingesting from a Kafka topic and you have a setup with decent performance, you may be able to 100% recreate a database by "replaying the topic" into a new Neo4j database.
+
+== Polyglot Persistence
+
+Given all of this, an important architectural pattern to be aware of is the idea of having one single "Source of Truth" database (such as Oracle) - which publishes all of its changes to Kafka, feeding multiple downstream "helper systems" such as ElasticSearch and Neo4j.  In this way, copies of the data are in 3 different places.   The "helper systems" probably don't accept writes, and just add new query capabilities to the existing data.   High Availability may be less of a topic for the helper systems, as they can always be recreated from the topic.
+

doc/docs/modules/ROOT/images/transformer-architecture.png

@@ -0,0 +1,10 @@
+= Optimizing Kafka
+
+Neo4j can't ingest fast if Kafka isn't set up correctly.   While this isn't a common source of problems, it has come up.  Confluent has https://www.confluent.io/blog/optimizing-apache-kafka-deployment/[good overall documentation] on optimizing Kafka that is worth being familiar with.
+
+The main trade offs are these, and they have to make sense at the Kafka layer before they can make sense for Neo4j.
+
+* Do you want to optimize for high throughput, which is the rate that data is moved from producers to brokers or brokers to consumers?
+* Do you want to optimize for low latency, which is the elapsed time moving messages end-to-end (from producers to brokers to consumers)? 
+* Do you want to optimize for high durability, which guarantees that messages that have been committed will not be lost?
+* Do you want to optimize for high availability, which minimizes downtime in case of unexpected failures? Kafka is a distributed system, and it is designed to tolerate failures.

doc/docs/modules/ROOT/images/unwind-consume.png

@@ -0,0 +1,38 @@
+= When to use Kafka Connect vs. Neo4j Streams as a Plugin
+
+[abstract]
+This section covers how to decide whether to run as a Kafka Connect worker, or as a Neo4j Plugin.
+
+== Kafka Connect
+
+=== Pros
+
+* Processing is outside of Neo4j so that memory & CPU impact doesn't impact Neo4j.  You don't need to size the database with Kafka utilization in mind.
+* Much easier for Kafka pros to manage; they benefit from the Confluent ecosystem, such as connecting the REST API to manipulate connectors, the control center to administer & monitor them.
+* By restarting the worker, you can restart your sink strategy without having downtime for Neo4j.
+* Upgrade Neo4j-Streams without restarting the cluster
+Strictly an external bolt client, so better overall security management of plugin actions.
+
+=== Cons
+
+* You can't do TransactionEventHandlers from outside of the database, so you can only sink to Neo4j, you can't produce from it.
+* If you're using Confluent Cloud, you can't host the connector in the cloud (yet).   So this requires a 3rd piece of architecture:  Confluent Cloud, Neo4j, and the Connect Worker (usually a separate VM)
+* Possibly worse throughput due to bolt latency & overhead, and separate network hop.
+
+== Neo4j-Streams Plugin
+
+=== Pros
+
+* Much easier for Neo4j pros to manage
+* You can produce records back out to Kafka
+* You can use Neo4j procedures so that "custom produce" (see later section) becomes an option.
+* Possibly better throughput, because you don't have bolt latency / overhead
+
+=== Cons
+
+* Memory & CPU consumption on your Neo4j Server
+* Requires restarting Neo4j in order to update your configuration and/or Cypher.
+* Upgrading plugin requires cluster restart
+* Need to track config to be identical across all members in the cluster
+* Lesser ability to manage the plugin because it is running inside of the database and not under a particular user account.
+