Motivation

The couchbase-kafka-connector allows you to import data from any couchbase bucket into Kafka topics using the Kafka Connect API.

Using the Kafka Connect API, it is possible to stream data between Apache Kafka and other systems using a scalable and reliable tool. Kafka Connect features include:

• A common framework for Kafka connectors
• Distributed and standalone modes
• REST interface
• Automatic offset management
• Distributed and scalable by default
• Streaming/batch integration

Data is loaded from Couchbase using the Database Change Protocol (DCP) and then converted into a record for each document streamed from Couchbase.

All the features of Kafka Connect, such as offset management and fault tolerance, work with the couchbase-kafka-connector. You can restart and kill the processes and they will pick up where they left off, copying only new data.