Skip to content

Commit f5d9ab2

Browse files
BillFarberBillFarber
committed
Trying to figure out the intermittent Jenkins failures.
Splitting into 2 docker-compose files to facilitate automated and manual testing.
1 parent b4bce2e commit f5d9ab2

File tree

4 files changed

+269
-238
lines changed

4 files changed

+269
-238
lines changed

‎CONTRIBUTING.md‎

Lines changed: 56 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
This guide describes how to develop and contribute pull requests to this connector. The focus is currently on how to
2-
develop and test the connector, either via a Docker cluster install of Confluent Platform or of the regular Kafka
3-
distribution.
2+
develop and test the connector. There are two methods available - automated and manual. Both methods are performed via a
3+
Docker stack. The automated tests stack creates MarkLogic, Sonar, and Postgres instance for the automated tests. The
4+
manual tests use Confluent Platform in a different Docker stack to allow testing the connector via Confluent Control
5+
Center with a MarkLogic instance in the same stack.
46

57
### Requirements:
68
* MarkLogic Server 11+
@@ -10,7 +12,7 @@ See [the Confluent compatibility matrix](https://docs.confluent.io/platform/curr
1012
for more information. After installing your desired version of Java, ensure that the `JAVA_HOME` environment variable
1113
points to your Java installation.
1214

13-
# Configuring Local Automated and Manual Testing
15+
# Configuring Local Automated Testing
1416

1517
The test suite for the MarkLogic Kafka connector, found at `src/test`, requires that the test application first be
1618
deployed to a MarkLogic instance. The recommendation is for this application to be deployed via Docker and
@@ -19,51 +21,17 @@ deployed to a MarkLogic instance. The recommendation is for this application to
1921
Note that you do not need to install [Gradle](https://gradle.org/) - the "gradlew" program used below will install the
2022
appropriate version of Gradle if you do not have it installed already.
2123

22-
## Virtual Server Preparation
23-
The project includes a docker-compose file in the repository root that includes MarkLogic, SonarQube with a Postgres server, and Confluent
24-
Platform servers.
25-
26-
### Confluent Platform
27-
[Confluent Platform](https://docs.confluent.io/platform/current/overview.html) provides an easy mechanism for running
28-
Kafka locally via a single Docker cluster. A primary benefit of testing with Confluent Platform is to test configuring
29-
the MarkLogic Kafka connector via the
30-
[Confluent Control Center](https://docs.confluent.io/platform/current/control-center/index.html) web application.
31-
The Confluent Platform servers in this docker-compose file are based on the Confluent files and instructions at
32-
[Install a Confluent Platform cluster in Docker using a Confluent docker-compose file](https://docs.confluent.io/platform/current/platform-quickstart.html).
33-
34-
## Docker Cluster Preparation
35-
To setup the docker cluster, use the docker-compose file in the repository root to build the Docker cluster with
36-
the command:
24+
## Docker Cluster Preparation for Automated Testing
25+
The automated tests require a MarkLogic server, SonarQube server, and Postgres server. The docker-compose file in the
26+
repository root includes these services. To prepare for running the automated tests, perform the following steps:
3727
```
3828
docker-compose up -d --build
3929
```
40-
When the setup is complete, you should be able to run
41-
```
42-
docker-compose ps
43-
```
44-
and see results similar to the following.
45-
```
46-
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
47-
broker confluentinc/cp-kafka:7.6.1 "/etc/confluent/dock…" broker 14 minutes ago Up 14 minutes 0.0.0.0:9092->9092/tcp, 0.0.0.0:9101->9101/tcp
48-
connect cnfldemos/cp-server-connect-datagen:0.6.4-7.6.0 "/etc/confluent/dock…" connect 14 minutes ago Up 14 minutes 0.0.0.0:8083->8083/tcp, 9092/tcp
49-
control-center confluentinc/cp-enterprise-control-center:7.6.1 "/etc/confluent/dock…" control-center 14 minutes ago Up 14 minutes 0.0.0.0:9021->9021/tcp
50-
ksql-datagen confluentinc/ksqldb-examples:7.6.1 "bash -c 'echo Waiti…" ksql-datagen 14 minutes ago Up 14 minutes
51-
ksqldb-cli confluentinc/cp-ksqldb-cli:7.6.1 "/bin/sh" ksqldb-cli 14 minutes ago Up 14 minutes
52-
ksqldb-server confluentinc/cp-ksqldb-server:7.6.1 "/etc/confluent/dock…" ksqldb-server 14 minutes ago Up 14 minutes 0.0.0.0:8088->8088/tcp
53-
marklogic marklogicdb/marklogic-db:11.2.0-centos-1.1.2 "/tini -- /usr/local…" marklogic 14 minutes ago Up 14 minutes 25/tcp, 7997-7999/tcp, 0.0.0.0:8000-8002->8000-8002/tcp, 0.0.0.0:8010-8013->8010-8013/tcp, 8003-8009/tcp, 0.0.0.0:8018-8019->8018-8019/tcp
54-
marklogic-kafka-confluent-postgres-1 postgres:15-alpine "docker-entrypoint.s…" postgres 14 minutes ago Up 14 minutes 5432/tcp
55-
marklogic-kafka-confluent-sonarqube-1 sonarqube:10.3.0-community "/opt/sonarqube/dock…" sonarqube 14 minutes ago Up 14 minutes 0.0.0.0:9000->9000/tcp
56-
rest-proxy confluentinc/cp-kafka-rest:7.6.1 "/etc/confluent/dock…" rest-proxy 14 minutes ago Up 14 minutes 0.0.0.0:8082->8082/tcp
57-
schema-registry confluentinc/cp-schema-registry:7.6.1 "/etc/confluent/dock…" schema-registry 14 minutes ago Up 14 minutes 0.0.0.0:8081->8081/tcp
58-
```
5930

60-
You can now visit several web applications:
31+
You can now visit these web applications:
6132
* http://localhost:8000 to access the MarkLogic server.
6233
* http://localhost:9000 to use the SonarQube server as described in the "Running Sonar Code Analysis"
6334
section below.
64-
* http://localhost:9021 to access
65-
[Confluent's Control Center GUI](https://docs.confluent.io/platform/current/control-center/index.html) application.
66-
Within Control Center, click on "controlcenter.cluster" to access the configuration for the Kafka cluster.
6735

6836
## MarkLogic Preparation
6937
To prepare the MarkLogic server for automated testing as well as testing with the Confluent Platform, the Data Hub based
@@ -100,8 +68,8 @@ To configure the SonarQube service, perform the following steps:
10068
8. On the "Analysis Method" page, click on "Locally".
10169
9. In the "Provide a token" panel, click on "Generate". Copy the token.
10270
10. Click the "Continue" button.
103-
11. Update `systemProp.sonar.token=<Replace With Your Sonar Token>` in `gradle-local.properties` in the root of your
104-
project.
71+
11. Update `systemProp.sonar.token=<Replace With Your Sonar Token>` in `gradle-local.properties` in the root directory
72+
of your project.
10573

10674
To run the SonarQube analysis, run the following Gradle task in the root directory, which will run all the tests with
10775
code coverage and then generate a quality report with SonarQube:
@@ -125,16 +93,59 @@ without having to re-run the tests.
12593
For more assistance with Sonar and Gradle, see the
12694
[Sonar Gradle plugin docs](https://docs.sonarqube.org/latest/analyzing-source-code/scanners/sonarscanner-for-gradle/).
12795

96+
# Configuring Local Manual Testing
97+
This project includes a Docker Compose file that creates a Kafka cluster using Confluent Platform along with a
98+
MarkLogic server. This allows you to test the MarkLogic Kafka connector via the Confluent Control Center web
99+
application. The instructions below describe how to get started.
100+
101+
## Docker Cluster Preparation for Manual Testing
102+
The docker-compose file in the test-app directory includes these services along with a MarkLogic server.
103+
```
104+
docker-compose --env-file ./.env -f test-app/docker-compose.yml up -d --build
105+
```
106+
107+
When the setup is complete, you should be able to run
108+
```
109+
docker-compose --env-file ./.env -f test-app/docker-compose.yml ps
110+
```
111+
and see results similar to the following.
112+
```
113+
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
114+
broker confluentinc/cp-kafka:7.6.1 "/etc/confluent/dock…" broker 14 minutes ago Up 14 minutes 0.0.0.0:9092->9092/tcp, 0.0.0.0:9101->9101/tcp
115+
connect cnfldemos/cp-server-connect-datagen:0.6.4-7.6.0 "/etc/confluent/dock…" connect 14 minutes ago Up 14 minutes 0.0.0.0:8083->8083/tcp, 9092/tcp
116+
control-center confluentinc/cp-enterprise-control-center:7.6.1 "/etc/confluent/dock…" control-center 14 minutes ago Up 14 minutes 0.0.0.0:9021->9021/tcp
117+
ksql-datagen confluentinc/ksqldb-examples:7.6.1 "bash -c 'echo Waiti…" ksql-datagen 14 minutes ago Up 14 minutes
118+
ksqldb-cli confluentinc/cp-ksqldb-cli:7.6.1 "/bin/sh" ksqldb-cli 14 minutes ago Up 14 minutes
119+
ksqldb-server confluentinc/cp-ksqldb-server:7.6.1 "/etc/confluent/dock…" ksqldb-server 14 minutes ago Up 14 minutes 0.0.0.0:8088->8088/tcp
120+
marklogic marklogicdb/marklogic-db:11.2.0-centos-1.1.2 "/tini -- /usr/local…" marklogic 14 minutes ago Up 14 minutes 25/tcp, 7997-7999/tcp, 0.0.0.0:8000-8002->8000-8002/tcp, 0.0.0.0:8010-8013->8010-8013/tcp, 8003-8009/tcp, 0.0.0.0:8018-8019->8018-8019/tcp
121+
marklogic-kafka-confluent-postgres-1 postgres:15-alpine "docker-entrypoint.s…" postgres 14 minutes ago Up 14 minutes 5432/tcp
122+
marklogic-kafka-confluent-sonarqube-1 sonarqube:10.3.0-community "/opt/sonarqube/dock…" sonarqube 14 minutes ago Up 14 minutes 0.0.0.0:9000->9000/tcp
123+
rest-proxy confluentinc/cp-kafka-rest:7.6.1 "/etc/confluent/dock…" rest-proxy 14 minutes ago Up 14 minutes 0.0.0.0:8082->8082/tcp
124+
schema-registry confluentinc/cp-schema-registry:7.6.1 "/etc/confluent/dock…" schema-registry 14 minutes ago Up 14 minutes 0.0.0.0:8081->8081/tcp
125+
```
126+
127+
You can now visit several web applications:
128+
* http://localhost:8000 to access the MarkLogic server.
129+
* http://localhost:9021 to access
130+
[Confluent's Control Center GUI](https://docs.confluent.io/platform/current/control-center/index.html) application.
131+
Within Control Center, click on "controlcenter.cluster" to access the configuration for the Kafka cluster.
132+
133+
### Confluent Platform for Manual Testing
134+
[Confluent Platform](https://docs.confluent.io/platform/current/overview.html) provides an easy mechanism for running
135+
Kafka locally via a single Docker cluster. A primary benefit of testing with Confluent Platform is to test configuring
136+
the MarkLogic Kafka connector via the
137+
[Confluent Control Center](https://docs.confluent.io/platform/current/control-center/index.html) web application.
138+
The Confluent Platform servers in this docker-compose file are based on the Confluent files and instructions at
139+
[Install a Confluent Platform cluster in Docker using a Confluent docker-compose file](https://docs.confluent.io/platform/current/platform-quickstart.html).
128140

129-
## Confluent Platform for Manual Testing
130141

131142
### Building and Sharing the Connector with the Docker Container
132143
Using gradle in the root directory, build the connector archive and copy it to a directory shared with the Confluent
133144
Platform Docker cluster built in the that section, using this gradle command in the root directory:
134145
```
135146
./gradlew copyConnectorToDockerVolume
136147
```
137-
**You MUST restart the "connect" server in the Docker "confluent-platform-example" cluster.**
148+
**You MUST restart the "connect" server in the Docker "manual-tests-marklogic-kafka-confluent" cluster.**
138149

139150
Now, verify the connector has loaded properly.
140151
1. Click on "Connect" in the left sidebar.

‎Jenkinsfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,14 +9,15 @@ def runtests(String marklogicVersion) {
99
sudo /usr/local/sbin/mladmin cleandata
1010
cd kafka-connector
1111
MARKLOGIC_LOGS_VOLUME=/tmp MARKLOGIC_IMAGE='''+marklogicVersion+''' docker-compose up -d --build
12-
sleep 120s;
12+
sleep 60s;
1313
'''
1414
sh label:'deploy project', script: '''#!/bin/bash
1515
export JAVA_HOME=$JAVA17_HOME_DIR
1616
export GRADLE_USER_HOME=$WORKSPACE/$GRADLE_DIR
1717
export PATH=$GRADLE_USER_HOME:$JAVA_HOME/bin:$PATH
1818
cd kafka-connector
1919
./gradlew hubInit
20+
./gradlew mlTestConnections -PmlUsername=admin
2021
./gradlew -i mlDeploy
2122
'''
2223
sh label:'test', script: '''#!/bin/bash

‎docker-compose.yml‎

Lines changed: 1 addition & 192 deletions
Original file line numberDiff line numberDiff line change
@@ -1,197 +1,6 @@
11
---
2-
name: marklogic-kafka-confluent
2+
name: docker-tests-marklogic-kafka-confluent
33
services:
4-
5-
# This compose file is based on:
6-
# This guide - https://docs.confluent.io/platform/current/platform-quickstart.html#step-6-uninstall-and-clean-up
7-
# This compose file - https://raw.githubusercontent.com/confluentinc/cp-all-in-one/7.6.1-post/cp-all-in-one-kraft/docker-compose.yml
8-
# Extended to include a MarkLogic container
9-
10-
broker:
11-
image: confluentinc/cp-kafka:7.6.1
12-
hostname: broker
13-
container_name: broker
14-
ports:
15-
- "9092:9092"
16-
- "9101:9101"
17-
18-
# The NET_RAW capability allows a process to create raw sockets. Polaris does not like that.
19-
# This setting removes the NET_RAW capability from the container.
20-
cap_drop:
21-
- NET_RAW
22-
environment:
23-
KAFKA_NODE_ID: 1
24-
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
25-
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
26-
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
27-
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
28-
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
29-
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
30-
KAFKA_JMX_PORT: 9101
31-
KAFKA_JMX_HOSTNAME: localhost
32-
KAFKA_PROCESS_ROLES: 'broker,controller'
33-
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
34-
KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'
35-
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
36-
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
37-
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
38-
# Replace CLUSTER_ID with a unique base64 UUID using "bin/kafka-storage.sh random-uuid"
39-
# See https://docs.confluent.io/kafka/operations-tools/kafka-tools.html#kafka-storage-sh
40-
CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
41-
42-
schema-registry:
43-
image: confluentinc/cp-schema-registry:7.6.1
44-
hostname: schema-registry
45-
container_name: schema-registry
46-
depends_on:
47-
- broker
48-
ports:
49-
- "8081:8081"
50-
cap_drop:
51-
- NET_RAW
52-
environment:
53-
SCHEMA_REGISTRY_HOST_NAME: schema-registry
54-
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:29092'
55-
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
56-
57-
connect:
58-
image: cnfldemos/cp-server-connect-datagen:0.6.4-7.6.0
59-
hostname: connect
60-
container_name: connect
61-
depends_on:
62-
- broker
63-
- schema-registry
64-
ports:
65-
- "8083:8083"
66-
cap_drop:
67-
- NET_RAW
68-
environment:
69-
CONNECT_BOOTSTRAP_SERVERS: 'broker:29092'
70-
CONNECT_REST_ADVERTISED_HOST_NAME: connect
71-
CONNECT_GROUP_ID: compose-connect-group
72-
CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
73-
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
74-
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
75-
CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
76-
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
77-
CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
78-
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
79-
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
80-
CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
81-
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
82-
# CLASSPATH required due to CC-2422
83-
CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-7.6.1.jar
84-
CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
85-
CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
86-
CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components,/usr/share/confluent-marklogic-components"
87-
CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR
88-
volumes:
89-
- ./docker/confluent-marklogic-components:/usr/share/confluent-marklogic-components
90-
91-
control-center:
92-
image: confluentinc/cp-enterprise-control-center:7.6.1
93-
hostname: control-center
94-
container_name: control-center
95-
depends_on:
96-
- broker
97-
- schema-registry
98-
- connect
99-
- ksqldb-server
100-
ports:
101-
- "9021:9021"
102-
cap_drop:
103-
- NET_RAW
104-
environment:
105-
CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker:29092'
106-
CONTROL_CENTER_CONNECT_CONNECT-DEFAULT_CLUSTER: 'connect:8083'
107-
CONTROL_CENTER_CONNECT_HEALTHCHECK_ENDPOINT: '/connectors'
108-
CONTROL_CENTER_KSQL_KSQLDB1_URL: "http://ksqldb-server:8088"
109-
CONTROL_CENTER_KSQL_KSQLDB1_ADVERTISED_URL: "http://localhost:8088"
110-
CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
111-
CONTROL_CENTER_REPLICATION_FACTOR: 1
112-
CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
113-
CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
114-
CONFLUENT_METRICS_TOPIC_REPLICATION: 1
115-
PORT: 9021
116-
117-
ksqldb-server:
118-
image: confluentinc/cp-ksqldb-server:7.6.1
119-
hostname: ksqldb-server
120-
container_name: ksqldb-server
121-
depends_on:
122-
- broker
123-
- connect
124-
ports:
125-
- "8088:8088"
126-
cap_drop:
127-
- NET_RAW
128-
environment:
129-
KSQL_CONFIG_DIR: "/etc/ksql"
130-
KSQL_BOOTSTRAP_SERVERS: "broker:29092"
131-
KSQL_HOST_NAME: ksqldb-server
132-
KSQL_LISTENERS: "http://0.0.0.0:8088"
133-
KSQL_CACHE_MAX_BYTES_BUFFERING: 0
134-
KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
135-
KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
136-
KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
137-
KSQL_KSQL_CONNECT_URL: "http://connect:8083"
138-
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1
139-
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true'
140-
KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true'
141-
142-
ksqldb-cli:
143-
image: confluentinc/cp-ksqldb-cli:7.6.1
144-
container_name: ksqldb-cli
145-
depends_on:
146-
- broker
147-
- connect
148-
- ksqldb-server
149-
entrypoint: /bin/sh
150-
tty: true
151-
cap_drop:
152-
- NET_RAW
153-
154-
ksql-datagen:
155-
image: confluentinc/ksqldb-examples:7.6.1
156-
hostname: ksql-datagen
157-
container_name: ksql-datagen
158-
depends_on:
159-
- ksqldb-server
160-
- broker
161-
- schema-registry
162-
- connect
163-
command: "bash -c 'echo Waiting for Kafka to be ready... && \
164-
cub kafka-ready -b broker:29092 1 40 && \
165-
echo Waiting for Confluent Schema Registry to be ready... && \
166-
cub sr-ready schema-registry 8081 40 && \
167-
echo Waiting a few seconds for topic creation to finish... && \
168-
sleep 11 && \
169-
tail -f /dev/null'"
170-
cap_drop:
171-
- NET_RAW
172-
environment:
173-
KSQL_CONFIG_DIR: "/etc/ksql"
174-
STREAMS_BOOTSTRAP_SERVERS: broker:29092
175-
STREAMS_SCHEMA_REGISTRY_HOST: schema-registry
176-
STREAMS_SCHEMA_REGISTRY_PORT: 8081
177-
178-
rest-proxy:
179-
image: confluentinc/cp-kafka-rest:7.6.1
180-
depends_on:
181-
- broker
182-
- schema-registry
183-
ports:
184-
- 8082:8082
185-
hostname: rest-proxy
186-
container_name: rest-proxy
187-
cap_drop:
188-
- NET_RAW
189-
environment:
190-
KAFKA_REST_HOST_NAME: rest-proxy
191-
KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092'
192-
KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"
193-
KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
194-
1954
marklogic:
1965
image: "${MARKLOGIC_IMAGE}"
1976
platform: linux/amd64

0 commit comments

Comments
 (0)