Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

By using the IBM Connectivity Pack, Connectivity Pack Kafka connectors enable data streaming between external systems and Kafka.

**Note:**
- Connectivity Pack v2.0.0 and earlier are compatible with Event Streams 11.8 and earlier, but not compatible with Kafka Connect 4.0.0.
- Connectivity Pack v3.0.0 is compatible with Event Streams 12.0.0 and later, and compatible with Kafka Connect 4.0.0.

## Contents

- [Prerequisites](#prerequisites)
Expand Down Expand Up @@ -35,7 +39,7 @@ helm install <RELEASE-NAME> <CONNECTIVITY-PACK-HELM-CHART-URL> --set license.lic
Where:

- `<RELEASE-NAME>` is the release name of your choice. For example, `ibm-connectivity-pack`
- `<CONNECTIVITY-PACK-HELM-CHART-URL>` is the URL of the latest version of the Connectivity Pack Helm chart. For example: `https://github.com/ibm-messaging/connectivity-pack-kafka-connectors/releases/download/2.0.0/ibm-connectivity-pack-2.0.0.tgz`
- `<CONNECTIVITY-PACK-HELM-CHART-URL>` is the URL of the latest version of the Connectivity Pack Helm chart. For example: `https://github.com/ibm-messaging/connectivity-pack-kafka-connectors/releases/download/3.0.0/ibm-connectivity-pack-3.0.0.tgz`
- `license.licenseId=<LICENSE-ID>` is the license identifier (ID) for the program that you purchased. For more information, see [licensing reference](https://ibm.github.io/event-automation/support/licensing/).
- `license.accept` determines whether the license is accepted (default is `false` if not specified).
- `<NAMESPACE>` is the namespace where you want to install the Connectivity Pack. This must be in the same namespace where an Event Streams instance is deployed.
Expand Down Expand Up @@ -204,7 +208,6 @@ Configure your connector with information about your external system by followin
connectivitypack.source.<customObject>.CREATED_POLLER.subscription.pollingInterval: <pollingInterval>
connectivitypack.source.<customObject>.CREATED_POLLER.subscription.timezone: <timezone>
connectivitypack.source.<customObject>.CREATED_POLLER.subscription.updatedField: <the field name that contains the timestamp of when the object was updated>
connectivitypack.source.<customObject>.CREATED_POLLER.subscription.timeFormat: <format of the time stamp>
connectivitypack.source.<customObject>.CREATED_POLLER.subscription.createdField: <the field name that contains the timestamp of when the object was created>

# Optional, sets the format for Kafka topic names created by the connector.
Expand Down Expand Up @@ -281,4 +284,4 @@ Configure your connector with information about your external system by followin

Copyright IBM Corp. 2025

IBM Connectivity Pack is licensed under the [IBM Event Automation license and IBM Cloud Pak for Integration license](https://ibm.biz/ea-license), while the Helm chart and documentation are licensed under the [Apache License, Version 2.0](./ibm-connectivity-pack/license.md).
IBM Connectivity Pack is licensed under the [IBM Event Automation license, IBM Cloud Pak for Integration license, and IBM webMethods Hybrid Integration license](https://ibm.biz/ea-license), while the Helm chart and documentation are licensed under the [Apache License, Version 2.0](./ibm-connectivity-pack/license.md).
1 change: 0 additions & 1 deletion examples/kafka-connector-source.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ spec:
connectivitypack.source.Test_CustomObject__c.CREATED_POLLER.subscription.pollingInterval: 1
connectivitypack.source.Test_CustomObject__c.CREATED_POLLER.subscription.timezone: UTC
connectivitypack.source.Test_CustomObject__c.CREATED_POLLER.subscription.updatedField: LastModifiedDate
connectivitypack.source.Test_CustomObject__c.CREATED_POLLER.subscription.timeFormat: 'YYYY-MM-DDTHH:mm:ssZ'
connectivitypack.source.Test_CustomObject__c.CREATED_POLLER.subscription.createdField: CreatedDate

# Optional, sets the format for Kafka topic names created by the connector.
Expand Down
4 changes: 2 additions & 2 deletions ibm-connectivity-pack/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
apiVersion: v2
appVersion: 1.4.0
appVersion: 1.4.8
description: Helm chart for IBM Connectivity Pack. The helm application is a Connectors as a Container offering that offers connectors as independently deployable application and provides a consistent interface for interaction. The application offers both action and event connectors.
keywords:
- ibm-connectivity-pack
Expand All @@ -19,4 +19,4 @@ keywords:
kubeVersion: '>=1.25.0'
name: ibm-connectivity-pack
type: application
version: 2.0.0
version: 3.0.0
16 changes: 8 additions & 8 deletions ibm-connectivity-pack/connector-config.json
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
{
"action-connectors": {
"tag": "2.0.0",
"digest": "sha256:800b56a821200174ac82e12d5b2649434c1fdadfc1d2061d02be99fe3b4cedd7"
"tag": "3.0.0",
"digest": "sha256:cbe955a55b1652b408c7622699dbf657cf6137dcd1eca97e9bb70ffb424c6695"
},
"event-connectors": {
"tag": "2.0.0",
"digest": "sha256:2368806303cbc30420bd0ba63274219a7d8f4c359dfe378f38624c6d15db26bc"
"tag": "3.0.0",
"digest": "sha256:2b116da33ce5202035374d1443e4af66e04223f174e033fbaf555df6aa6224aa"
},
"java-tech-connectors": {},
"connectivity-pack-proxy": {
"tag": "2.0.0",
"digest": "sha256:887bb7629a16f5246e8777bc4cf2707347d4c537461f0bd4aa523cb84145df6d"
"tag": "3.0.0",
"digest": "sha256:708c72ca56391e6d336bc549546788aee66985a25c39880dd11ebd136b1c9ccd"
},
"connectivity-pack-prehook": {
"tag": "2.0.0",
"digest": "sha256:c544bc6336c92e6002042fc8fdd84d2b5e2d09ca6e8cd53d9a4575ea3c2bf14f"
"tag": "3.0.0",
"digest": "sha256:8d4c21d45c8030cff35cc2e65c36567b20805691946ebde5442a8b0d5bb35929"
}
}
68 changes: 67 additions & 1 deletion ibm-connectivity-pack/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -303,14 +303,23 @@ Create the name for Horizontal Pod Autoscaler
{{- end }}

{{/*
Create the name of the service
Create the name of prehook job
*/}}
{{- define "ibm-connectivity-pack.preHookJob" -}}
{{- if .Release.Name }}
{{- default .Release.Name }}-prehook-job
{{- end }}
{{- end }}

{{/*
Create the name of the posthook job
*/}}
{{- define "ibm-connectivity-pack.postDeleteHookJob" -}}
{{- if .Release.Name }}
{{- default .Release.Name }}-post-delete-job
{{- end }}
{{- end }}


{{/*
Create the name of the service
Expand Down Expand Up @@ -356,4 +365,61 @@ Create the name of the service
{{- if .Release.Name }}
{{- default .Release.Name }}-env-config
{{- end }}
{{- end }}

{{/*
Create UDA connector configmap volume
*/}}
{{- define "ibm-connectivity-pack.udaConnectors" -}}
{{- if .Values.action.udaPersistentVolumeClaimName }}
- name: {{ .Values.action.udaPersistentVolumeClaimName }}-uda-vol
persistentVolumeClaim:
claimName: {{ .Values.action.udaPersistentVolumeClaimName }}
{{- else }}
{{- $files := .Files.Glob "connectors/*.{json,yaml,yml,car}" }}
{{- range $path, $file := $files }}
- name: {{ base $path | replace "." "-" | lower }}-vol
configMap:
name: {{ base $path | replace "." "-" | lower }}-config
{{- end }}
{{- end }}
{{- end }}

{{/*
Create UDA connector configmap volume mount
*/}}
{{- define "ibm-connectivity-pack.udaConnectorsMount" -}}
{{- if .Values.action.udaPersistentVolumeClaimName }}
- name: {{ .Values.action.udaPersistentVolumeClaimName }}-uda-vol
mountPath: /opt/ibm/app/connectors
{{- else }}
{{- $files := .Files.Glob "connectors/*.{json,yaml,yml,car}" }}
{{- range $path, $file := $files }}
- name: {{ base $path | replace "." "-" | lower }}-vol
mountPath: /opt/ibm/app/connectors/{{ base $path }}
subPath: {{ base $path }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Render environment variables for node-runtime-metrics
*/}}
{{- define "ibm-connectivity-pack.nodeRuntimeMetrics" }}
{{- if hasKey .Values.action "enableMetrics" }}
- name: ENABLE_METRICS
value: {{ ternary "true" "false" (eq .Values.action.enableMetrics "true") | quote }}
{{- end }}
{{- if hasKey .Values.action "metricsRotationWindowSec" }}
name: METRICS_ROTATION_WINDOW_SEC
value: {{ .Values.action.metricsRotationWindowSec | quote }}
{{- end }}
{{- if hasKey .Values.action "metricsDirectory" }}
name: METRICS_DIRECTORY
value: {{ .Values.action.metricsDirectory | quote }}
{{- end }}
{{- if hasKey .Values.action "metricsCollectIntervalMs" }}
- name: METRICS_COLLECT_INTERVAL_MS
value: {{ .Values.action.metricsCollectIntervalMs | quote }}
{{- end }}
{{- end }}
17 changes: 17 additions & 0 deletions ibm-connectivity-pack/templates/connectors.template.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{- $root := . }}
{{- $files := $root.Files.Glob "connectors/*.{json,yaml,yml,car}" }}
{{- range $path, $file := $files }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ base $path | replace "." "-" | lower }}-config
namespace: {{ include "ibm-connectivity-pack.namespace" $root }}
labels:
{{- include "ibm-connectivity-pack.labels" $root | nindent 4 }}
annotations:
{{- toYaml $root.Values.annotations | nindent 4 }}
data:
{{ base $path }}: |
{{ $file | toString | nindent 4 }}
---
{{- end }}
6 changes: 4 additions & 2 deletions ibm-connectivity-pack/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ spec:
optional: true
name: {{ include "ibm-connectivity-pack.envConfig" . }}
env:
{{- include "ibm-connectivity-pack.nodeRuntimeMetrics" . | nindent 12 }}
- name: CONNECTOR_SERVICE_ROUTE
value: {{ include "ibm-connectivity-pack.actionServiceRoute" .}}
- name: ENABLE_DYNAMIC_CREDENTIALS_MODE
Expand Down Expand Up @@ -134,11 +135,10 @@ spec:
valueFrom:
fieldRef:
fieldPath: status.podIP

securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
ports:
- containerPort: 3001
protocol: TCP
- containerPort: 3020
protocol: TCP
imagePullPolicy: IfNotPresent
Expand All @@ -158,6 +158,7 @@ spec:
readOnly: true
mountPath: "/opt/ibm/app/creds"
{{ end }}
{{- include "ibm-connectivity-pack.udaConnectorsMount" . | nindent 12 }}
image: {{ .Values.image.registry }}/{{ .Values.image.path }}/{{ .Values.action.image }}@{{ .Values.action.digest }}
{{ if .Values.event.enable -}}
- name: {{ .Values.event.name }}
Expand Down Expand Up @@ -394,6 +395,7 @@ spec:
secretName: {{ include "ibm-connectivity-pack.basicAuthCreds" . }}
optional: true
{{- include "ibm-connectivity-pack.stunnelvolume" . | nindent 8 }}
{{- include "ibm-connectivity-pack.udaConnectors" . | nindent 8 }}
strategy:
type: RollingUpdate
rollingUpdate:
Expand Down
3 changes: 2 additions & 1 deletion ibm-connectivity-pack/templates/role.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ rules:
- update
- delete
- create
- patch
apiGroups:
- ''
resources:
Expand Down Expand Up @@ -80,4 +81,4 @@ rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "list", "update", "patch", "delete"]
{{ end }}
{{ end }}
2 changes: 1 addition & 1 deletion ibm-connectivity-pack/templates/uninstallPostHook.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "ibm-connectivity-pack.preHookJob" . }}
name: {{ include "ibm-connectivity-pack.postDeleteHookJob" . }}
namespace: {{ include "ibm-connectivity-pack.namespace" . }}
labels:
{{- include "ibm-connectivity-pack.labels" . | nindent 4 }}
Expand Down
26 changes: 15 additions & 11 deletions ibm-connectivity-pack/values.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
###############################################################################
#
# © Copyright IBM Corp. 2025
# © Copyright IBM Corp. 2024
#
###############################################################################
# Default values for IBM Connectivity pack.
Expand All @@ -15,7 +15,10 @@ bunyan: "'{\"loglevel\" : \"info\", \"logsrc\" : true, \"logstdouttext\" : {}}'"
environmentVariables: {} # Yaml object ENV to be added on action and event pods
enableRefreshTokenMode: false
annotations:
productName: IBM Connectivity pack
productID: "0c44732a8b794b2d9e9b842b2dfc8361"
productName: "IBM Connectivity Pack - Free"
productMetric: "FREE"
productChargedContainers: "All"
image:
registry: cp.icr.io # Image registry base URL
path: cp/ibm-eventstreams
Expand Down Expand Up @@ -44,19 +47,20 @@ basicAuth:
username: 'csuser'
preHook:
image: connectivity-pack-prehook
digest: sha256:c544bc6336c92e6002042fc8fdd84d2b5e2d09ca6e8cd53d9a4575ea3c2bf14f
tag: 2.0.0
digest: sha256:8d4c21d45c8030cff35cc2e65c36567b20805691946ebde5442a8b0d5bb35929
tag: 3.0.0
proxy:
name: connectivity-pack-proxy
image: connectivity-pack-proxy
digest: sha256:887bb7629a16f5246e8777bc4cf2707347d4c537461f0bd4aa523cb84145df6d
tag: 2.0.0
digest: sha256:708c72ca56391e6d336bc549546788aee66985a25c39880dd11ebd136b1c9ccd
tag: 3.0.0
action:
name: action-connectors
fileStreamingTimeout: '120000'
udaPersistentVolumeClaimName: ''
maxFileSize: '262144000'
image: action-connectors
digest: sha256:800b56a821200174ac82e12d5b2649434c1fdadfc1d2061d02be99fe3b4cedd7
digest: sha256:cbe955a55b1652b408c7622699dbf657cf6137dcd1eca97e9bb70ffb424c6695
resources: # Update below value for vertical scaling of container
limits:
cpu: '1'
Expand All @@ -66,12 +70,12 @@ action:
cpu: 250m
ephemeral-storage: 5Gi
memory: 256Mi
tag: 2.0.0
tag: 3.0.0
event:
enable: true # Set to true if Event container is needed
name: event-connectors
image: event-connectors # Update image
digest: sha256:2368806303cbc30420bd0ba63274219a7d8f4c359dfe378f38624c6d15db26bc
digest: sha256:2b116da33ce5202035374d1443e4af66e04223f174e033fbaf555df6aa6224aa
resources: # Update below value for vertical scaling of container
limits:
cpu: '1'
Expand All @@ -81,7 +85,7 @@ event:
cpu: 250m
ephemeral-storage: 5Gi
memory: 400Mi
tag: 2.0.0
tag: 3.0.0
javaservice:
enable: false
autoScaling:
Expand All @@ -101,4 +105,4 @@ securityContext:
seccompProfile:
type: RuntimeDefault
csCommon:
eventList: "['salesforceevent','jira','github','googlecalendar','googlebigquery']"
eventList: "['salesforceevent','jira','github','googlecalendar','googlebigquery','servicenow']"
8 changes: 4 additions & 4 deletions systems/source systems/github.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@ spec:
# By default, the format is '${object}-${eventType}', but it's shown here for clarity.
connectivitypack.topic.name.format: '${object}-${eventType}'

# `tasksMax` must be equal to the number of object-eventType combinations
# In this example it is 1
tasksMax: 1

# Specifies the converter class used to deserialize the message value.
# Change this to a different converter (for example, AvroConverter) as applicable.
value.converter: org.apache.kafka.connect.json.JsonConverter
Expand All @@ -135,8 +139,4 @@ spec:

# Optional, set the topic for producing the heartbeat events.
connectivitypack.source.heartbeat.topic: heartbeat-${topic}

# `tasksMax` must be equal to the number of object-eventType combinations
# In this example it is 1
tasksMax: 1
```
2 changes: 0 additions & 2 deletions systems/source systems/salesforce.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,6 @@ Subscription parameters configure how the Salesforce Polling connector polls for

| **Parameter**| **Description**| **KafkaConnector configuration** |
|--------------|----------------|------------------------------------|
| `timeFormat` | The date-time format to use when polling. For example, `YYYY-MM-DDTHH:mm:ssZ`. | `connectivitypack.source.<object>.<event>.subscription.timeFormat` |
| `timeZone` | The time zone for polling operations. Default value is UTC. For the complete list of supported time zone values, see [supported time zone values](../connectors/supported-timezones.md).| `connectivitypack.source.<object>.<event>.subscription.timezone`|
| `createdField` | The field that contains the object’s creation timestamp. | `connectivitypack.source.<object>.<event>.subscription.createdField` |
| `updatedField` | The field that contains the object’s last modified timestamp. | `connectivitypack.source.<object>.<event>.subscription.updatedField` |
Expand Down Expand Up @@ -146,7 +145,6 @@ spec:
connectivitypack.source.<object3>.CREATEDORUPDATED_POLLER.subscription.pollingInterval: 1
connectivitypack.source.<object3>.CREATEDORUPDATED_POLLER.subscription.timezone: UTC
connectivitypack.source.<object3>.CREATEDORUPDATED_POLLER.subscription.updatedField: LastModifiedDate
connectivitypack.source.<object3>.CREATEDORUPDATED_POLLER.subscription.timeFormat: 'YYYY-MM-DDTHH:mm:ssZ'
connectivitypack.source.<object3>.CREATEDORUPDATED_POLLER.subscription.createdField: CreatedDate
# Optional, sets the format for Kafka topic names created by the connector.
# You can use placeholders such as '${object}' and '${eventType}', which the connector will replace automatically.
Expand Down
Loading