Skip to content

Commit 8cf8713

Browse files
freeznetfreeznet
authored
feat: Add ClusterParamsOverride for geo-replication configuration (#319)
* feat: Add ClusterParamsOverride for geo-replication configuration - Introduced ClusterParamsOverride struct to allow customization of cluster parameters during geo-replication setup. - Updated PulsarGeoReplicationSpec to include ClusterParamsOverride. - Enhanced reconciliation logic to utilize ClusterParamsOverride for authentication and connection parameters. - Updated CRD and documentation to reflect new configuration options for geo-replication. * address file path option
1 parent bf0f293 commit 8cf8713

File tree

5 files changed

+311
-32
lines changed

5 files changed

+311
-32
lines changed

‎api/v1alpha1/pulsargeoreplication_types.go‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,12 @@ type PulsarGeoReplicationSpec struct {
3636
// +kubebuilder:validation:Enum=CleanUpAfterDeletion;KeepAfterDeletion
3737
// +optional
3838
LifecyclePolicy PulsarResourceLifeCyclePolicy `json:"lifecyclePolicy,omitempty"`
39+
40+
// ClusterParamsOverride allows overriding specific cluster parameters when setting up
41+
// geo-replication. This is useful when the destination cluster requires different
42+
// configuration than what's defined in the DestinationConnectionRef.
43+
// +optional
44+
ClusterParamsOverride *ClusterParamsOverride `json:"clusterParamsOverride,omitempty"`
3945
}
4046

4147
// PulsarGeoReplicationStatus defines the observed state of PulsarGeoReplication
@@ -89,3 +95,53 @@ type ClusterInfo struct {
8995
// ConnectionRef is the connection reference that can connect to the pulsar cluster
9096
ConnectionRef corev1.LocalObjectReference `json:"connectionRef"`
9197
}
98+
99+
// ClusterParamsOverride allows overriding specific parameters when creating/updating cluster info
100+
// for geo-replication. This provides flexibility to customize cluster configuration without
101+
// modifying the underlying PulsarConnection.
102+
type ClusterParamsOverride struct {
103+
// ServiceURL overrides the HTTP(S) URL for the Pulsar cluster's admin service
104+
// +optional
105+
ServiceURL *string `json:"serviceURL,omitempty"`
106+
107+
// ServiceSecureURL overrides the HTTPS URL for secure connections to the Pulsar admin service
108+
// +optional
109+
ServiceSecureURL *string `json:"serviceSecureURL,omitempty"`
110+
111+
// BrokerServiceURL overrides the non-TLS URL for connecting to Pulsar brokers
112+
// +optional
113+
BrokerServiceURL *string `json:"brokerServiceURL,omitempty"`
114+
115+
// BrokerServiceSecureURL overrides the TLS-enabled URL for secure connections to Pulsar brokers
116+
// +optional
117+
BrokerServiceSecureURL *string `json:"brokerServiceSecureURL,omitempty"`
118+
119+
// BrokerClientTrustCertsFilePath overrides the file path to the trusted TLS certificate
120+
// for outgoing connections to Pulsar brokers
121+
// +optional
122+
BrokerClientTrustCertsFilePath *string `json:"brokerClientTrustCertsFilePath,omitempty"`
123+
124+
// Authentication overrides the authentication configuration for the cluster.
125+
// When this field is set, the secret update check will be skipped for this geo-replication.
126+
// +optional
127+
Authentication *ClusterAuthOverride `json:"authentication,omitempty"`
128+
}
129+
130+
// ClusterAuthOverride allows overriding authentication parameters for cluster configuration.
131+
// This is useful when the geo-replication target requires different authentication than
132+
// the source connection.
133+
type ClusterAuthOverride struct {
134+
// AuthPlugin specifies the authentication plugin class name
135+
// Common values: "org.apache.pulsar.client.impl.auth.AuthenticationToken",
136+
// "org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2"
137+
// +optional
138+
AuthPlugin *string `json:"authPlugin,omitempty"`
139+
140+
// AuthParameters contains the authentication parameters as a string.
141+
// Format depends on the AuthPlugin:
142+
// - For Token: "token:your-token-here"
143+
// - For Token: "file://your-token-file-path-on-brokers"
144+
// - For OAuth2: JSON string with client credentials
145+
// +optional
146+
AuthParameters *string `json:"authParameters,omitempty"`
147+
}

‎api/v1alpha1/zz_generated.deepcopy.go‎

Lines changed: 76 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎config/crd/bases/resource.streamnative.io_pulsargeoreplications.yaml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,54 @@ spec:
5454
spec:
5555
description: PulsarGeoReplicationSpec defines the desired state of PulsarGeoReplication
5656
properties:
57+
clusterParamsOverride:
58+
description: |-
59+
ClusterParamsOverride allows overriding specific cluster parameters when setting up
60+
geo-replication. This is useful when the destination cluster requires different
61+
configuration than what's defined in the DestinationConnectionRef.
62+
properties:
63+
authentication:
64+
description: |-
65+
Authentication overrides the authentication configuration for the cluster.
66+
When this field is set, the secret update check will be skipped for this geo-replication.
67+
properties:
68+
authParameters:
69+
description: |-
70+
AuthParameters contains the authentication parameters as a string.
71+
Format depends on the AuthPlugin:
72+
- For Token: "token:your-token-here"
73+
- For Token: "file://your-token-file-path-on-brokers"
74+
- For OAuth2: JSON string with client credentials
75+
type: string
76+
authPlugin:
77+
description: |-
78+
AuthPlugin specifies the authentication plugin class name
79+
Common values: "org.apache.pulsar.client.impl.auth.AuthenticationToken",
80+
"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2"
81+
type: string
82+
type: object
83+
brokerClientTrustCertsFilePath:
84+
description: |-
85+
BrokerClientTrustCertsFilePath overrides the file path to the trusted TLS certificate
86+
for outgoing connections to Pulsar brokers
87+
type: string
88+
brokerServiceSecureURL:
89+
description: BrokerServiceSecureURL overrides the TLS-enabled
90+
URL for secure connections to Pulsar brokers
91+
type: string
92+
brokerServiceURL:
93+
description: BrokerServiceURL overrides the non-TLS URL for connecting
94+
to Pulsar brokers
95+
type: string
96+
serviceSecureURL:
97+
description: ServiceSecureURL overrides the HTTPS URL for secure
98+
connections to the Pulsar admin service
99+
type: string
100+
serviceURL:
101+
description: ServiceURL overrides the HTTP(S) URL for the Pulsar
102+
cluster's admin service
103+
type: string
104+
type: object
57105
connectionRef:
58106
description: ConnectionRef is the reference to the source PulsarConnection
59107
properties:

‎docs/pulsar_geo_replication.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ The `PulsarGeoReplication` resource has the following specifications:
2828
| `connectionRef` | Reference to the PulsarConnection resource used to connect to the source Pulsar cluster. | Yes |
2929
| `destinationConnectionRef` | Reference to the PulsarConnection resource used to connect to the destination Pulsar cluster. | Yes |
3030
| `lifecyclePolicy` | Determines whether to keep or delete the geo-replication configuration when the Kubernetes resource is deleted. Options: `CleanUpAfterDeletion`, `KeepAfterDeletion`. Default is `CleanUpAfterDeletion`. | No |
31+
| `clusterParamsOverride` | Allows overriding specific cluster parameters when setting up geo-replication. This is useful when the destination cluster requires different configuration than what's defined in the `destinationConnectionRef`. See [Cluster Parameters Override](#cluster-parameters-override) for details. | No |
3132

3233
The `PulsarGeoReplication` resource is designed to configure geo-replication between separate Pulsar instances. It creates a new "Cluster" in the destination Pulsar cluster identified by `destinationConnectionRef`. This setup allows configuring the replication of data from the source cluster (identified by `connectionRef`) to the destination cluster. By establishing this connection, the brokers in the source cluster can communicate with and replicate data to the brokers in the destination cluster, enabling geo-replication between the two separate Pulsar instances.
3334

@@ -49,6 +50,67 @@ Note: When configuring geo-replication between `connectionRef` and `destinationC
4950

5051
1. The brokers in the `connectionRef` cluster are able to communicate with the `destinationConnectionRef` cluster, and the `destinationConnectionRef` cluster is able to authenticate the connections from the `connectionRef` cluster.
5152

53+
### Cluster Parameters Override
54+
55+
The `clusterParamsOverride` field provides a powerful way to customize cluster configuration for geo-replication without modifying the underlying `PulsarConnection` resource. This is particularly useful when:
56+
57+
1. **Different authentication is required** for geo-replication compared to regular cluster operations
58+
2. **Alternative URLs need to be used** for inter-cluster communication
59+
3. **Specific TLS configurations** are needed for cross-cluster connections
60+
61+
#### Supported Override Parameters
62+
63+
The `clusterParamsOverride` supports the following fields:
64+
65+
- **URL Configuration**:
66+
- `serviceURL`: Override the HTTP(S) URL for the Pulsar cluster's admin service
67+
- `serviceSecureURL`: Override the HTTPS URL for secure admin connections
68+
- `brokerServiceURL`: Override the non-TLS URL for broker connections
69+
- `brokerServiceSecureURL`: Override the TLS-enabled URL for secure broker connections
70+
71+
- **TLS Configuration**:
72+
- `brokerClientTrustCertsFilePath`: Override the path to trusted TLS certificates
73+
74+
- **Authentication Configuration**:
75+
- `authentication.authPlugin`: Override the authentication plugin class name
76+
- `authentication.authParameters`: Override the authentication parameters
77+
78+
#### Authentication Override Benefits
79+
80+
When `authentication` is specified in the override, the system automatically:
81+
- **Skips secret validation checks** for the destination connection
82+
- **Avoids unnecessary Secret API calls** for improved performance
83+
- **Uses the override authentication directly** without processing the destinationConnectionRef authentication
84+
85+
#### Example Usage
86+
87+
```yaml
88+
apiVersion: resource.streamnative.io/v1alpha1
89+
kind: PulsarGeoReplication
90+
metadata:
91+
name: us-east-to-west-geo-replication
92+
namespace: us-east
93+
spec:
94+
connectionRef:
95+
name: us-east-local-connection
96+
destinationConnectionRef:
97+
name: us-east-to-west-connection
98+
clusterParamsOverride:
99+
# Override URLs for cross-cluster communication
100+
serviceURL: "https://geo-replication-admin.us-west.example.com:8443"
101+
brokerServiceURL: "pulsar://geo-replication-broker.us-west.example.com:6650"
102+
# Override authentication for geo-replication
103+
authentication:
104+
authPlugin: "org.apache.pulsar.client.impl.auth.AuthenticationToken"
105+
authParameters: "token:geo-replication-specific-token"
106+
```
107+
108+
**Important Notes**:
109+
- Override parameters take precedence over the corresponding fields in `destinationConnectionRef`
110+
- Only non-null override values will replace the destination connection values
111+
- Authentication override is particularly useful for scenarios requiring different credentials for geo-replication
112+
- The override does not affect how the operator connects to manage other resources in the destination cluster
113+
52114
### Lifecycle Policy
53115

54116
The `lifecyclePolicy` field determines what happens to the geo-replication configuration when the Kubernetes PulsarGeoReplication resource is deleted:

0 commit comments

Comments
 (0)