Skip to content

Commit 4a0aa56

Browse files
tombentleytombentley
authored
Docs for addition of filterDefinitions, defaultFilters and deprecation of filters (kroxylicious#1716)
Signed-off-by: Tom Bentley <tbentley@redhat.com>
1 parent 8ae6a27 commit 4a0aa56

9 files changed

+279
-89
lines changed

‎docs/assemblies/assembly-configuring-proxy.adoc‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,16 @@
22
= Configuring proxies
33

44
[role="_abstract"]
5-
Fine-tune your deployment by configuring proxies to include additional features according to your specific requirements.
5+
Fine-tune your deployment by configuring proxies to include additional features according to your specific requirements.
66

7+
include::../modules/configuring/con-configuration-outline.adoc[leveloffset=+1]
8+
include::../modules/configuring/con-configuring-filters.adoc[leveloffset=+1]
79
include::../modules/configuring/con-configuring-virtual-clusters.adoc[leveloffset=+1]
8-
include::../modules/configuring/ref-configuring-proxy-example.adoc[leveloffset=+1]
9-
include::../modules/configuring/con-configuring-client-connections.adoc[leveloffset=+1]
10-
include::../modules/configuring/con-configuring-target-cluster-connections.adoc[leveloffset=+1]
11-
include::../modules/configuring/con-configuring-network-addresses.adoc[leveloffset=+1]
10+
include::../modules/configuring/con-configuring-vc-network-addresses.adoc[leveloffset=+1]
11+
include::../modules/configuring/con-configuring-vc-client-tls.adoc[leveloffset=+1]
12+
include::../modules/configuring/con-configuring-vc-target-tls.adoc[leveloffset=+1]
13+
14+
include::../modules/configuring/con-configuring-vc-other-settings.adoc[leveloffset=+1]
15+
include::../modules/configuring/con-configuring-toplevel-other-settings.adoc[leveloffset=+1]
16+
17+
include::../modules/configuring/ref-configuring-proxy-example.adoc[leveloffset=+1]

‎docs/modules/configuring/con-configuration-outline.adoc‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
[id='con-configuration-outline-{context}']
2+
= Outline of a Kroxylicious configuration
3+
4+
[role="_abstract"]
5+
The following example shows the overall outline of a simple Kroxylicious configuration.
6+
While not complete (as indicated by `# ...`), it illustrates the essential structure.
7+
8+
[id='con-basic-structure-{context}']
9+
.Basic outline of a Kroxylicious configuration
10+
[source,yaml]
11+
----
12+
filterDefinitions: # <1>
13+
- name: example # <2>
14+
type: org.example.filter.Example # <3>
15+
config: # <4>
16+
# ...
17+
defaultFilters: <5>
18+
- example
19+
virtualClusters: # <6>
20+
my-cluster-proxy:
21+
targetCluster: # <7>
22+
# ...
23+
clusterNetworkAddressConfigProvider: # <8>
24+
# ...
25+
# ...
26+
----
27+
<1> A list of named filter definitions.
28+
<2> A filter definition called `example`. The definitions must each have a unique name.
29+
<3> The name of the filter class implementation for the `example` filter. Required.
30+
<4> The configuration for the `example` filter instance. Usually required for non-trivial filters.
31+
<5> A list of default filters. It's possible to override this list at the virtual cluster level.
32+
<6> List of virtual clusters specified by name, with cluster-specific configurations.
33+
<7> Configuration of the actual Kafka cluster that is proxied by the 'my-cluster-proxy' virtual cluster.
34+
<8> Configuration of the networking model for this virtual cluster.
35+
36+

‎docs/modules/configuring/con-configuring-filters.adoc‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
[id='ref-configuring-filters-{context}']
2+
= Defining filters
3+
4+
Filters in Kroxylicious can be defined globally with `filterDefinitions`, applied by default using `defaultFilters`, or customized for specific virtual clusters.
5+
The following example shows how these elements work together flexibly:
6+
7+
[id='con-filterDefinitions-defaultFilters-{context}']
8+
.Example configuration showing global filter definitions applied as defaults and to virtual cluster
9+
[source,yaml]
10+
----
11+
filterDefinitions:
12+
- name: encryption
13+
type: RecordEncryption
14+
config:
15+
# ...
16+
- name: validation
17+
type: RecordValidation
18+
config:
19+
# ...
20+
- name: special-encryption
21+
type: RecordEncryption
22+
config:
23+
# ...
24+
defaultFilters:
25+
- validation
26+
- encryption
27+
virtualClusters:
28+
my-proxy-with-default-filters:
29+
# ...
30+
my-proxy-with-custom-filters:
31+
filters:
32+
- validation
33+
- special-encryption
34+
# ...
35+
# ...
36+
----
37+
38+
* The order of definitions in `filterDefinitions` does not matter.
39+
* Each filter definition in `filterDefinitions` must have a unique `name`, but you can have multiple definitions with the same type and different configurations (as with `encryption` and `special-encryption` in the example).
40+
* The order of `defaultFilters` determines the sequence in which the filters are applied to incoming client requests. In the example, records are first validated and then encrypted.
41+
* The `defaultFilters` are used for all virtual clusters which don't define their own `filters`, such as `my-proxy-with-default-filters`.
42+
* The `defaultFilters` property is optional. It is useful when all virtual clusters must use the same filters. There's no need to specify it if all virtual clusters have specific `filters` defined.
43+
* When a virtual cluster has defined `filters`, like `my-proxy-with-custom-filters`, then those filters are used instead of the `defaultFilters`.
44+
* When using `defaultFilters` or a virtual cluster's `filters` to reference a filter definition, you must define a filter with the corresponding name in `filterDefinitions`.
45+
46+
The top-level `filters` property allows defining _anonymous_ filters without a `name`.
47+
48+
WARNING: The `filters` property is deprecated and will be removed in a future version of Kroxylicious. Update existing configurations to use `filterDefinitions` instead.
49+
50+
[id='con-filters-{context}']
51+
.The deprecated `filters` property
52+
[source,yaml]
53+
----
54+
filters: # deprecated!
55+
- type: RecordEncryption
56+
config:
57+
# ...
58+
virtualClusters:
59+
my-proxy-with-default-filters:
60+
# ...
61+
----
62+
63+
The top-level `filters` property applies to all virtual clusters in the configuration.
64+
You cannot use both the top-level `filters` and `filterDefinitions` properties in the same configuration.
65+

‎docs/modules/configuring/con-configuring-toplevel-other-settings.adoc‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
[id='ref-configuring-toplevel-other-settings-{context}']
2+
= Configuring other top level settings
3+
4+
== Management HTTP endpoints
5+
6+
The proxy can run an HTTP server for exposing basic management information.
7+
This is configured with the top level `adminHttp` property.
8+
9+
[id='con-configuring-admin-http-{context}']
10+
.Configuration fragment showing the `adminHttp` property
11+
[source,yaml]
12+
----
13+
# ... (filterDefinitions, virtualClusters etc)
14+
adminHttp:
15+
host: 0.0.0.0 <1>
16+
port: 9190 <2>
17+
endpoints: # <3>
18+
prometheus: {} # <4>
19+
----
20+
<1> The address the HTTP server should bind to. Defaults to `0.0.0.0`.
21+
<2> The port the HTTP server should bind to.
22+
<3> Control over the exposed endpoints
23+
<4> If present and not null, exposes a Prometheus scrape endpoint at path `/metrics`.
24+

docs/modules/configuring/con-configuring-client-connections.adoc renamed to docs/modules/configuring/con-configuring-vc-client-tls.adoc

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,10 @@ NOTE: TLS is recommended on Kafka clients and virtual clusters for production co
2222
.Example applying a PKCS #12 server certificate to a virtual cluster
2323
[source,yaml]
2424
----
25+
# ...
2526
virtualClusters:
2627
my-cluster-proxy:
28+
# ...
2729
tls:
2830
key:
2931
storeFile: <path>/server.p12 # <1>
@@ -33,6 +35,7 @@ virtualClusters:
3335
passwordFile: <path>/key.password # <3>
3436
storeType: PKCS12 # <4>
3537
# ...
38+
# ...
3639
----
3740
<1> PKCS #12 store containing the private-key and certificate/intermediates of the virtual cluster.
3841
<2> Password to protect the PKCS #12 store.
@@ -42,15 +45,17 @@ virtualClusters:
4245
.Example applying a PEM server certificate and key pair to a virtual cluster
4346
[source,yaml]
4447
----
48+
# ...
4549
virtualClusters:
4650
my-cluster-proxy:
51+
# ...
4752
tls:
4853
key:
4954
privateKeyFile: <path>/server.key # <1>
5055
certificateFile: <path>/server.crt # <2>
5156
keyPassword:
5257
passwordFile: <path>/key.password # <3>
53-
# …
58+
# ...
5459
----
5560
<1> Private key of the virtual cluster.
5661
<2> Public certificate of the virtual cluster.
@@ -63,8 +68,10 @@ If verification fails, the client's connection is refused.
6368
.Example applying TLS client authentication using a PKCS #12 truststore
6469
[source,yaml]
6570
----
71+
# ...
6672
virtualClusters:
6773
demo:
74+
# ...
6875
tls:
6976
key:
7077
# ...
@@ -75,7 +82,7 @@ virtualClusters:
7582
storeType: PKCS12 # <3>
7683
trustOptions:
7784
clientAuth: REQUIRED # <4>
78-
# …
85+
# ...
7986
----
8087
<1> PKCS #12 store containing CA certificate(s) used to verify that the client's certificate is trusted.
8188
<2> (Optional) Password to protect the PKCS #12 store.

docs/modules/configuring/con-configuring-network-addresses.adoc renamed to docs/modules/configuring/con-configuring-vc-network-addresses.adoc

Lines changed: 54 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
[id='con-configuring-network-addresses-{context}']
2-
= Configuring network addresses
2+
= Configuring virtual cluster network addresses
33

44
[role="_abstract"]
5-
Virtual cluster configuration requires a network address configuration provider that manages network communication and provides broker address information to clients.
5+
Virtual cluster configuration requires a network address configuration provider that determines how the brokers in the Kafka cluster are presented to clients in terms of broker ids, hostnames, and ports.
66

77
Kroxylicious has the following built-in providers:
88

@@ -24,18 +24,25 @@ Ideally, the target cluster should have sequential, stable broker IDs and a know
2424

2525
The provider supports both cleartext and TLS downstream connections.
2626

27+
[id='con-networkAddressProvider-{context}']
2728
.Example broker address configuration
2829
[source,yaml]
2930
----
30-
clusterNetworkAddressConfigProvider:
31-
type: PortPerBrokerClusterNetworkAddressConfigProvider
32-
config:
33-
bootstrapAddress: mycluster.kafka.com:9192 # <1>
34-
brokerAddressPattern: mybroker-$(nodeId).mycluster.kafka.com # <2>
35-
brokerStartPort: 9193 # <3>
36-
numberOfBrokerPorts: 3 # <4>
37-
lowestTargetBrokerId: 1000 # <5>
38-
bindAddress: 192.168.0.1 # <6>
31+
# ...
32+
virtualClusters:
33+
my-cluster-proxy:
34+
# ...
35+
clusterNetworkAddressConfigProvider:
36+
type: PortPerBrokerClusterNetworkAddressConfigProvider
37+
config:
38+
bootstrapAddress: mycluster.kafka.com:9192 # <1>
39+
brokerAddressPattern: mybroker-$(nodeId).mycluster.kafka.com # <2>
40+
brokerStartPort: 9193 # <3>
41+
numberOfBrokerPorts: 3 # <4>
42+
lowestTargetBrokerId: 1000 # <5>
43+
bindAddress: 192.168.0.1 # <6>
44+
# ...
45+
# ...
3946
----
4047
<1> The hostname and port of the bootstrap address used by Kafka clients.
4148
<2> (Optional) The broker address pattern used to form broker addresses. If not defined, it defaults to the hostname part of the bootstrap address and the port number allocated to the broker.
@@ -78,19 +85,25 @@ This ensures a deterministic mapping of node IDs to ports while minimizing the n
7885
.Example node ID ranges configuration
7986
[source, yaml]
8087
----
81-
clusterNetworkAddressConfigProvider:
82-
type: RangeAwarePortPerNodeClusterNetworkAddressConfigProvider
83-
config:
84-
bootstrapAddress: mycluster.kafka.com:9192
85-
brokerAddressPattern: mybroker-$(nodeId).mycluster.kafka.com
86-
brokerStartPort: 9193
87-
nodeIdRanges: # <1>
88-
- name: brokers # <2>
89-
range:
90-
startInclusive: 0 # <3>
91-
endExclusive: 3 # <4>
88+
# ...
89+
virtualClusters:
90+
my-cluster-proxy:
91+
# ...
92+
clusterNetworkAddressConfigProvider:
93+
type: RangeAwarePortPerNodeClusterNetworkAddressConfigProvider
94+
config:
95+
bootstrapAddress: mycluster.kafka.com:9192
96+
brokerAddressPattern: mybroker-$(nodeId).mycluster.kafka.com
97+
brokerStartPort: 9193
98+
nodeIdRanges: # <1>
99+
- name: brokers # <2>
100+
range:
101+
startInclusive: 0 # <3>
102+
endExclusive: 3 # <4>
103+
# ...
104+
# ...
92105
----
93-
<1> The list of Node ID ranges, which must be non-empty.
106+
<1> The list of Node ID ranges, which must not be empty.
94107
<2> The name of the range, which must be unique within the `nodeIdRanges` list.
95108
<3> The start of the range (inclusive).
96109
<4> The end of the range (exclusive). It must be greater than `startInclusive`; empty ranges are not allowed.
@@ -113,6 +126,10 @@ This can be modeled as three node ID ranges, as shown in the following example.
113126
.Example node ID ranges configuration with KRaft roles
114127
[source, yaml]
115128
----
129+
# ...
130+
virtualClusters:
131+
my-cluster-proxy:
132+
# ...
116133
clusterNetworkAddressConfigProvider:
117134
type: RangeAwarePortPerNodeClusterNetworkAddressConfigProvider
118135
config:
@@ -130,6 +147,8 @@ This can be modeled as three node ID ranges, as shown in the following example.
130147
range:
131148
startInclusive: 99999
132149
endExclusive: 100000
150+
# ...
151+
# ...
133152
----
134153

135154
This configuration results in the following mapping from node ID to port:
@@ -151,12 +170,18 @@ The SNI routing provider uses SNI information to determine where to route the tr
151170
.Example SNI routing address provider configuration
152171
[source,yaml]
153172
----
154-
clusterNetworkAddressConfigProvider:
155-
type: SniRoutingClusterNetworkAddressConfigProvider
156-
config:
157-
bootstrapAddress: mycluster.kafka.com:9192 # <1>
158-
advertisedBrokerAddressPattern: mybroker-$(nodeId).mycluster.kafka.com
159-
bindAddress: 192.168.0.1
173+
# ...
174+
virtualClusters:
175+
my-cluster-proxy:
176+
# ...
177+
clusterNetworkAddressConfigProvider:
178+
type: SniRoutingClusterNetworkAddressConfigProvider
179+
config:
180+
bootstrapAddress: mycluster.kafka.com:9192 # <1>
181+
brokerAddressPattern: mybroker-$(nodeId).mycluster.kafka.com
182+
bindAddress: 192.168.0.1
183+
# ...
184+
# ...
160185
----
161186
<1> A single address for all traffic, including bootstrap address and brokers.
162187

‎docs/modules/configuring/con-configuring-vc-other-settings.adoc‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
[id='ref-configuring-vc-other-settings-{context}']
2+
= Configuring other Virtual Cluster settings
3+
4+
== Per-virtual cluster logging
5+
6+
You can enable low level logging on a per-virtual cluster basis.
7+
The `logNetwork` property controls logging of information about requests and responses at the network level, before they've been decoded into Kafka requests and responses.
8+
The `logFrames` property controls logging of the decoded requests and responses.
9+
10+
11+
[id='con-configuring-vc-logging-{context}']
12+
.Configuration fragment showing logging properties
13+
[source,yaml]
14+
----
15+
# ...
16+
virtualClusters:
17+
my-cluster-proxy:
18+
# ...
19+
logNetwork: true <1>
20+
logFrames: true <2>
21+
# ...
22+
----
23+
<1> Enables low-level network logging for the virtual cluster.
24+
<2> Enables low-level protocol frame logging for the virtual cluster.
25+

0 commit comments

Comments
 (0)