Skip to content

Commit 6e19d15

Browse files
k-wallk-wall
authored
Merge pull request #57 from k-wall/publish_060_docs
Publish v0.6.0 docs
2 parents d297b15 + 512de1c commit 6e19d15

File tree

18 files changed

+2082
-1
lines changed

18 files changed

+2082
-1
lines changed

‎_data/kroxylicious.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
versions:
22
- title: 'Development'
33
url: '/kroxylicious'
4+
- title: 'v0.6.0'
5+
url: '/docs/v0.6.0/'
46
- title: 'v0.5.1'
57
url: '/docs/v0.5.1/'
68
- title: 'v0.5.0'
@@ -10,4 +12,4 @@ versions:
1012
- title: 'v0.4.0'
1113
url: '/docs/v0.4.0/'
1214
- title: 'v0.3.0'
13-
url: '/docs/v0.3.0/'
15+
url: '/docs/v0.3.0/'

‎docs/v0.6.0/_files/available-filters.adoc‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
= Filters
2+
3+
== Built-in filters
4+
5+
The following filters are provided built-in as part of the distribution.
6+
7+
include::available-filters/record-encryption/record-encryption.adoc[leveloffset=2]
8+
include::available-filters/multi-tenancy/multi-tenancy.adoc[leveloffset=2]
9+
include::available-filters//schema-validation/schema-validation.adoc[leveloffset=2]
10+
include::available-filters/oauthbearer/oauthbearer.adoc[leveloffset=2]
11+
12+
== Community filters
13+
14+
Community contributed filters are showcased in the
15+
https://github.com/kroxylicious/kroxylicious-community-gallery[Community Gallery].

‎docs/v0.6.0/_files/available-filters/multi-tenancy/multi-tenancy.adoc‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
:github: https://github.com/kroxylicious/kroxylicious
2+
= Multi-tenancy
3+
4+
== What is it?
5+
6+
Multi-tenancy filter enables isolation prefixing each metadata transiting through virtual cluster to target cluster.
7+
See https://kafka.apache.org/documentation/#multitenancy[Apache Kafka multi-tenancy documentation] for more information.
8+
9+
== How to use the filter
10+
11+
There are two steps to using the filter.
12+
13+
1. <<Configuring virtual clusters>>
14+
2. Configuring the filter within Kroxylicious.
15+
16+
=== Configuring the filter within Kroxylicious.
17+
18+
[source, yaml]
19+
----
20+
filters:
21+
- type: MultiTenantTransformationFilterFactory
22+
config:
23+
prefixResourceNameSeparator: "." #<1>
24+
----
25+
<1> `-` is the default separator if no config is provided
26+
27+
If the virtual cluster name is `demo`, the created prefix will be `demo.`
28+
29+
NOTE: Currently, only the prefix with separator is validated.
30+
31+
=== Verifying that multi-tenancy is occurring
32+
33+
Set up two virtual clusters `devenv1` & `devenv2`. See this {github}/blob/main/kubernetes-examples/multi-tenant/base/kroxylicious/kroxylicious-config.yaml[example].
34+
35+
Create a topic on `devenv1`
36+
37+
[source]
38+
----
39+
kafka-topics.sh --bootstrap-server devenv1:9392 --create --if-not-exists --topic first-topic
40+
----
41+
42+
Create a topic on `devenv2`
43+
44+
[source]
45+
----
46+
kafka-topics.sh --bootstrap-server devenv2:9392 --create --if-not-exists --topic second-topic
47+
----
48+
49+
List topics from `devenv1`
50+
51+
[source]
52+
----
53+
kafka-topics.sh --bootstrap-server devenv1:9392 --list
54+
----
55+
56+
List topics from `devenv2`
57+
58+
[source]
59+
----
60+
kafka-topics.sh --bootstrap-server devenv2:9392 --list
61+
----
62+
63+
A full example with kubernetes is available {github}/blob/main/kubernetes-examples/multi-tenant/script.txt[here]

‎docs/v0.6.0/_files/available-filters/oauthbearer/oauthbearer.adoc‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
= OAUTHBEARER validation
2+
3+
== What is it?
4+
5+
OauthBearerValidation filter enables a validation on the JWT token received from client before forwarding it to cluster.
6+
7+
If the token is not validated, then the request is short-circuited.
8+
It reduces resource consumption on the cluster when a client sends too many invalid SASL requests.
9+
10+
[a2s, format="svg"]
11+
....
12+
.----------------------------------------------------------------------------------------.
13+
| |
14+
| '---------' '---------------' '----------' |
15+
| | | | | | | |
16+
| | Client | | Kroxylicious | | Cluster | |
17+
| | | | | | | |
18+
| '---------' '---------------' '----------' |
19+
| | | | |
20+
| | Handshake request | | |
21+
| |==============================>| Forward handshake request | |
22+
| | |===============================>| |
23+
| | | Handshake response | |
24+
| | Handshake response |<===============================| |
25+
| |<==============================| | |
26+
| | | | |
27+
| | | | |
28+
| | Authenticate request | | |
29+
| |==============================>| | |
30+
| | |==+ | |
31+
| | | : Validate Token | |
32+
| | |<=+ | |
33+
| | | | |
34+
| '=======|===============================|==' | |
35+
| :if | [Validation fails] | : | |
36+
| : | | : | |
37+
| : | Invalid token | : | |
38+
| : |<==============================| : | |
39+
| : | | : | |
40+
| '=======|===============================|==' | |
41+
| | | | |
42+
| '=======|===============================|================================|==' |
43+
| :if | [Validation succeeds] | | : |
44+
| : | | Forward authenticate request | : |
45+
| : | |===============================>| : |
46+
| : | | Authenticate response | : |
47+
| : | |<===============================| : |
48+
| : | Authenticate response | | : |
49+
| : |<==============================| | : |
50+
| : | | | : |
51+
| '=======|===============================|================================|==' |
52+
| | | | |
53+
| | | | |
54+
| '---------' '---------------' '----------' |
55+
| | | | | | | |
56+
| | Client | | Kroxylicious | | Cluster | |
57+
| | | | | | | |
58+
| '---------' '---------------' '----------' |
59+
| |
60+
.----------------------------------------------------------------------------------------.
61+
[0,0]: {"fill":"#99d","a2s:delref":1}
62+
....
63+
64+
== How to use the filter
65+
66+
There are two steps to using the filter.
67+
68+
1. <<Configuring virtual clusters>>
69+
2. Configuring the filter within Kroxylicious.
70+
71+
=== Configuring the filter within Kroxylicious.
72+
73+
[source, yaml]
74+
----
75+
filters:
76+
- type: OauthBearerValidation
77+
config:
78+
jwksEndpointUrl: https://oauth/JWKS #<1>
79+
jwksEndpointRefreshMs: 3600000 #<2>
80+
jwksEndpointRetryBackoffMs: 100 #<3>
81+
jwksEndpointRetryBackoffMaxMs: 10000 #<4>
82+
scopeClaimName: scope #<5>
83+
subClaimName: sub #<6>
84+
authenticateBackOffMaxMs: 60000 #<7>
85+
authenticateCacheMaxSize: 1000 #<8>
86+
expectedAudience: https://first.audience, https//second.audience #<9>
87+
expectedIssuer: https://your-domain.auth/ #<10>
88+
----
89+
90+
<1> The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved.
91+
<2> The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
92+
<3> The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider.
93+
<4> The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider.
94+
<5> This (optional) setting can provide a different name to use for the scope included in the JWT payload's claims.
95+
<6> This (optional) setting can provide a different name to use for the subject included in the JWT payload's claims.
96+
<7> The (optional) maximum value in milliseconds to limit the client sending authenticate request. Setting 0 will never limit the client. Otherwise, an exponential delay is added to each authenticate request until the authenticateBackOffMaxMs has been reached.
97+
<8> The (optional) maximum number of failed tokens kept in cache.
98+
<9> The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences.
99+
<10> The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer.
100+
101+
Note: OauthBearer config follows https://kafka.apache.org/documentation/#security_ssl[kafka's properties]

‎docs/v0.6.0/_files/available-filters/record-encryption/aws-kms/creating_keys.adoc‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
:aws: https://docs.aws.amazon.com/
2+
3+
= AWS Key Management Service
4+
5+
As the Administrator, use either the AWS Console or CLI to
6+
{aws}/kms/latest/developerguide/create-keys.html#create-symmetric-cmk[create] a *Symmetric key* with *Encrypt and decrypt*
7+
usage. Multi-region keys are supported. It is not possible to make use of keys from other AWS accountsfootnote:[https://github.com/kroxylicious/kroxylicious/issues/1217].
8+
9+
Give the key an alias following the link:setup.adoc#_establish_an_aliasing_convention_for_keys_within_aws_kms[alias naming convention] established above.
10+
11+
If using the CLI, this can be done with commands like this:
12+
13+
[source,shell]
14+
----
15+
KEY_ALIAS="KEK_<name>"
16+
KEY_ID=$(aws kms create-key | jq -r '.KeyMetadata.KeyId')
17+
# the create key command will produce JSON output including the KeyId
18+
aws kms create-alias --alias-name alias/${KEY_ALIAS} --target-key-id ${KEY_ID}
19+
----
20+
21+
Once the key is created, it is recommended to use a key rotation policy.
22+
23+
[source,shell]
24+
----
25+
aws kms enable-key-rotation --key-id ${KEY_ID} --rotation-period-in-days 180
26+
----
27+
28+

‎docs/v0.6.0/_files/available-filters/record-encryption/aws-kms/service.adoc‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
:aws: https://docs.aws.amazon.com/
2+
3+
= AWS Key Management Service
4+
5+
For AWS KMS, the KMS configuration looks like this.
6+
7+
[source, yaml]
8+
----
9+
kms: AwsKmsService # <1>
10+
kmsConfig:
11+
endpointUrl: https://kms.<region>.amazonaws.com # <2>
12+
tls: # <3>
13+
accessKey:
14+
passwordFile: /opt/aws/accessKey # <4>
15+
secretKey:
16+
passwordFile: /opt/aws/secretKey # <5>
17+
region: <region> # <6>
18+
----
19+
<1> Name of the KMS provider. This must be `AwsKmsService`.
20+
<2> AWS Endpoint URL. This must include the `https://` scheme part.
21+
<3> (Optional) TLS trust configuration.
22+
<4> File containing the AWS Access Key
23+
<5> File containing the AWS Secret Key
24+
<6> AWS region identifier (e.g. `us-east-1`)
25+
26+
For TLS trust and TLS client authentication configuration, the filter accepts the same TLS parameters as link:../../deploying.adoc#_upstream_tls[Upstream TLS]
27+
except the `PEM` store type is currently https://github.com/kroxylicious/kroxylicious/issues/933[not supported].

‎docs/v0.6.0/_files/available-filters/record-encryption/aws-kms/setup.adoc‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
:aws: https://docs.aws.amazon.com/
2+
3+
= AWS Key Management Service
4+
5+
The filter is able to integrate with {aws}/kms/latest/developerguide/overview.html[AWS Key Management Service].
6+
7+
Follow the instructions below to prepare your AWS account. You'll need a privileged AWS user
8+
that is capable of creating users and policies to perform the set-up.
9+
10+
[#_establish_an_aliasing_convention_for_keys_within_aws_kms]
11+
== Establish an aliasing convention for keys within AWS KMS
12+
13+
The filter references KEKs within AWS via an {aws}/kms/latest/developerguide/alias-about.html[AWS key alias].
14+
15+
It is necessary to establish a naming convention for the alias names. This will allow the keys used by the
16+
filter to be kept separate from any keys used by other systems. This document assumes that the naming convention
17+
will be to prefix the alias used by the filter with the word `KEK_`. If a different naming convention is used, adapt
18+
the instructions accordingly.
19+
20+
== Administrator Actor
21+
22+
To use the filter, there must be an administrative identity established within AWS IAM. This user, which is likely to be a human,
23+
has the responsibility to manage keys and aliases within AWS KMS for use by the filter.
24+
25+
Use AWS IAM to create a `kroxylicious-admin` user. Attach the policies `AWSKeyManagementServicePowerUser` and `IAMUserChangePassword`
26+
to the user. You may wish to attach policy `AWSCloudShellFullAccess` so the Administrator can use the AWS CloudShell to
27+
use CLI to manage the KEKs. Grant the user access to the Console.
28+
29+
If using the CLI, the following commands can be used to establish the Administrator user. This example illustrates using the
30+
user-name `kroxylicious-admin`. The choice of name is not significant. If a different user-name is used, adapt the
31+
instructions accordingly.
32+
33+
[source,shell]
34+
----
35+
ADMIN=kroxylicious-admin
36+
INITIAL_PASSWORD=$(tr -dc 'A-Za-z0-9!?%=' < /dev/urandom | head -c 10)
37+
CONSOLE_URL=https://$(aws sts get-caller-identity --query Account --output text).signin.aws.amazon.com/console
38+
aws iam create-user --user-name ${ADMIN}
39+
aws iam attach-user-policy --user-name ${ADMIN} --policy-arn arn:aws:iam::aws:policy/AWSKeyManagementServicePowerUser
40+
aws iam attach-user-policy --user-name ${ADMIN} --policy-arn arn:aws:iam::aws:policy/IAMUserChangePassword
41+
aws iam attach-user-policy --user-name ${ADMIN} --policy-arn arn:aws:iam::aws:policy/AWSCloudShellFullAccess
42+
aws iam create-login-profile --user-name ${ADMIN} --password ${INITIAL_PASSWORD} --password-reset-required
43+
echo Now log in at ${CONSOLE_URL} with user name ${ADMIN} password ${INITIAL_PASSWORD} and change the password.
44+
----
45+
46+
== Filter Actor
47+
48+
The Record Encryption Filter needs to be able to log in to AWS itself. For this there needs to be service account
49+
identity established within AWS IAM.
50+
51+
Use AWS IAM to create the `kroxylicious` user. Create an *Access Key* for this user. The Access Key/Secret Key pair
52+
will be used by the Filter. Do not enable the Console for this user.
53+
54+
If using the CLI, these commands can be used to establish the Filter Actor. This example illustrates using the user-name
55+
`kroxylicious`. The choice of name is not significant. If a different user-name is used, adapt the
56+
instructions accordingly.
57+
58+
[source,shell]
59+
----
60+
aws iam create-user --user-name kroxylicious
61+
aws iam create-access-key --user-name kroxylicious
62+
----
63+
64+
== Create Alias Based Policy
65+
66+
Create an alias based policy granting permissions to use keys aliased by the established alias naming convention.
67+
68+
[source,shell]
69+
----
70+
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
71+
cat > /tmp/policy << EOF
72+
{
73+
"Version": "2012-10-17",
74+
"Statement": [
75+
{
76+
"Sid": "AliasBasedIAMPolicy",
77+
"Effect": "Allow",
78+
"Action": [
79+
"kms:Encrypt",
80+
"kms:Decrypt",
81+
"kms:GenerateDataKey*",
82+
"kms:DescribeKey"
83+
],
84+
"Resource": [
85+
"arn:aws:kms:*:${AWS_ACCOUNT_ID}:key/*"
86+
],
87+
"Condition": {
88+
"ForAnyValue:StringLike": {
89+
"kms:ResourceAliases": "alias/KEK_*"
90+
}
91+
}
92+
}
93+
]
94+
}
95+
EOF
96+
aws iam create-policy --policy-name KroxyliciousRecordEncryption --policy-document file:///tmp/policy
97+
----
98+
99+
== Apply Alias Based Policy to Filter Actor
100+
101+
Attach the alias policy to the Filter Actor user. This will allow the Filter actor to invoke the
102+
necessary key operations on all KEKs with aliases of the prescribed form.
103+
104+
[source,shell]
105+
----
106+
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
107+
aws iam attach-user-policy --user-name kroxylicious --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KroxyliciousRecordEncryption"
108+
----

‎docs/v0.6.0/_files/available-filters/record-encryption/hashicorp-vault/creating_keys.adoc‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
2+
= HashiCorp Vault
3+
4+
Using the Administrator actor, use either the HashiCorp UI or CLI to create AES-256 symmetric keys following your
5+
key naming convention. The key type must be `aes256-gcm96`, which is Vault's default key type.
6+
7+
TIP: It is recommended to use a key rotation policy.
8+
9+
If using the Vault CLI, the command will look like:
10+
11+
[source, shell]
12+
----
13+
vault write -f transit/keys/KEK_trades type=aes256-gcm96 auto_rotate_period=90d
14+
----

0 commit comments

Comments
 (0)