Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 58 additions & 8 deletions docs/tutorial/cleanup.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,32 +4,83 @@
This is a part of the [Charmed Apache Kafka Tutorial](index.md).

(remove-kafka-and-juju)=
## Remove Charmed Apache Kafka and Juju

If you're done using Charmed Apache Kafka and Juju and would like to free up resources on your machine, you can safely remove both.
## Remove tutorial

```{caution}
Removing Charmed Apache Kafka as shown below will delete all the data in the Apache Kafka. Further, when you remove Juju as shown below you lose access to any other applications you have hosted on Juju.
Removing a Juju model may result in data loss for all applications in this model.
```

To remove Charmed Apache Kafka and the model it is hosted on run the command:
To remove Charmed Apache Kafka and the `tutorial` model it is hosted on,
along with all other applications:

```shell
juju destroy-model tutorial --destroy-storage --force
```

Next step is to remove the Juju controller. You can see all of the available controllers by entering `juju controllers`. To remove the controller enter:
This will remove all applications in the `tutorial` model (Charmed Apache Kafka,
OpenSearch, PostgreSQL).
Your Juju controller and other models (if any) will remain intact for future use.

(remove-juju)=
## (Optional) Remove Juju and LXD

If you don't need Juju anymore and want to free up additional resources on your machine,
you can remove the Juju controller and Juju itself.

```{caution}
When you remove Juju as shown below,
you lose access to any other applications you have hosted on Juju.
```

### Remove the Juju controller

Check the list of controllers:

```shell
juju controllers
```

Remove the Juju controller created in this tutorial:

```shell
juju destroy-controller overlord
```

Finally to remove Juju altogether, enter:
### Remove Juju

To remove Juju altogether:

```shell
sudo snap remove juju --purge
```

### Clean up LXD

If you also want to remove LXD containers and free up all resources:

List all remaining LXD containers:

```shell
lxc list
```

Delete unnecessary containers:

```shell
lxc delete <container-name> --force
```

If you want to uninstall LXD completely:

```shell
sudo snap remove lxd --purge
```

```{warning}
Only remove LXD if you're not using it for other purposes.
LXD may be managing other containers or VMs on your system.
```

## What's next?

In this tutorial, we've successfully deployed Apache Kafka, added/removed replicas, added/removed users to/from the cluster, and even enabled and disabled TLS.
Expand All @@ -42,4 +93,3 @@ If you're looking for what to do next you can:
- [Report](https://github.com/canonical/kafka-operator/issues) any problems you encountered.
- [Give us your feedback](https://matrix.to/#/#charmhub-data-platform:ubuntu.com).
- [Contribute to the code base](https://github.com/canonical/kafka-operator)

168 changes: 106 additions & 62 deletions docs/tutorial/deploy.md

Large diffs are not rendered by default.

129 changes: 95 additions & 34 deletions docs/tutorial/enable-encryption.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@

This is a part of the [Charmed Apache Kafka Tutorial](index.md).

## Transport Layer Security (TLS)

[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) is used to encrypt data exchanged between two applications; it secures data transmitted over the network. Typically, enabling TLS within a highly available database, and between a highly available database and client/server applications, requires domain-specific knowledge and a high level of expertise. Fortunately, the domain-specific knowledge has been encoded into Charmed Apache Kafka. This means (re-)configuring TLS on Charmed Apache Kafka is readily available and requires minimal effort on your end.

Juju relations are particularly useful for enabling TLS.
Expand All @@ -18,90 +16,153 @@ In this tutorial, we will distribute [self-signed certificates](https://en.wikip
This setup is only for testing and demonstrating purposes and self-signed certificates are not recommended in a production cluster. For more information about which charm may better suit your use-case, please see the [Security with X.509 certificates](https://charmhub.io/topics/security-with-x-509-certificates) page.
```

### Configure TLS
## Configure TLS

Before enabling TLS on Charmed Apache Kafka we must first deploy the `self-signed-certificates` charm:

```shell
juju deploy self-signed-certificates --config ca-common-name="Tutorial CA"
```

Wait for the charm to settle into an `active/idle` state, as shown by the `juju status`:
Wait for the charm to settle into an `active`/`idle` state, as shown by the `juju status` command.

<details> <summary> Output example</summary>

```shell
Model Controller Cloud/Region Version SLA Timestamp
tutorial overlord localhost/localhost 3.6.8 unsupported 23:27:35Z
Model Controller Cloud/Region Version SLA Timestamp
tutorial overlord localhost/localhost 3.6.13 unsupported 17:56:56Z

App Version Status Scale Charm Channel Rev Exposed Message
data-integrator blocked 1 data-integrator latest/stable 180 no Please relate the data-integrator with the desired product
kafka 4.0.0 active 3 kafka 4/edge 245 no
kraft 4.0.0 active 3 kafka 4/edge 245 no
self-signed-certificates active 1 self-signed-certificates 1/stable 317 no

Unit Workload Agent Machine Public address Ports Message
data-integrator/0* blocked idle 6 10.109.154.254 Please relate the data-integrator with the desired product
kafka/0* active idle 0 10.109.154.47 19093/tcp
kafka/1 active idle 1 10.109.154.171 19093/tcp
kafka/2 active idle 2 10.109.154.82 19093/tcp
kraft/0* active idle 3 10.109.154.49 9098/tcp
kraft/1 active idle 4 10.109.154.148 9098/tcp
kraft/2 active idle 5 10.109.154.50 9098/tcp
self-signed-certificates/0* active idle 8 10.109.154.248

Machine State Address Inst id Base AZ Message
0 started 10.109.154.47 juju-030538-0 ubuntu@24.04 dev Running
1 started 10.109.154.171 juju-030538-1 ubuntu@24.04 dev Running
2 started 10.109.154.82 juju-030538-2 ubuntu@24.04 dev Running
3 started 10.109.154.49 juju-030538-3 ubuntu@24.04 dev Running
4 started 10.109.154.148 juju-030538-4 ubuntu@24.04 dev Running
5 started 10.109.154.50 juju-030538-5 ubuntu@24.04 dev Running
6 started 10.109.154.254 juju-030538-6 ubuntu@24.04 dev Running
8 started 10.109.154.248 juju-030538-8 ubuntu@24.04 dev Running
```

App Version Status Scale Charm Channel Rev Exposed Message
self-signed-certificates active 1 self-signed-certificates 1/edge 336 no
</details>

Unit Workload Agent Machine Public address Ports Message
self-signed-certificates/0* active idle 7 10.233.204.134
To enable TLS on Charmed Apache Kafka, integrate with `self-signed-certificates` charm:

Machine State Address Inst id Base AZ Message
7 started 10.233.204.134 juju-07a730-7 ubuntu@24.04 Running
```shell
juju integrate kafka:certificates self-signed-certificates
```

To enable TLS on Charmed Apache Kafka, integrate with `self-signed-certificates` charm:
After the charms settle into `active`/`idle` states, the Apache Kafka listeners
should now have been swapped to the default encrypted port `9093`.
This can be tested by testing whether the ports are open/closed with `telnet`:

```shell
juju integrate kafka:certificates self-signed-certificates
telnet <Public IP address> 9092
telnet <Public IP address> 9093
```

where `Public IP address` is the IP of any Charmed Apache Kafka application units.

Both commands will be **unable to connect** now, as our Apache Kafka cluster
has no active listeners due to absence of integrated applications.

```{caution}
When no other application is integrated to Charmed Apache Kafka,
the cluster is secured-by-default and external listeners (bound to port `9092`) are disabled,
thus preventing any external incoming connection.
```

After the charms settle into `active/idle` states, the Apache Kafka listeners should now have been swapped to the
default encrypted port 9093. This can be tested by testing whether the ports are open/closed with `telnet`:
Let's integrate the `data-integrator` application to the Apache Kafka cluster:

```shell
telnet <IP> 9092
telnet <IP> 9093
juju integrate data-integrator kafka
```

### Enable TLS encrypted connection
After all units are back to `active`/`idle`, you will see the new ports in the `juju status` output.
Now try connecting with `telnet` again:

Once TLS is configured on the cluster side, client applications should be configured as well to connect to
the correct port and trust the self-signed CA provided by the `self-signed-certificates` charm.
```shell
telnet <Public IP address> 9092
telnet <Public IP address> 9093
```

The `9092` port connection now should show a connection error,
while the `9093` port should establish a connection.

Make sure that the `kafka-test-app` is not connected to the Charmed Apache Kafka, by removing the relation if it exists:
## Enable TLS encrypted connection

Once TLS is configured on the cluster side, client applications should be configured as well
to connect to the correct port and trust the self-signed CA provided by
the `self-signed-certificates` charm.

Let's deploy our [Apache Kafka Test App](https://charmhub.io/kafka-test-app) again:

```shell
juju remove-relation kafka-test-app kafka
juju deploy kafka-test-app --channel edge
```

Then, enable encryption on the `kafka-test-app` by relating with the `self-signed-certificates` charm:
Then, enable encryption on the `kafka-test-app` by integrating with
the `self-signed-certificates` charm:

```shell
juju integrate kafka-test-app self-signed-certificates
```

We can then set up the `kafka-test-app` to produce messages with the usual configuration (note that there is no difference
here with the unencrypted workflow):
We can then set up the `kafka-test-app` to produce messages with the usual configuration
(note that that the process here is the same as with the unencrypted workflow):

```shell
juju config kafka-test-app topic_name=HOT-TOPIC role=producer num_messages=25
juju config kafka-test-app topic_name=HOT-TOPIC role=producer num_messages=20
```

Then relate with the `kafka` cluster:
Finally, relate with the `kafka` cluster:

```shell
juju integrate kafka kafka-test-app
```

As before, you can check that the messages are pushed into the Charmed Apache Kafka cluster by inspecting the logs:
Wait for `active`/`idle` status in `juju status` and check that the messages are pushed into
the Charmed Apache Kafka cluster by inspecting the logs:

```shell
juju exec --application kafka-test-app "tail /tmp/*.log"
```

Note that if the `kafka-test-app` was running before, there may be multiple logs related to the different
runs. Refer to the latest logs produced and also check that in the logs the connection is indeed established
with the encrypted port `9093`.
Refer to the latest logs produced and also check that in the logs the connection
is indeed established with the encrypted port `9093`.

### Remove external TLS certificate
## Remove external TLS certificate

To remove the external TLS and return to the locally generated one, remove relation with certificates provider:
To remove the external TLS and return to the locally generated one,
remove relation with certificates provider:

```shell
juju remove-relation kafka self-signed-certificates
```

The Charmed Apache Kafka application is not using TLS anymore for client connections.

## Clean up

Before proceeding further, let's remove the `kafka-test-app` application:

```shell
juju remove-relation kafka-test-app kafka
juju remove-relation kafka-test-app self-signed-certificates
juju remove-application kafka-test-app --destroy-storage
```
Loading