Skip to content

Commit 7c10519

Browse files
committed
Addressed comments and added instruction to make the ha-example more standalone
Signed-off-by: Thomas Cooper <code@tomcooper.dev>
1 parent 8202ee1 commit 7c10519

File tree

3 files changed

+36
-9
lines changed

3 files changed

+36
-9
lines changed

ha-example/README.md

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -87,22 +87,26 @@ spec:
8787
image: quay.io/streamshub/flink-sql-runner:0.2.0
8888
flinkVersion: v2_0
8989
flinkConfiguration:
90-
# job manager HA settings
91-
execution.checkpointing.interval: 1min
92-
state.checkpoints.dir: s3://test/cp
90+
# job manager HA settings
91+
execution.checkpointing.interval: 1min
92+
state.checkpoints.dir: s3://test/cp
9393
```
9494
The settings above will checkpoint the Task state every 1 minute under the s3 path provided.
9595

9696
## Example: Making the `recommendation-app` fault-tolerant and highly available
9797

9898
Here, we will use the [recommendation-app](../recommendation-app) as an example to demonstrate the job manager HA.
99-
99+
We are installing several additional components for this example so you may need a larger K8s cluster (more CPU and memory for you minikube deployment), than for the other examples.
100100

101101
1. If you haven't already, install cert-manager (this creates cert-manager in a namespace called `cert-manager`):
102102
```
103103
kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.17.2/cert-manager.yaml
104104
kubectl wait deployment --all --for=condition=Available=True --timeout=300s -n cert-manager
105105
```
106+
1. Create the `flink` namespace:
107+
```shell
108+
kubectl create namespace flink
109+
```
106110
1. Installing Flink Kubernetes Operator with leader election enabled like this:
107111
```shell
108112
helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator \
@@ -115,6 +119,17 @@ Here, we will use the [recommendation-app](../recommendation-app) as an example
115119
-n flink
116120
```
117121
1. Follow the [guide](minio-install/README.md) to deploy and create a local S3 compatible storage service using minio and add a bucket named `test`.
122+
1. Deploy Apicurio, Strimzi and Kafka as per the instructions in the [README](../README.md#installing-apache-kafka-apache-flink-and-apicurio-registry).
123+
1. Deploy the data generator application to generate data for the recommendation app:
124+
```shell
125+
kubectl -n flink apply -f ../recommendation-app/data-generator.yaml
126+
```
127+
This will create a deployment that generates data and sends it to the Kafka topic `flink.recommendation.products`.
128+
1. Create a ConfigMap that holds product inventory data in CSV format.
129+
```shell
130+
kubectl create configmap product-inventory --from-file ../recommendation-app/productInventory.csv -n flink
131+
```
132+
The ConfigMap will be volume mounted to the recommendation-app pods.
118133
1. Deploy the `FlinkDeployment` CR with HA configured
119134
```shell
120135
kubectl -n flink apply -f recommendation-app-HA/flink-deployment-ha.yaml

ha-example/minio-install/README.md

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,10 @@
33
This folder contains an example minio deployment yaml and the instructions are based on the [minio documentation](https://min.io/docs/minio/kubernetes/upstream/index.html).
44

55
1. Deploy the minio with default configurations:
6-
```
6+
```shell
77
kubectl -n flink apply -f minio.yaml
88
```
9+
This will create the minio deployment and service for the minio API endpoint (that Flink will use to store state).
910
1. Expose the minio WebUI:
1011
```shell
1112
kubectl -n flink expose deployment minio --name=minio-ui --type=NodePort --port=9090
@@ -22,10 +23,6 @@ This folder contains an example minio deployment yaml and the instructions are b
2223
1. Create a bucket via MinIO WebUI. To do this, open the address found above and login with username: `minioadmin` and password: `minioadmin`, then go to `Buckets` -> `Create Bucket` to create a bucket called `test`.
2324
1. Monitor the files in the bucket.
2425
Click on the `Object Browser` to view the files in the buckets.
25-
1. Expose the minio API:
26-
```shell
27-
kubectl -n flink expose deployment minio --name=minio-api --type=ClusterIP --port=9000
28-
```
2926

3027
After minio is deployed and bucket is created, the flink configuration can be set like this in the `FlinkDeployment` CR:
3128
```yaml

ha-example/minio-install/minio.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,18 @@ spec:
3030
hostPath: # MinIO generally recommends using locally-attached volumes
3131
path: /mnt/disk1/data # Specify a path to a local drive or volume on the Kubernetes worker node
3232
type: DirectoryOrCreate # The path to the last directory must exist
33+
---
34+
apiVersion: v1
35+
kind: Service
36+
metadata:
37+
labels:
38+
app: minio
39+
name: minio-api
40+
spec:
41+
type: ClusterIP
42+
selector:
43+
app: minio
44+
ports:
45+
- port: 9000
46+
protocol: TCP
47+
targetPort: 9000

0 commit comments

Comments
 (0)