You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+93-19Lines changed: 93 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,10 +12,11 @@ This repo builds out and runs requests against a NATS SuperCluster running in AW
12
12
-`/eks-setup`: OpenTofu and Kubernetes configuration files specific to the AWS environment running in `us-east-1`
13
13
-`/k8s-configs`: the Kubernetes configuration files used to deploy services into both environments
14
14
-`/services`: the Python code for the various services that are used for the demo
15
+
-`/jetstream`: OpenTofu configurations for managing Streams and Consumers in JetStream
15
16
16
17
## Some FYI
17
18
18
-
This is project meant for demo purposes only. While it employs authenticationa and authorization for all of the exposed endpoints that it creates, the authentication credentials are stored in this repo and are therefore insecure. Deploy this repo only in ad hoc environments that you can stand up and tear down without impacting your production environments. **Do not run this project in its current stat in production.**
19
+
This is project meant for demo purposes only. While it employs authenticationa and authorization for all of the exposed endpoints that it creates, the authentication credentials are stored in this repo and are therefore insecure. Deploy this repo only in ad hoc environments that you can stand up and tear down without impacting your production environments. **Do not run this project in its current state in production.**
19
20
20
21
## Setting up the environments
21
22
@@ -29,7 +30,7 @@ Both the Azure and AWS environments are set up using a Bash script, which can be
29
30
30
31
To set up the Azure environment, `cd` into the `/aks-setup` folder and run:
31
32
32
-
```
33
+
```sh
33
34
sh aks-setup.sh
34
35
```
35
36
@@ -42,7 +43,7 @@ This will:
42
43
43
44
To set up the AWS environment, `cd` into the `/eks-setup` folder and run:
44
45
45
-
```
46
+
```sh
46
47
sh eks-setup.sh
47
48
```
48
49
@@ -58,23 +59,13 @@ This will:
58
59
59
60
When AWS adds a new context to your default kubeconfig, it uses the AWS CLI to authenticate each request made by kubectl. If you want to use a Kubernetes UI like [Lens](https://k8slens.dev/) to interact with your clusters, you'll need to create a separate kubeconfig file. The script and K8s config file necessary to do that have been provided in `/eks-setup/kubeconfig-setup`. Assuming you have the EKS cluster set as your current context, run the following from the `/eks-setup` directory:
60
61
61
-
```
62
+
```sh
62
63
kubectl apply -f kubeconfig-setup/admin-sa.yaml
63
64
sh kubeconfig-setup/create-sa-token.sh
64
65
```
65
66
66
67
That will create a kubeconfig file tied to a ServiceAccount that has cluster-admin priviliges. You can then load that into Lens as a new kubeconfig file.
67
68
68
-
## Deploying the Services
69
-
70
-
When configured against either context, you can deploy the services in bulk by running:
71
-
72
-
```
73
-
kubectl apply -f k8s-configs
74
-
```
75
-
76
-
This will run all four mathmatical services as Deployments with 3 instances, as well as the requester Job. Note that the requester Job will deploy and possibly run before the other services are ready, so you may need to run it again to see intended results.
77
-
78
69
## NATS Gateways and Cluster Connectivity
79
70
80
71
In a production environment, you'd likely already have a domain name associated with your K8s cluster ingress that you can map to the service used by the NATS servers.
@@ -87,15 +78,15 @@ Again, in production, you likely wouldn't have to do this.
87
78
88
79
The quickest way to get this information is to run the following against the AWS context:
89
80
90
-
```
81
+
```sh
91
82
kubectl get svc nats-east -o json | jq .status.loadBalancer.ingress
92
83
```
93
84
94
85
That will print out a hostname that you can use as the endpoint for your gateway in AWS.
95
86
96
87
For Azure:
97
88
98
-
```
89
+
```sh
99
90
kubectl get svc nats-east -o json | jq .status.loadBalancer.ingress
100
91
```
101
92
@@ -105,6 +96,88 @@ If you don't have `jq` installed, you can find it here: https://jqlang.org/
105
96
106
97
Otherwise you can just run `kubectl get svc nats-east -o json` and sift through the output.
107
98
99
+
## TLS Connections
100
+
101
+
This project deviates from the [previous one](https://github.com/colinjlacy/nats-cluster-demo) by [using mTLS](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls) to connect services to the NATS resources running in each Kubernetes cluster.
102
+
103
+
**Please keep in mind that this approach is for demo purposes, and is not necessarily recommended for production use. Check with your team and business to see what security and compliance requirements exist for using TLS certs within your organization.**
104
+
105
+
To get started, deploy cert-manager into each Kubernetes cluster. You can find installation instructions [on the cert-manager website](https://cert-manager.io/docs/installation/helm/).
106
+
107
+
Once you have that installed, you can start creating certs to match your deployment stack. Each Kubernetes cluster requires its own cert heirarchy, since they'll each use TLS for connecting internally. In a production setting, you would use a common Certificate Authority (CA) to create certs within each cluster, so that they could all reference the same heirarchy. For now, and for this demo, cluster-specific CA's will do.
108
+
109
+
In each setup folder, you'll see a YAML file for setting up certs - `nats-tls-east.yaml` in `eks-setup/` and `nats-tls-west.yaml` in `aks-setup/`. Each one is *almost* ready to use. However they each need the public endpoint of the load balancer used to expose NATS to the public internet. This is important because without this, you won't be able to create a NATS context in the CLI, nor connect locally using OpenTofu to create JetStream resources.
110
+
111
+
In `eks-setup/nats-tls-east.yaml`, populate line 55 with the hostname of the NLB that was used to expose NATS.
112
+
113
+
In `aks-setup/nats-tls-west.yaml`, populate line 56 with the IP of the Azure Load Balancer that was used to expose NATS.
114
+
115
+
Now apply each one to their respective clusters. It should create a series of Secret resources in each cluster. Note that all of the service Deployment and Job YAML files have been updated to reference these Secret resources and pull in their values. You don't have to do anything there.
116
+
117
+
## JetStream Streams and Consumers
118
+
119
+
With TLS certs all set up, you can start to set up JetStream. First, you'll need to pull down the values in the `nats-admin-tls` Secret, and store each in the respective cluster's folder. So, for example, for EKS you would pull each entry from `secrets/nats-admin-tls` - `tls.ca`, `tls.crt`, and `tls.key` - and store each one in a file bearing that name in `jetstream/eks/`. The environment folders have a `.gitkeep` to make sure they are there when you pull down this repo. Once your done, there should be three files in each folder, e.g.:
120
+
121
+
-`jetstream/eks/tls.ca`
122
+
-`jetstream/eks/tls.cert`
123
+
-`jetstream/eks/tls.key`
124
+
125
+
**You'll also need to populate the different `tfvars` files with the public endpoint of the NATS server, provided by the cloud load balancer - the NLB hostname for EKS, and the IP for Azure.** A placeholder and comment has been added to the top of each `tfvars` file, and needs to be replaced.
126
+
127
+
With those in place you should be able to run the OpenTofu configs, for example, against EKS:
128
+
129
+
```sh
130
+
tofu init --var-file=eks.tfvars
131
+
tofu plan --out plan --var-file eks.tfvars
132
+
tofu apply plan
133
+
```
134
+
135
+
## Deploying the Services
136
+
137
+
When configured against either context, you can deploy the services in bulk by running:
138
+
139
+
```sh
140
+
kubectl apply -f k8s-configs
141
+
```
142
+
143
+
This will run all four mathmatical services as Deployments with 3 instances, as well as the requester Job. Note that the requester Job will deploy and possibly run before the other services are ready, so you may need to run it again to see intended results:
The NATS CLI requires a context to connect to when it needs to communicate with a remote cluster. In order to set it up, we can use the same TLS credentials that we used in OpenTofu. We'll use the absolute paths for the TLS files, so that when you run the NATS CLI against this context, you can run it from anywhere in your file structure. The following is for creating a context against the Azure load balancer:
You'll want to replace the file paths with your own file paths, and replace the `<azure-lb-ip>` placeholder with the actual IP of the Azure LB. You would then do the same for EKS, pointing to the AWS NLB hostname for the `-s` value, and using the file paths for the EKS TLS files.
158
+
159
+
## Running the Recorder in Kubernetes
160
+
161
+
The Recorder service has its own deployment YAML that will work against a properly set up K8s cluster. However you'll have to make sure that there's a MySQL database accessible from he Kubernetes cluster in order to properly run it.
162
+
163
+
For that I used the [Bitnami MySQL Helm Chart](https://artifacthub.io/packages/helm/bitnami/mysql).
164
+
165
+
Whatever solution you decide to go with, be sure to update the ConfigMap in `k8s-configs/recorder.yaml`.
166
+
167
+
## Running the Recorder Locally
168
+
169
+
This is a bit more involved. You'll need to have a MySQL database stood up locally for the Recorder to connect to. I used a simple local installation of MySQL Server, which allowed for unauthenticated connectivity. Whatever solution you choose, be sure to update the default values in `services/recorder/main.py`.
170
+
171
+
**Remember to create the schema and tables that will be used for storing data!**
172
+
173
+
You'll also need to pull down the TLS credentials that the Recorder service uses, which are stored in `secrets/nats-recorder-tls` in your Kubernetes cluster. Those are expected to be stored in their respective files alongside the `main.py` in the `services/recorder` folder, so:
174
+
175
+
- services/recorder/tls.ca
176
+
- services/recorder/tls.cert
177
+
- services/recorder/tls.key
178
+
179
+
With those all set up, you can run the Recorder
180
+
108
181
## Environment teardown
109
182
110
183
The Azure environment can be entirely torn down by `cd`ing into the `aks-setup/tofu-aks` folder and running:
0 commit comments