Merge pull request #51 from superstreamlabs/staging

Release v1.5.0

Author: valeraBr

README.md

@@ -226,8 +226,55 @@ helmfile -e default diff
 ``` bash
 helmfile -e default apply
 ``` 
+## Appendix C - Superstream Client Configuration
 
-## Appendix C - Uninstall
+**Client Connection**
+
+To establish a connection of a new client, the data plane's Fully Qualified Domain Name (FQDN) for the Superstream service must be accessible and exposed. Below are the connection procedures based on the client’s location relative to the Superstream service:
+
+1. For clients in environments like AWS EKS, you can expose the Superstream service using a LoadBalancer. Below is an example of the required service configuration in a YAML file (svc.yaml):
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: superstream-host-external
+  annotations:
+    service.beta.kubernetes.io/aws-load-balancer-type: external
+    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
+    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
+    service.beta.kubernetes.io/aws-load-balancer-name: superstream-host-external
+spec:
+  ports:
+  - name: superstream-host-external
+    port: 4222
+    protocol: TCP
+    targetPort: 4222
+  selector:
+    app.kubernetes.io/component: nats
+    app.kubernetes.io/instance: nats
+    app.kubernetes.io/name: nats
+  type: LoadBalancer
+```
+
+2. To deploy this configuration, use the following command, replacing <NAMESPACE> with the appropriate namespace:
+```bash
+kubectl apply -f svc.yaml -n <NAMESPACE>
+```
+
+3. Validate that the load balancer is created successfully and the FQDN is created:
+```bash
+$ kubectl get svc superstream-host-external -n <NAMESPACE>
+NAME                             TYPE           CLUSTER-IP      EXTERNAL-IP                                                                   PORT(S)                         AGE
+superstream-host-external   LoadBalancer   10.100.100.100   superstream-host-external.elb.us-east-1.amazonaws.com   4222:32074/TCP   1d
+```
+
+The exposed FQDN should be used together with the provided activation token with the following variables in the client configuration:
+```yaml
+SUPERSTREAM_HOST=<FQDN>
+SUPERSTREAM_TOKEN=<ACTIVATION_TOKEN>
+```
+
+## Appendix D - Uninstall
 
 **Steps to Uninstall Superstream Data Plane Deployment.**
 
@@ -244,58 +291,46 @@ It's crucial to delete the stateful storage linked to the data plane. Ensure you
 kubectl delete pvc -l app.kubernetes.io/instance=nats -n <NAMESPACE>
 ```
 
-## Appendix D - Custom changes to the helmfile
+## Appendix E - Custom changes
 
-**StorageClass definition for NATS service**
+**StorageClass definition**
 
-If there is no default storageClass configured for the Kubernetes cluster, it should be configured manually from the helmfile.yaml.
+If there is no default storageClass configured for the Kubernetes cluster or there is a need to choose a custom storageClass, it can be done by specifying its name in the `environments/default.yaml` file.
 
-1. Open helmfile.yaml with preferred editor and navigate to the nats configuration section:
+1. Open `environments/default.yaml` with a preferred editor:
 
 ```yaml
-releases:
-  - name: {{ .Values.natsReleaseName }}
-    installed: true
-    namespace: {{ .Values.namespace }}
-    chart: nats/nats
-    version: 1.1.7
-    values:
-      - container:
-          image: 
-            tag: alpine3.19
-          env: 
-            ACTIVATION_TOKEN:
-              valueFrom:
-                secretKeyRef:
-                  name: superstream-creds
-                  key: ACTIVATION_TOKEN 
-      - promExporter:
-          enabled: true
-      - natsBox:
-          enabled: false 
-      - config:
-            cluster: 
-              enabled: {{ .Values.haDeployment }}
-            jetstream: 
-              enabled: true
+helmVersion: 0.2.3 # Define the version of the superstream helm chart.
+namespace: superstream # Specify the Kubernetes namespace where the resources will be deployed, isolating them within this designated namespace.
+storageClassName: "" # Leave blank if you want to use default K8s cluster storageClass
+...
 ```
 
-2. Add the following section after `jetsream.enabled` line and mention the name of the desired storageClass:
+2. Fill the name of the desired storageClass name.
 
 ```yaml
-  jetstream:
-    enabled: true
-    fileStore:
-      pvc:
-        storageClassName: <THE_NAME>
+helmVersion: 0.2.3 # Define the version of the superstream helm chart.
+namespace: superstream # Specify the Kubernetes namespace where the resources will be deployed, isolating them within this designated namespace.
+storageClassName: "exampleSsdStorageClass" # Leave blank if you want to use default K8s cluster storageClass
+...
 ```
 
-3. Run the deployment
+3. Run the deployment.
 
 ```bash
 helmfile -e default apply
 ```
 
+4. Validate that the created PVCs are assigned to the desired storageClass.
+   
+```bash
+kubectl get pvc -n superstream
+NAME             STATUS   VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS             AGE
+nats-js-nats-0   Bound    pvc-ac65bfe7   10Gi       RWO            exampleSsdStorageClass   45h
+nats-js-nats-1   Bound    pvc-d3982397   10Gi       RWO            exampleSsdStorageClass   45h
+nats-js-nats-2   Bound    pvc-e85b69e0   10Gi       RWO            exampleSsdStorageClass   45h
+```
+
 ## Disable HPA - autoscalling ability of the Data Plane service
 
 If no autoscaling CRD is configured for the Kubernetes cluster, it should be configured manually from the helmfile.yaml.

environments/default.yaml

@@ -1,5 +1,6 @@
-helmVersion: 0.2.3 # Define the version of the superstream helm chart.
+helmVersion: 0.3.0 # Define the version of the superstream helm chart.
 namespace: superstream # Specify the Kubernetes namespace where the resources will be deployed, isolating them within this designated namespace.
+storageClassName: "" # Leave blank if you want to use default K8s cluster storageClass
 name: <DATAPLANE_NAME> # Define the data plane name within 32 characters, excluding '.', and using only letters, numbers, '-', and '_'.
 accountId: "" # Provide the account ID that is associated with the deployment, which could be used for identifying resources or configurations tied to a specific account.
 activationToken: "" # Enter the activation token required for services or resources that need an initial token for activation or authentication.

helmfile.yaml

@@ -12,6 +12,7 @@ environments:
     values:
     - ./environments/default.yaml
     - haDeployment: true
+    - skipLocalAuthentication: true
     - requestCpu: '2'
     - requestMemory: 2Gi
     - natsReleaseName: nats # Set the release name for the NATS deployment, which is used to uniquely identify the set of resources deployed for NATS.
@@ -50,15 +51,20 @@ releases:
               enabled: {{ .Values.haDeployment }}
             jetstream: 
               enabled: true
+              {{ if .Values.storageClassName }}
+              fileStore:
+                pvc:
+                  storageClassName: {{ .Values.storageClassName }}
+              {{ end }}
             merge: { 
                             accounts: {
                               SYS: {
-                                users: [{user: superstream_sys, password: << $ACTIVATION_TOKEN >>}]
+                                users: [{user: superstream_sys, password: {{ if .Values.skipLocalAuthentication }}"no-auth"{{ else }}"<< $ACTIVATION_TOKEN >>"{{ end }}}]
                                 # System account allows subscribing to $SYS.>
                               },
                               internal: {
                                 jetstream: enable,
-                                users: [{user: superstream_internal, password: << $ACTIVATION_TOKEN >>}]
+                                users: [{user: superstream_internal, password: {{ if .Values.skipLocalAuthentication }}"no-auth"{{ else }}"<< $ACTIVATION_TOKEN >>"{{ end }}}]
                                 # Regular user account for clients
                               }
                             },
@@ -87,6 +93,22 @@ releases:
             - pods
             verbs: ["get", "list", "watch"]          
       - config:
+          agent:
+            interval: "10s"
+            round_interval: true
+            metric_batch_size: 1000
+            metric_buffer_limit: 10000
+            collection_jitter: "0s"
+            flush_interval: "10s"
+            flush_jitter: "0s"
+            precision: ""
+            debug: false
+            quiet: false
+            logfile: "/tmp/telegraf.log"
+            logfile_rotation_max_size: "10MB"
+            logfile_rotation_max_archives: 5
+            hostname: "$HOSTNAME"
+            omit_hostname: false        
           outputs:
             - influxdb_v2:
                 urls:
@@ -96,7 +118,7 @@ releases:
                 content_encoding: "gzip"
                 http_headers:
                   Authorization: "{{ .Values.activationToken }}"
-                namepass: ["syslog"]
+                namepass: ["syslog","telegraf_logs"]
             - influxdb_v2:
                 urls:
                 - "https://superstream-monitoring.mgmt.memphis-gcp.dev"
@@ -105,7 +127,7 @@ releases:
                 content_encoding: "gzip"
                 http_headers:
                   Authorization: "{{ .Values.activationToken }}"
-                namedrop: ["syslog"] 
+                namedrop: ["syslog","telegraf_logs"] 
           processors:
             - enum:
                 mapping:
@@ -140,6 +162,19 @@ releases:
               tags:
                 accountId: "{{ .Values.accountId }}_{{ .Values.name }}"
                 chart: "telegraf-{{ .Values.telegrafHelmVersion }}"
+          - tail:
+              files:
+                - "/tmp/telegraf.log"
+              from_beginning: false  
+              data_format: "grok"
+              grok_patterns:
+                - "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}! %{GREEDYDATA:message}"
+              grok_custom_patterns:
+                LOGLEVEL [IWE] 
+              name_override: "telegraf_logs" 
+              tags:
+                accountId: "{{ .Values.accountId }}_{{ .Values.name }}"
+                appname: "telegraf"                               
 
   - name: {{ .Values.superstreamReleaseName }} 
     installed: true

version.conf

@@ -1 +1 @@
-1.4.0
+1.5.0