Merge pull request #37 from Helene/merge_7.0

Merge changes for 7.0.1 release

Author: Helene

.s2i/environment

@@ -0,0 +1 @@
+APP_FILE = ../source/zimonGrafanaIntf.py

docs/CHANGELOG.md

@@ -97,6 +97,9 @@ Define a new data source (Data Sources -> Add New)
 
 ![](/docs/Example_Add_DataSource.png)
 
-**NOTE**: The IBM Spectrum Scale bridge listens on port 4242 for HTTP connections, and on port 8443 for HTTPS(SSL) connections. Follow the instructions [Generate SSL certificates](https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/wiki/How-to-setup-HTTPS%28SSL%29-connection-for-the-IBM-Spectrum-Scale-bridge-for-Grafana#generate-ssl-certificates) to generate a private ssl key and a ssl certificate
+**NOTE**: Per default the IBM Spectrum Scale bridge listens on port 4242 for HTTP connections. For HTTPS(SSL) connections you need to set the appropriate protocol settings on the bridge start.
+          Follow the instructions [Generate SSL certificates](https://github.com/IBM/ibm-spectrum-scale-bridge-for-grafana/wiki/How-to-setup-HTTPS%28SSL%29-connection-for-the-IBM-Spectrum-Scale-bridge-for-Grafana#generate-ssl-certificates) to generate a private ssl key and a ssl certificate
+
+
 
 Grafana now can talk to Spectrum Scale Performance Monitoring tool via the bridge. Follow the grafana instructions to create dashboards.

docs/CLASSIC_SETUP.md

@@ -75,7 +75,7 @@ Now you can add the host running the bridge container to the Grafana monitoring
 2. Start the bridge running in a container:
 
 ```shell
-# podman run -dt -p 4242:4242,8443:8443 -e "SERVER=9.XXX.XXX.XXX" -e "APIKEYVALUE=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" -e "PORT=8443" -e "TLSKEYPATH=/etc/bridge_ssl/certs" -e "TLSKEYFILE=privkey.pem" -e "TLSCERTFILE=cert.pem" \ -v /tmp:/var/log/ibm_bridge_for_grafana -v /etc/bridge_ssl/certs:/etc/bridge_ssl/certs \ --pod new:my-bridge-ssl-test-pod --name bridge-ssl-test bridge_image:latest 
+# podman run -dt -p 4242:4242,8443:8443 -e "SERVER=9.XXX.XXX.XXX" -e "APIKEYVALUE=XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" -e "PORT=8443" -e "PROTOCOL=https" -e "TLSKEYPATH=/etc/bridge_ssl/certs" -e "TLSKEYFILE=privkey.pem" -e "TLSCERTFILE=cert.pem" \ -v /tmp:/var/log/ibm_bridge_for_grafana -v /etc/bridge_ssl/certs:/etc/bridge_ssl/certs \ --pod new:my-bridge-ssl-test-pod --name bridge-ssl-test bridge_image:latest
 
 # podman logs bridge-ssl-test
 2021-04-25 16:05 - INFO     -  *** IBM Spectrum Scale bridge for Grafana - Version: 7.0 ***

docs/RUNNING_AS_DOCKER_CONTAINER.md

@@ -21,10 +21,10 @@ Openssl will then ask you a series of questions. You can enter whatever values a
 Note: The file names for the key and the certificate should be ‘privkey.pem’ and ‘cert.pem’.
 
 
-3. Start the bridge listening on the socket port 8443. Don't forget to provide the location and the file name of the tls private key and the tls certificate:
+3. Start the bridge listening on the socket port 8443 using HTTPS protocol. Don't forget to provide the location and the file name of the tls private key and the tls certificate:
 
 ```shell
-# python3 zimonGrafanaIntf.py -p 8443 -t /etc/bridge_ssl/certs -k privkey.pem -m cert.pem
+# python3 zimonGrafanaIntf.py -p 8443 -r https -t /etc/bridge_ssl/certs -k privkey.pem -m cert.pem
 ```
 
 

docs/SETUP_SSL_CONNECTION.md

@@ -1,10 +1,31 @@
 The following matrix gives a quick overview of the supported software for the IBM Spectrum Scale bridge for Grafana packages by version number:
+# Version 7.0.1 (07/21/2021)
+- Python 3.6
+- CherryPy 18.6.0
+- IBM Spectrum Scale system must run 5.1.1 and above or
+- IBM Spectrum Scale Container Native Storage Access(CNSA) devices having minReleaseLevel 5.1.1.1
+- Grafana 7.5.0 and above
+
 # Version 7 (04/26/2021)
 - Python 3.6
 - CherryPy 18.6.0
 - IBM Spectrum Scale system must run 5.1.1 and above
 - Grafana 7.1.0 and above
 
+# Version 6.1.4 (07/21/2021)
+- Python 3.6
+- CherryPy 18.6.0
+- IBM Spectrum Scale system must run 5.1.0.3 and above or
+- IBM Spectrum Scale Container Native Storage Access(CNSA) devices having minReleaseLevel 5.1.0.1
+- Grafana 8.0.0 and above
+
+# Version 6.1.3 (05/21/2021)
+- Python 3.6
+- CherryPy 18.6.0
+- IBM Spectrum Scale system must run 5.1.0 and above or
+- IBM Spectrum Scale Container Native Storage Access(CNSA) devices having minReleaseLevel 5.1.0.1
+- Grafana 7.5.0 and above
+
 # Version 6.1.2 (03/20/2021)
 - Python 3.6
 - CherryPy 18.6.0

docs/SUPPORT_MATRIX.md

@@ -63,7 +63,7 @@ Check that the both .pem files are created in the /tmp directory
 
 ```
 [root@mycluster-inf ~]# cd /etc/bridge_ssl/certs
-[root@mycluster-inf tmp]# cat privkey.pem
+[root@mycluster-inf certs]# cat privkey.pem
 -----BEGIN PRIVATE KEY-----
 MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDZ7v0Oq9n9zx8T
 v/VPln5K74udrUVPA/4xz/+MsUIThlt+RLcaVuBx2jTbxDlSdPiQc/rSfZRtBTBt
@@ -92,7 +92,7 @@ ehAHRd4XxCH0p4/lAI+qQZxmkRfho1IXg/sAQoFXCF2dj+hMZ9nlV26m3n6iZxFY
 bjRUDSzBGHtV+WhPLzrKrsRFn4USCQHDA9+0aL2jGnfwC3En4SeMfk/3usLPVSAX
 B4MtCX8SrTMkQ7PutwU2/R+i
 -----END PRIVATE KEY-----
-[root@mycluster-inf tmp]# cat cert.pem
+[root@mycluster-inf certs]# cat cert.pem
 -----BEGIN CERTIFICATE-----
 MIIDNzCCAh+gAwIBAgIJAMlcr+FBcrAnMA0GCSqGSIb3DQEBCwUAMDIxFzAVBgNV
 BAMMDmdyYWZhbmEtYnJpZGdlMRcwFQYDVQQKDA5ncmFmYW5hLWJyaWRnZTAeFw0y
@@ -119,7 +119,7 @@ O+6FAarz5qVR/D09GhoohwQDT8m5N2hfS61JQ/f3iN3KUYK00ppwWzJeumabVY+Z
 5. Create the 'grafana-bridge-secret' secret for the TLS keys
 
 ```
-[root@mycluster-inf tmp]# kubectl create secret tls grafana-bridge-secret --key="privkey.pem" --cert="cert.pem"
+[root@mycluster-inf certs]# kubectl create secret tls grafana-bridge-secret --key="privkey.pem" --cert="cert.pem"
 secret/grafana-bridge-secret created
 
 [root@mycluster-inf tmp]# oc describe secret grafana-bridge-secret
@@ -138,10 +138,77 @@ tls.key:  1704 bytes
 ```
 
 
-6. Change to the directory example_deployment_scripts/bridge_deployment/ and apply the following .yaml files:
+6. Since the IBM Spectrum Scale version 5.1.1 any client querying the performance data from the IBM Spectrum Scale cluster needs the API key authentication. You need to create API key secret for the grafana-bridge application using key name 'scale_default'. 
+
+First check the secret 'ibm-spectrum-scale-perfmon-default-api-key' exists in your CNSA project
+
+```
+[root@mycluster-inf ~]# oc get secret ibm-spectrum-scale-perfmon-default-api-key
+NAME                                         TYPE     DATA   AGE
+ibm-spectrum-scale-perfmon-default-api-key   Opaque   1      16h
+
+```
+Copy the scale_default api key value from that secret, mounted as a file in core pods
+
+```
+[root@mycluster-inf ~]# oc get po -o wide
+NAME                                           READY   STATUS    RESTARTS   AGE   IP              NODE                               NOMINATED NODE   READINESS GATES
+ibm-spectrum-scale-core-8gqvm                  1/1     Running   0          19h   10.16.105.140   worker1.mycluster.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-core-h5m6n                  1/1     Running   0          19h   10.16.105.141   worker2.mycluster.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-core-mwstf                  1/1     Running   0          19h   10.16.97.21     worker0.mycluster.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-gui-0                       9/9     Running   0          19h   10.254.16.108   worker1.mycluster.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-operator-79cd7dfb68-q4k49   1/1     Running   0          19h   10.254.12.168   worker2.mycluster.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-pmcollector-0               2/2     Running   0          19h   10.254.4.84     worker0.mycluster.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-pmcollector-1               2/2     Running   0          19h   10.254.16.109   worker1.mycluster.os.fyre.ibm.com   <none>           <none>
+
+[root@mycluster-inf ~]# oc rsh ibm-spectrum-scale-core-8gqvm
+Defaulting container name to gpfs.
+Use 'oc describe pod/worker0 -n ibm-spectrum-scale' to see all of the containers in this pod.
+
+sh-4.4# cat /etc/perfmon-api-keys/scale_default
+7h7p860oF99YwIBKJag8TZPkvILQAx3p8F9E
+sh-4.4#
+```
+
+ As next encode the plaintext API key name and value using base_64:
 
 ```
-[root@mycluster-inf tmp]# cd /opt/example_deployment_scripts/bridge_deployment
+[root@mycluster-inf certs]# echo -n 'scale_default'| base64
+c2NhbGVfZGVmYXVsdA==
+[root@mycluster-inf certs]# echo -n '7h7p860oF99YwIBKJag8TZPkvILQAx3p8F9E'| base64
+YjBmYmMzNDAtNDNlOC00MDM5LTk4OTUtZTA3ZWIzNGQxMTUz
+```
+
+
+7. Change to the directory example_deployment_scripts/bridge_deployment/ and open the secret.yaml file in edit mode. 
+Replace the <YOUR_BASE64_ENCODED_API_KEY_NAME> and the <YOUR_BASE64_ENCODED_API_KEY_VALUE> with the values you generated with base_64.
+
+
+8. Create the 'scale-grafana-api' secret for the 'scale_grafana' API key authentication
+
+```
+[root@mycluster-inf certs]# oc create -f secret.yaml
+secret/scale-grafana-api created
+
+[root@mycluster-inf certs]# oc describe secret scale-grafana-api
+Name:         scale-grafana-api
+Namespace:    ibm-spectrum-scale-ns
+Labels:       <none>
+Annotations:  <none>
+
+Type:  Opaque
+
+Data
+====
+api-key-name:   13 bytes
+api-key-value:  36 bytes
+```
+
+
+9. Apply the following .yaml files:
+
+```
+[root@mycluster-inf certs]# cd /opt/example_deployment_scripts/bridge_deployment
 
 [root@mycluster-inf bridge_deployment]# oc create -f role.yaml
 role.rbac.authorization.k8s.io/grafana-bridge created
@@ -157,29 +224,27 @@ deployment.apps/grafana-bridge-deployment created
 ```
 
 
-7. Verify the grafana-bridge pods are up and running
+10. Verify the grafana-bridge pods are up and running
 
 ```
 [root@mycluster-inf bridge_deployment]# oc get po -o wide
-NAME                                           READY   STATUS    RESTARTS   AGE   IP              NODE                               NOMINATED NODE   READINESS GATES
-grafana-bridge-deployment-5bdfcb6956-cpmjl     1/1     Running   0          9s    10.254.4.92     worker0.mycluster.os.fyre.ibm.com   <none>           <none>
-grafana-bridge-deployment-5bdfcb6956-zcq59     1/1     Running   0          9s    10.254.16.118   worker1.mycluster.os.fyre.ibm.com   <none>           <none>
-ibm-spectrum-scale-core-8gqvm                  1/1     Running   0          20h   10.16.105.140   worker1.mycluster.os.fyre.ibm.com   <none>           <none>
-ibm-spectrum-scale-core-h5m6n                  1/1     Running   0          20h   10.16.105.141   worker2.mycluster.os.fyre.ibm.com   <none>           <none>
-ibm-spectrum-scale-core-mwstf                  1/1     Running   0          20h   10.16.97.21     worker0.mycluster.os.fyre.ibm.com   <none>           <none>
-ibm-spectrum-scale-gui-0                       9/9     Running   0          20h   10.254.16.108   worker1.mycluster.os.fyre.ibm.com   <none>           <none>
-ibm-spectrum-scale-operator-79cd7dfb68-q4k49   1/1     Running   0          20h   10.254.12.168   worker2.mycluster.os.fyre.ibm.com   <none>           <none>
-ibm-spectrum-scale-pmcollector-0               2/2     Running   0          20h   10.254.4.84     worker0.mycluster.os.fyre.ibm.com   <none>           <none>
-ibm-spectrum-scale-pmcollector-1               2/2     Running   0          20h   10.254.16.109   worker1.mycluster.os.fyre.ibm.com   <none>           <none>
-
-[root@mycluster-inf bridge_deployment]# oc logs grafana-bridge-deployment-5bdfcb6956-cpmjl
-Connection to the collector server established successfully
-Successfully retrieved MetaData
-Received sensors:
-
-CPU     DiskFree        GPFSNodeAPI     GPFSRPCS        GPFSVFSX        GPFSWaiters     Load    Memory  Netstat Network TopProc
-Initial cherryPy server engine start have been invoked. Python version: 3.6.8 (default, Dec  5 2019, 15:45:45)
+NAME                                           READY   STATUS    RESTARTS   AGE   IP             NODE                             NOMINATED NODE   READINESS GATES
+grafana-bridge-deployment-686f577585-jcr6r     1/1     Running   8          89m   10.254.20.19   worker0.helene.os.fyre.ibm.com   <none>           <none>
+grafana-bridge-deployment-686f577585-wdrcs     1/1     Running   8          89m   10.254.16.19   worker1.helene.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-core-bpdwp                  0/1     Running   0          14h   10.16.69.109   worker1.helene.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-core-dsnx7                  0/1     Running   0          14h   10.16.69.105   worker0.helene.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-core-rdf54                  0/1     Running   0          14h   10.16.69.110   worker2.helene.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-gui-0                       9/9     Running   0          14h   10.254.16.11   worker1.helene.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-operator-6877d5c9bb-6xtxj   1/1     Running   0          15h   10.254.16.9    worker1.helene.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-pmcollector-0               2/2     Running   9          8h    10.254.20.16   worker0.helene.os.fyre.ibm.com   <none>           <none>
+ibm-spectrum-scale-pmcollector-1               2/2     Running   0          8h    10.254.16.14   worker1.helene.os.fyre.ibm.com   <none>           <none>
+
+[root@mycluster-inf bridge_deployment]# oc logs grafana-bridge-deployment-686f577585-jcr6r
+2021-04-08 21:12 - INFO     -  *** IBM Spectrum Scale bridge for Grafana - Version: 7.0 ***
+2021-04-08 21:12 - INFO     - Successfully retrieved MetaData
+2021-04-08 21:12 - INFO     - Received sensors:CPU, DiskFree, GPFSNodeAPI, GPFSRPCS, GPFSVFSX, GPFSWaiters, Load, Memory, Netstat, Network, TopProc
+2021-04-08 21:12 - INFO     - Initial cherryPy server engine start have been invoked. Python version: 3.6.8 (default, Aug 18 2020, 08:33:21)
 [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)], cherryPy version: 18.6.0.
-server started
+2021-04-08 21:12 - INFO     - server started
 
 ```

docs/grafana_bridge_deployment_ocp.md

@@ -63,6 +63,20 @@ spec:
             value: "tls.crt"
           - name: PORT
             value: "8443"
+          - name: PROTOCOL
+            value: "https"
+          - name: SERVERPORT
+            value: "9981"
+          - name: APIKEYNAME
+            valueFrom:
+              secretKeyRef:
+                name: scale-grafana-api
+                key: api-key-name
+          - name: APIKEYVALUE
+            valueFrom:
+              secretKeyRef:
+                name: scale-grafana-api
+                key: api-key-value
           imagePullPolicy: Always
           ports:
             - name: http

example_deployment_scripts/bridge_deployment/bridge-deployment.yaml

@@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: scale-grafana-api
+type: Opaque
+data:
+  api-key-name: <YOUR_BASE64_ENCODED_API_KEY_NAME>
+  api-key-value: <YOUR_BASE64_ENCODED_API_KEY_VALUE>