pull & push with datasource CloudWatch breaks dashboards

[iptizer]

We have more going on, but I am restricting this PR to the essence. What I am describing here works perfectly for the datasource Prometheus. Also copy & paste from "json model" via WebUI seems to work fine. It seems that some options (default options or so) are not synced correctly.

Steps to reproduce:

1. Create a new dashboard and add a metric that exists in CloudWatch.
2. save & copy and paste "json model" to a file. before.json.txt
3. Pull the dashboard to a json using grafana-sync, see README. Nothing special.
4. Push the dashboard from the json using grafana-sync, see README. Nothing special.
5. THE DASHBOARD IS BROKEN
6. copy and paste "json model" to a file. after.json.txt

before.json.txt
after.json.txt

"Broken" is defined as: The panel just shows blank. See

Also the Annotations seems to be duplicated.

Also some things have been removed when diffing.

This can be "fixed" as follows:

• Click on edit on panel.
• Now "refresh dashboard".
• Grafana seems to be doing something, as now, values start to appear.

Using Grafana v9.1.0. Deployed as part of prometheus-operator on k8s.

This could be an issue with Grafan itself, but could also be some "default value not sync"-optimization or so.

Would be great to get some insights, let me know where I can help.

[mpostument]

Did you have the same issue on grafana 8? Not sure if grafana SDK is supporting grafana 9. Maybe this is the issue

[iptizer]

I did a downgrade to Grafana 8.3.11 (changed image and restart deployment). This shows the same problems.

From the code I see no specific things done. So maybe this is an issue with the CloudWatch datasource. What do you think would be a good approach to tackle this?

[iptizer]

I did some further debugging. I made some findings that I would like to explain here.

The json pulled by grafana-sync and when clicking JSON Modelin the Grafana UI differ. The JSON in the WebUI seems to be longer (xx lines vs 1379 lines). One guess could be, that default values are ignored or sth.
I am wrangling the jsons with the following command cat ${i}_tmp | jq -S . > ${i} to get a diff-able json. This shows e.g. the following diff:

❯ git diff dashboards/status.json | cat
diff --git a/dashboards/status.json b/dashboards/status.json
index c2aaa5b..5772933 100644
--- a/dashboards/status.json
+++ b/dashboards/status.json
@@ -2,34 +2,31 @@
   "annotations": {
     "list": [
       {
+        "builtIn": 1,
         "datasource": {
           "type": "grafana",
           "uid": "-- Grafana --"
         },
         "enable": true,
-        "expr": "",
+        "hide": true,
         "iconColor": "rgba(0, 211, 255, 1)",
-        "iconSize": 0,
-        "lineColor": "",
         "name": "Annotations & Alerts",
-        "query": "",
-        "showLine": false,
-        "step": "",
-        "tagKeys": "",
-        "tags": null,
-        "tagsField": "",
-        "textField": "",
-        "textFormat": "",
-        "titleFormat": "",
+        "target": {
+          "limit": 100,
+          "matchAny": false,
+          "tags": [],
+          "type": "dashboard"
+        },
         "type": "dashboard"
       }
     ]
   },
   "editable": true,
+  "fiscalYearStartMonth": 0,
   "graphTooltip": 1,
-  "hideControls": false,
+  "id": 422,
   "links": [],
-  "originalTitle": "",
+  "liveNow": false,
   "panels": [
     {
       "collapsed": false,
@@ -43,10 +40,8 @@
       },
       "id": 23,
       "isNew": false,
-      "panels": null,
       "span": 0,
       "title": "Health",
-      "transparent": false,
       "type": "row"
     },
     {
@@ -62,6 +57,9 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
             "axisPlacement": "auto",
             "barAlignment": 0,
             "drawStyle": "line",
@@ -91,6 +89,7 @@
               "mode": "off"
             }
           },
+          "mappings": [],
           "thresholds": {
             "mode": "absolute",
             "steps": [
@@ -105,7 +104,8 @@
             ]
           },
           "unit": ""
-        }
+        },
+        "overrides": []
[.....]

When manually pulling the JSON via WebUI. Then pushing it with grafana-sync, it seems that again the same key-value pairs are lost.

assumption)

Can it be that some values are "forgotten" by the sdk used?

One way to check would be to compare grafana-sync result with a manual query via the rest API of Grafana.

[mpostument]

Yes, it may be that not all values are pushed/pull when doing it with sdk for cloud watch data source. I will try to take a look once i have a chance but not able to give any ETA right now