Skip to content

Commit 096e2a8

Browse files
julienduchesnejulienduchesne
authored
Datasources: Support reading data from Grafana API (#609)
* Datasources: Support read I messed up in my last PR (#607) Turns out that in the Terraform logic, `TypeMap` means that it's a map[string]string, so the provider was writing strings as values The Grafana API accepted those string values but the plugins were essentially misconfigured. I noticed this while trying to implement the read functionality To fix that, I changed the `json_data_map` and `secure_json_data_map` to `json_data_encoded` and `secure_json_data_encoded` The flow for users is essentially the same but they have to `jsonencode(<map>)`. This allows Terraform to keep the correct attribute types Closes #253 Needs grafana/grafana-api-golang-client#108 * Update client
1 parent b7cc812 commit 096e2a8

File tree

7 files changed

+188
-46
lines changed

7 files changed

+188
-46
lines changed

‎docs/resources/data_source.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,16 +23,16 @@ resource "grafana_data_source" "arbitrary-data" {
2323
type = "stackdriver"
2424
name = "sd-arbitrary-data"
2525
26-
json_data_map = {
26+
json_data_encoded = jsonencode({
2727
"tokenUri" = "https://oauth2.googleapis.com/token"
2828
"authenticationType" = "jwt"
2929
"defaultProject" = "default-project"
3030
"clientEmail" = "[email protected]"
31-
}
31+
})
3232
33-
secure_json_data_map = {
33+
secure_json_data_encoded = jsonencode({
3434
"privateKey" = "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n"
35-
}
35+
})
3636
}
3737
3838
resource "grafana_data_source" "influxdb" {
@@ -106,11 +106,11 @@ resource "grafana_data_source" "stackdriver" {
106106
- `database_name` (String) (Required by some data source types) The name of the database to use on the selected data source server. Defaults to ``.
107107
- `http_headers` (Map of String, Sensitive) Custom HTTP headers
108108
- `is_default` (Boolean) Whether to set the data source as default. This should only be `true` to a single data source. Defaults to `false`.
109-
- `json_data` (Block List, Deprecated) (Required by some data source types). Deprecated: Use json_data_map instead. It supports arbitrary JSON data, and therefore all attributes. (see [below for nested schema](#nestedblock--json_data))
110-
- `json_data_map` (Map of String) Replaces the json_data attribute, this attribute can be used to pass configuration options to the data source. To figure out what options a datasource has available, see its docs or inspect the network data when saving it from the Grafana UI.
109+
- `json_data` (Block List, Deprecated) (Required by some data source types). Deprecated: Use json_data_encoded instead. It supports arbitrary JSON data, and therefore all attributes. (see [below for nested schema](#nestedblock--json_data))
110+
- `json_data_encoded` (String) Serialized JSON string containing the json data. Replaces the json_data attribute, this attribute can be used to pass configuration options to the data source. To figure out what options a datasource has available, see its docs or inspect the network data when saving it from the Grafana UI.
111111
- `password` (String, Sensitive, Deprecated) (Required by some data source types) The password to use to authenticate to the data source. Deprecated: Use secure_json_data.password instead. This attribute is removed in Grafana 9.0+. Defaults to ``.
112112
- `secure_json_data` (Block List, Deprecated) Deprecated: Use secure_json_data instead. It supports arbitrary JSON data, and therefore all attributes. (see [below for nested schema](#nestedblock--secure_json_data))
113-
- `secure_json_data_map` (Map of String, Sensitive) Replaces the secure_json_data attribute, this attribute can be used to pass secure configuration options to the data source. To figure out what options a datasource has available, see its docs or inspect the network data when saving it from the Grafana UI.
113+
- `secure_json_data_encoded` (String, Sensitive) Serialized JSON string containing the secure json data. Replaces the secure_json_data attribute, this attribute can be used to pass secure configuration options to the data source. To figure out what options a datasource has available, see its docs or inspect the network data when saving it from the Grafana UI.
114114
- `uid` (String) Unique identifier. If unset, this will be automatically generated.
115115
- `url` (String) The URL for the data source. The type of URL required varies depending on the chosen data source type.
116116
- `username` (String) (Required by some data source types) The username to use to authenticate to the data source. Defaults to ``.

‎examples/resources/grafana_data_source/resource.tf‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,16 @@ resource "grafana_data_source" "arbitrary-data" {
22
type = "stackdriver"
33
name = "sd-arbitrary-data"
44

5-
json_data_map = {
5+
json_data_encoded = jsonencode({
66
"tokenUri" = "https://oauth2.googleapis.com/token"
77
"authenticationType" = "jwt"
88
"defaultProject" = "default-project"
99
"clientEmail" = "[email protected]"
10-
}
10+
})
1111

12-
secure_json_data_map = {
12+
secure_json_data_encoded = jsonencode({
1313
"privateKey" = "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n"
14-
}
14+
})
1515
}
1616

1717
resource "grafana_data_source" "influxdb" {

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ go 1.18
55
require (
66
github.com/Masterminds/semver/v3 v3.1.1
77
github.com/grafana/amixr-api-go-client v0.0.5
8-
github.com/grafana/grafana-api-golang-client v0.11.0
8+
github.com/grafana/grafana-api-golang-client v0.11.1
99
github.com/grafana/machine-learning-go-client v0.1.1
1010
github.com/grafana/synthetic-monitoring-agent v0.9.3
1111
github.com/grafana/synthetic-monitoring-api-go-client v0.6.0

‎go.sum‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,8 +114,8 @@ github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
114114
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
115115
github.com/grafana/amixr-api-go-client v0.0.5 h1:jqmljnd5FozuOsCNuyhZVpooxmj0BW9MmeLA7PaLK6U=
116116
github.com/grafana/amixr-api-go-client v0.0.5/go.mod h1:N6x26XUrM5zGtK5zL5vNJnAn2JFMxLFPPLTw/6pDkFE=
117-
github.com/grafana/grafana-api-golang-client v0.11.0 h1:4KuqBJMTiuDcWMGa2MIGQOC1E0uzFJ7NcGHCBwN7ODs=
118-
github.com/grafana/grafana-api-golang-client v0.11.0/go.mod h1:24W29gPe9yl0/3A9X624TPkAOR8DpHno490cPwnkv8E=
117+
github.com/grafana/grafana-api-golang-client v0.11.1 h1:VmjdgKkxtt9Un6ScZ41PxOEETps3LI2pRD7u2cJrqYA=
118+
github.com/grafana/grafana-api-golang-client v0.11.1/go.mod h1:24W29gPe9yl0/3A9X624TPkAOR8DpHno490cPwnkv8E=
119119
github.com/grafana/machine-learning-go-client v0.1.1 h1:Gw6cX8xAd6IVF2LApkXOIdBK8Gzz07B3jQPukecw7fc=
120120
github.com/grafana/machine-learning-go-client v0.1.1/go.mod h1:QFfZz8NkqVF8++skjkKQXJEZfpCYd8S0yTWJUpsLLTA=
121121
github.com/grafana/synthetic-monitoring-agent v0.9.3 h1:1GAjyUMWPYndRF6tux8NWgk97bRFx9RVYMkEMerPNKU=

‎grafana/json.go‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
package grafana
2+
3+
import (
4+
"bytes"
5+
"encoding/json"
6+
"reflect"
7+
8+
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
9+
)
10+
11+
func SuppressEquivalentJSONDiffs(k, old, new string, d *schema.ResourceData) bool {
12+
ob := bytes.NewBufferString("")
13+
if err := json.Compact(ob, []byte(old)); err != nil {
14+
return false
15+
}
16+
17+
nb := bytes.NewBufferString("")
18+
if err := json.Compact(nb, []byte(new)); err != nil {
19+
return false
20+
}
21+
22+
return JSONBytesEqual(ob.Bytes(), nb.Bytes())
23+
}
24+
25+
func JSONBytesEqual(b1, b2 []byte) bool {
26+
var o1 interface{}
27+
if err := json.Unmarshal(b1, &o1); err != nil {
28+
return false
29+
}
30+
31+
var o2 interface{}
32+
if err := json.Unmarshal(b2, &o2); err != nil {
33+
return false
34+
}
35+
36+
return reflect.DeepEqual(o1, o2)
37+
}

‎grafana/resource_data_source.go‎

Lines changed: 76 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package grafana
22

33
import (
44
"context"
5+
"encoding/json"
56
"fmt"
67
"log"
78
"regexp"
@@ -11,6 +12,7 @@ import (
1112
"github.com/hashicorp/go-cty/cty"
1213
"github.com/hashicorp/terraform-plugin-sdk/v2/diag"
1314
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
15+
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/structure"
1416
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/validation"
1517

1618
gapi "github.com/grafana/grafana-api-golang-client"
@@ -37,6 +39,9 @@ source selected (via the 'type' argument).
3739
// Import either by ID or UID
3840
Importer: &schema.ResourceImporter{
3941
StateContext: func(c context.Context, rd *schema.ResourceData, meta interface{}) ([]*schema.ResourceData, error) {
42+
// Set this attribute on imports so that the condition in the read function is met
43+
// This means that when we're importing, we'll always read the JSON data from the API into this attribute
44+
rd.Set("json_data_encoded", "{}")
4045
_, err := strconv.ParseInt(rd.Id(), 10, 64)
4146
if err != nil {
4247
// If the ID is not a number, then it may be a UID
@@ -109,8 +114,8 @@ source selected (via the 'type' argument).
109114
"json_data": {
110115
Type: schema.TypeList,
111116
Optional: true,
112-
Description: "(Required by some data source types). Deprecated: Use json_data_map instead. It supports arbitrary JSON data, and therefore all attributes.",
113-
Deprecated: "Use json_data_map instead. It supports arbitrary JSON data, and therefore all attributes.",
117+
Description: "(Required by some data source types). Deprecated: Use json_data_encoded instead. It supports arbitrary JSON data, and therefore all attributes.",
118+
Deprecated: "Use json_data_encoded instead. It supports arbitrary JSON data, and therefore all attributes.",
114119
Elem: &schema.Resource{
115120
Schema: map[string]*schema.Schema{
116121
"assume_role_arn": {
@@ -557,18 +562,30 @@ source selected (via the 'type' argument).
557562
Default: "",
558563
Description: "(Required by some data source types) The username to use to authenticate to the data source.",
559564
},
560-
"json_data_map": {
561-
Type: schema.TypeMap,
565+
"json_data_encoded": {
566+
Type: schema.TypeString,
562567
Optional: true,
563-
ConflictsWith: []string{"json_data"},
564-
Description: "Replaces the json_data attribute, this attribute can be used to pass configuration options to the data source. To figure out what options a datasource has available, see its docs or inspect the network data when saving it from the Grafana UI.",
568+
ConflictsWith: []string{"json_data", "secure_json_data"},
569+
Description: "Serialized JSON string containing the json data. Replaces the json_data attribute, this attribute can be used to pass configuration options to the data source. To figure out what options a datasource has available, see its docs or inspect the network data when saving it from the Grafana UI.",
570+
ValidateFunc: validation.StringIsJSON,
571+
StateFunc: func(v interface{}) string {
572+
json, _ := structure.NormalizeJsonString(v)
573+
return json
574+
},
575+
DiffSuppressFunc: SuppressEquivalentJSONDiffs,
565576
},
566-
"secure_json_data_map": {
567-
Type: schema.TypeMap,
577+
"secure_json_data_encoded": {
578+
Type: schema.TypeString,
568579
Optional: true,
569580
Sensitive: true,
570-
ConflictsWith: []string{"secure_json_data"},
571-
Description: "Replaces the secure_json_data attribute, this attribute can be used to pass secure configuration options to the data source. To figure out what options a datasource has available, see its docs or inspect the network data when saving it from the Grafana UI.",
581+
ConflictsWith: []string{"json_data", "secure_json_data"},
582+
Description: "Serialized JSON string containing the secure json data. Replaces the secure_json_data attribute, this attribute can be used to pass secure configuration options to the data source. To figure out what options a datasource has available, see its docs or inspect the network data when saving it from the Grafana UI.",
583+
ValidateFunc: validation.StringIsJSON,
584+
StateFunc: func(v interface{}) string {
585+
json, _ := structure.NormalizeJsonString(v)
586+
return json
587+
},
588+
DiffSuppressFunc: SuppressEquivalentJSONDiffs,
572589
},
573590
},
574591
}
@@ -639,6 +656,43 @@ func ReadDataSource(ctx context.Context, d *schema.ResourceData, meta interface{
639656
d.Set("username", dataSource.User)
640657
d.Set("uid", dataSource.UID)
641658

659+
// If `json_data` is not set, then we'll use the new attribute: `json_data_encoded`. This allows support of imports.
660+
gottenJSONData, gottenSecureJSONData, gottenHeaders := gapi.ExtractHeadersFromJSONData(dataSource.JSONData, dataSource.SecureJSONData)
661+
if _, ok := d.GetOk("json_data_encoded"); ok {
662+
encodedJSONData, err := json.Marshal(gottenJSONData)
663+
if err != nil {
664+
return diag.Errorf("Failed to marshal JSON data: %s", err)
665+
}
666+
d.Set("json_data_encoded", string(encodedJSONData))
667+
}
668+
669+
// For headers and secure data, we do not know the value (the API does not return secret data)
670+
// so we only remove keys from the state that are no longer present in the API.
671+
if currentSecureJSONDataEncoded := d.Get("secure_json_data_encoded").(string); currentSecureJSONDataEncoded != "" {
672+
var currentSecureJSONData map[string]interface{}
673+
if err := json.Unmarshal([]byte(currentSecureJSONDataEncoded), &currentSecureJSONData); err != nil {
674+
return diag.Errorf("Failed to unmarshal current secure JSON data: %s", err)
675+
}
676+
for key := range currentSecureJSONData {
677+
if _, ok := gottenSecureJSONData[key]; !ok {
678+
delete(currentSecureJSONData, key)
679+
}
680+
}
681+
encodedSecureJSONData, err := json.Marshal(currentSecureJSONData)
682+
if err != nil {
683+
return diag.Errorf("Failed to marshal secure_json_data_encoded: %s", err)
684+
}
685+
d.Set("secure_json_data_encoded", string(encodedSecureJSONData))
686+
}
687+
688+
currentHeaders := d.Get("http_headers").(map[string]interface{})
689+
for key := range currentHeaders {
690+
if _, ok := gottenHeaders[key]; !ok {
691+
delete(currentHeaders, key)
692+
}
693+
}
694+
d.Set("http_headers", currentHeaders)
695+
642696
// TODO: these fields should be migrated to SecureJSONData.
643697
d.Set("basic_auth_enabled", dataSource.BasicAuth)
644698
d.Set("basic_auth_username", dataSource.BasicAuthUser) //nolint:staticcheck // deprecated
@@ -710,8 +764,12 @@ func makeDataSource(d *schema.ResourceData) (*gapi.DataSource, error) {
710764
}
711765

712766
func makeJSONData(d *schema.ResourceData) (map[string]interface{}, error) {
713-
if v, ok := d.GetOk("json_data_map"); ok {
714-
return v.(map[string]interface{}), nil
767+
if v, ok := d.GetOk("json_data_encoded"); ok {
768+
var jd map[string]interface{}
769+
if err := json.Unmarshal([]byte(v.(string)), &jd); err != nil {
770+
return nil, err
771+
}
772+
return jd, nil
715773
}
716774

717775
var derivedFields []gapi.LokiDerivedField
@@ -788,8 +846,12 @@ func makeJSONData(d *schema.ResourceData) (map[string]interface{}, error) {
788846
}
789847

790848
func makeSecureJSONData(d *schema.ResourceData) (map[string]interface{}, error) {
791-
if v, ok := d.GetOk("secure_json_data_map"); ok {
792-
return v.(map[string]interface{}), nil
849+
if v, ok := d.GetOk("secure_json_data_encoded"); ok {
850+
var sjd map[string]interface{}
851+
if err := json.Unmarshal([]byte(v.(string)), &sjd); err != nil {
852+
return nil, err
853+
}
854+
return sjd, nil
793855
}
794856

795857
return gapi.SecureJSONData{

‎grafana/resource_data_source_test.go‎

Lines changed: 61 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ func TestAccDataSource_basic(t *testing.T) {
2323
config string
2424
attrChecks map[string]string
2525
additionalChecks []resource.TestCheckFunc
26+
verifyImport bool // Only test import when `json_data_encoded` is set. Data sources with `json_data` cannot have their data imported.
2627
}{
2728
{
2829
resource: "grafana_data_source.loki",
@@ -212,25 +213,21 @@ func TestAccDataSource_basic(t *testing.T) {
212213
http_headers = {
213214
Authorization = "Token sdkfjsdjflkdsjflksjdklfjslkdfjdksljfldksjsflkj"
214215
}
215-
json_data_map = {
216+
json_data_encoded = jsonencode({
216217
defaultBucket = "telegraf"
217218
organization = "organization"
218219
tlsAuth = false
219220
tlsAuthWithCACert = false
220221
version = "Flux"
221-
}
222+
})
222223
}
223224
`,
224225
attrChecks: map[string]string{
225-
"type": "influxdb",
226-
"name": "influx",
227-
"url": "http://acc-test.invalid/",
228-
"json_data_map.defaultBucket": "telegraf",
229-
"json_data_map.organization": "organization",
230-
"json_data_map.tlsAuth": "false",
231-
"json_data_map.tlsAuthWithCACert": "false",
232-
"json_data_map.version": "Flux",
233-
"http_headers.Authorization": "Token sdkfjsdjflkdsjflksjdklfjslkdfjdksljfldksjsflkj",
226+
"type": "influxdb",
227+
"name": "influx",
228+
"url": "http://acc-test.invalid/",
229+
"json_data_encoded": `{"defaultBucket":"telegraf","organization":"organization","tlsAuth":false,"tlsAuthWithCACert":false,"version":"Flux"}`,
230+
"http_headers.Authorization": "Token sdkfjsdjflkdsjflksjdklfjslkdfjdksljfldksjsflkj",
234231
},
235232
additionalChecks: []resource.TestCheckFunc{
236233
func(s *terraform.State) error {
@@ -240,17 +237,18 @@ func TestAccDataSource_basic(t *testing.T) {
240237
expected := map[string]interface{}{
241238
"defaultBucket": "telegraf",
242239
"organization": "organization",
243-
"tlsAuth": "false",
244-
"tlsAuthWithCACert": "false",
240+
"tlsAuth": false,
241+
"tlsAuthWithCACert": false,
245242
"version": "Flux",
246243
"httpHeaderName1": "Authorization",
247244
}
248245
if !reflect.DeepEqual(dataSource.JSONData, expected) {
249-
return fmt.Errorf("bad json_data: %#v. Expected: %+v", dataSource.JSONData, expected)
246+
return fmt.Errorf("bad json_data_encoded: %#v. Expected: %+v", dataSource.JSONData, expected)
250247
}
251248
return nil
252249
},
253250
},
251+
verifyImport: true,
254252
},
255253
{
256254
resource: "grafana_data_source.elasticsearch",
@@ -295,6 +293,45 @@ func TestAccDataSource_basic(t *testing.T) {
295293
},
296294
},
297295
},
296+
{
297+
resource: "grafana_data_source.elasticsearch-arbitrary",
298+
config: `
299+
resource "grafana_data_source" "elasticsearch-arbitrary" {
300+
type = "elasticsearch"
301+
name = "elasticsearch-arbitrary"
302+
database_name = "[filebeat-]YYYY.MM.DD"
303+
url = "http://acc-test.invalid/"
304+
json_data_encoded = jsonencode({
305+
esVersion = "7.0.0"
306+
interval = "Daily"
307+
timeField = "@timestamp"
308+
logMessageField = "message"
309+
logLevelField = "fields.level"
310+
maxConcurrentShardRequests = 8
311+
xpack = true
312+
})
313+
}
314+
`,
315+
attrChecks: map[string]string{
316+
"type": "elasticsearch",
317+
"name": "elasticsearch-arbitrary",
318+
"database_name": "[filebeat-]YYYY.MM.DD",
319+
"url": "http://acc-test.invalid/",
320+
"json_data_encoded": `{"esVersion":"7.0.0","interval":"Daily","logLevelField":"fields.level","logMessageField":"message","maxConcurrentShardRequests":8,"timeField":"@timestamp","xpack":true}`,
321+
},
322+
additionalChecks: []resource.TestCheckFunc{
323+
func(s *terraform.State) error {
324+
if dataSource.Name != "elasticsearch-arbitrary" {
325+
return fmt.Errorf("bad name: %s", dataSource.Name)
326+
}
327+
if dataSource.JSONData["xpack"].(bool) != true {
328+
return errors.New("xpack should be true")
329+
}
330+
return nil
331+
},
332+
},
333+
verifyImport: true,
334+
},
298335
{
299336
resource: "grafana_data_source.opentsdb",
300337
config: `
@@ -656,13 +693,19 @@ func TestAccDataSource_basic(t *testing.T) {
656693
},
657694
// Test import using ID
658695
{
659-
ResourceName: test.resource,
660-
ImportState: true,
696+
ResourceName: test.resource,
697+
ImportState: true,
698+
ImportStateVerify: test.verifyImport,
699+
// Ignore sensitive attributes, we mostly only care about "json_data_encoded"
700+
ImportStateVerifyIgnore: []string{"secure_json_data_encoded", "http_headers."},
661701
},
662702
// Test import using UID
663703
{
664-
ResourceName: test.resource,
665-
ImportState: true,
704+
ResourceName: test.resource,
705+
ImportState: true,
706+
ImportStateVerify: test.verifyImport,
707+
// Ignore sensitive attributes, we mostly only care about "json_data_encoded"
708+
ImportStateVerifyIgnore: []string{"secure_json_data_encoded", "http_headers."},
666709
ImportStateIdFunc: func(s *terraform.State) (string, error) {
667710
rs, ok := s.RootModule().Resources[test.resource]
668711
if !ok {

0 commit comments

Comments
 (0)