[Bug] elasticstack_fleet_integration_policy render consistent diff for policy secret

[Holi0317]

Describe the bug

When configuring policy secret via integration policy resource, the provider and terraform always renders a diff even though the value is not changed.

I'm trying to configure the universal profiler integration.

To Reproduce
Steps to reproduce the behavior:

resource "elasticstack_fleet_integration_policy" "profiling" {
  name      = "profiling"

  agent_policy_id     = elasticstack_fleet_agent_policy.this.policy_id
  integration_name    = data.elasticstack_fleet_integration.profiling.name
  integration_version = data.elasticstack_fleet_integration.profiling.version

  input {
    input_id     = "universal_profiling_agent-pf-host-agent"
    streams_json = jsonencode({})
    vars_json = jsonencode({
      "profiler.collector"    = "cloud-es.io"
      "profiler.secret_token" = "Hunter2"
      "profiler.disable_tls"  = false
    })
  }
}

Apply once, then make a plan.

Expected behavior
There should not be any diff in the second plan.

Versions (please complete the following information):

• OS: MacOS
• Terraform Version: v1.7.5
• Provider version 0.1.4
• Elasticsearch Version 8.14.3

[Holi0317]

CC @Kushmaro. Maybe related to #617.

[breml]

We are affected by this issue as well. In our case, it is with the Azure Logs integration where the connection_string and storage_account_key values in the vars_json are now replaced with something like:

{
  "id":  "oJo...",
  "isSecretRef": true
}

when the old value was just a string.

Now it is unclear how to continue, since executing our Terraform would overwrite the referrences to the secret values (and we don't know, if this is safe to do), but on the other hand we do not have a way to manage and import the secret values either.

[breml]

I looked at this problem again and I wonder, what the plan is at Elastic on this.

Based on secrets.md the idea is to prevent Kibana and Fleet users from having read access to the secrets, that are part of the fleet integrations. So far, so good. But now it looks like there is no API left that could be used by e.g. Terraform to manage this as Iaas.

I would be thankful, if someone from Elastic could elaborate on this problem and provide some more details on the plan forward and if possible on a timeline.

[tobio]

provide some more details on the plan forward

To be clear, there's no definitive plan here at the moment. This is just based on similar issues in other resources. Some options though:

1. Having write only attributes. For example API keys will not detect 'drift' in the actual API Key, it's read during the create request and assumed to be static as it's not returned during subsequent read operations. Technically the key could be deleted and recreated outside of Terraform and we wouldn't detect the change. This works well for concrete attributes (i.e not part of a homogeneous JSON blob, which we're not dealing with in this case.
2. lifecycle.ignore_changes. This obviously isn't ideal since Terraform wouldn't correct any legitimate drift within vars_json.

In the providers current state, we can't easily apply 1 in this case since there's just a JSON bag of properties that is used for a variety of input types, and 2 is far from ideal for practitioners. To properly address this issue, we'd need to make some changes to how this resource is implemented, but that's a larger chunk of work and something that will likely be addressed as part of #617. There's no timeframe on either of these issues at the moment sorry.

[breml]

@tobio Thanks for your reply. The current situation is really unfortunate for us. We have a paid subscription of Elasticsearch in the Elastic Cloud. We use terraform to manage the Elastic Cloud deployment as well as the configuration of Elasticsearch including the Fleet integrations.

This worked until we updated our Elasticsearch instance to 8.14.3 and now, we are basically in a dead end, since we can not update our Terraform configuration to compensate for the drift cause by the updates of the fleet integrations, which became necessary due to the update of Elasticsearch.
Already before this change, it was not simple to keep the drift under control, which is caused by the updates of the Fleet integrations. But we managed to do so using tfreveal which showed us the differences in the nested json values (which are additionally marked as sensitive).

But with this recent change, I feel like we are in a dead end.

At some point I read something about a setting, which would allow to disable the new behavior for secrets. Do you have any information about this and how we could enable this in our Elasticsearch instance on Elastic Cloud?

[kpollich]

Hey folks I'm the team lead on Fleet + Integrations and I can probably offer some insight here.

For context, "phase 1" of Fleet's secrets support (preventing read access for sensitive integration data) landed in 8.10. The scope was tracked in elastic/kibana#154715 and the feature flag was enabled by default in 8.10 in elastic/kibana#163782.

The goal of these integration secrets is to stop allowing all ES users to read sensitive data (like your Azure API key or AWS access key) directly from the .fleet-policies index or from Kibana saved object documents (e.g. .kibana* indices) using the Elasticsearch API. Phase 1 of this work involved moving to a reference-based model where secrets are stored in a less accessible system index .fleet_secrets where only the Fleet service account has read access. Fleet Server is responsible for replacing these secret references on-the-fly when it creates policy updates for Elastic Agents enrolled in the fleet.

At some point I read something about a setting, which would allow to disable the new behavior for secrets. Do you have any information about this and how we could enable this in our Elasticsearch instance on Elastic Cloud?

Secrets are unfortunately not disable-able at this time. It'd be nontrivial to support toggling secrets from enabled -> disabled, and this would require a user to re-enter all previously secret values manually as Kibana can't read them back.

We are affected by this issue as well. In our case, it is with the Azure Logs integration where the connection_string and storage_account_key values in the vars_json are now replaced with something like:

{
  "id":  "oJo...",
  "isSecretRef": true
}
when the old value was just a string.

Now it is unclear how to continue, since executing our Terraform would overwrite the referrences to the secret values (and we don't know, if this is safe to do), but on the other hand we do not have a way to manage and import the secret values either.

I'm not sure I follow this. bit I have nearly 0 experience with Terraform so please forgive my ignorance here. I am going to reiterate the current workflow around upgrading packages + secrets for clarity here.

From the Fleet side, we use the PUT /api/fleet/package_policies/:id API to perform the package policy upgrade, and including the secrets as plaintext during the upgrade request (e.g. includes bumped package.version property) will automatically convert them to secret references behind the scenes:

# Previewed API request
PUT kbn:/api/fleet/package_policies/b1452a92-bc5d-49e4-b3fe-105f24744da3
{
  "package": {
    "name": "1password",
    "version": "1.27.0",
    "experimental_data_stream_features": []
  },
  "name": "1password-3",
  "namespace": "",
  "description": "",
  "policy_ids": [
    "5e14e167-4efa-4cb1-8749-73b1f69b450c"
  ],
  "vars": {},
  "inputs": {
    "1password-httpjson": {
      "enabled": true,
      "vars": {
        "url": "https://events.1password.com",
        "token": "changeme", # <--- Previously existing non-secret value before upgrade
        "disable_keep_alive": false
      },
      "streams": {
        "1password.audit_events": {
          "enabled": true,
          "vars": {
            "limit": 1000,
            "interval": "10s",
            "tags": [
              "forwarded",
              "1password-audit_events"
            ],
            "preserve_original_event": false
          }
        },
        "1password.item_usages": {
          "enabled": true,
          "vars": {
            "limit": 1000,
            "interval": "10s",
            "tags": [
              "forwarded",
              "1password-item_usages"
            ],
            "preserve_original_event": false
          }
        },
        "1password.signin_attempts": {
          "enabled": true,
          "vars": {
            "limit": 1000,
            "interval": "10s",
            "tags": [
              "forwarded",
              "1password-signin_attempts"
            ],
            "preserve_original_event": false
          }
        }
      }
    }
  }
}

The response to this request no longer contains the plaintext version of the secret, as its been converted to a secret reference, e.g.

The isSecretRef: true and id property tell Fleet where to find the relevant secret document in .fleet_secrets when it comes time to handle templating for the policy.

From this point forward, the original changeme string is no longer stored on the policy object in any way, and is replaced entirely with this new secret reference structure. I can understand how this signals drift to Terraform at a basic level.

Unfortunately, looking at this I don't think there's anything we can do immediately in the Fleet API to improve this - it seems like an issue with the Elastic Terraform provider in general. This API isn't implemented in a very IAC-friendly way, obviously, so it's a bummer that we have to push work into the provider to correct its idiosyncrasies like this, but I don't see a change we could make on the Fleet side that would improve this right now.

From Fleet's perspective, we have a requirement of "user puts plaintext in -> user receives a secret reference out" for these package policy secrets (and output secrets as well). From what I understand here this is always going to signal drift to Terraform because we PUT { password: "changeme" } and receive a nested JSON structure instead of "changeme" in the response.

The only real recourse I can offer at this time is to lock down the version of each one of your integrations to a version in which no secrets are set up. Secrets are defined at the integration level, so freezing integration versions will mean you don't have to deal with secrets. This is obviously problematic when it comes to receiving patches for integrations, but it's all I can provide here 😞

[Kushmaro]

@kpollich lets sync on this internally please.

[breml]

@Kushmaro , @kpollich I would appreciate a short update on this issue after your internal sync. Thanks.

[breml]

Hi @kpollich

Thanks for the insights you shared in your post above. I understand the underlying need to better protect the secrets, that are stored within Elasticsearch.
That being said, the whole situation is very unfortunate for us. We invested significantly in managing our Elastic Stack infrastructure with Terraform. Now we learn, that this is an afterthought for Elastic and it might break at any time, to the extend, where one is forced to lock down versions, basically leading to a situation, where one can no longer update Elasticsearch and Kibana. This is pretty problematic in my opinion and I am now really unsure, how I can sell / explain this to my managers.

Having a situation, where one half of the setup is handled as Iaas (configured with Terraform) and the other half is done manually basically renders the whole Iaas approach useless, since the setup will still be manual and error prone and it can not be automated with CI/CD systems, since there are manual steps involved.

Above, you mentioned the API call PUT /api/fleet/package_policies/:id. Is this API documented somewhere?

[daemitus]

One solution to this is to migrate this resource to the framework from sdk-v2, and use the privatestate.ProviderData to store a map of var/key to secretRef. Then when doing a read, iterate the vars and replace anything that has a stored ref. Should work in theory, I haven't had the time to fully test out the bit I've been working on though.

[kpollich]

Above, you mentioned the API call PUT /api/fleet/package_policies/:id. Is this API documented somewhere?

The API is documented here: https://www.elastic.co/docs/api/doc/kibana/operation/operation-get-package-policies. These are sourced from an OpenAPI spec available here: https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/common/openapi/bundled.yaml.

I would appreciate a short update on this issue after your internal sync. Thanks.

@Kushmaro and I chatted internally and involved some other folks about getting improvements to the terraform provider around Fleet secrets prioritized.

In short, the team that develops Fleet has no knowledge or insight into Elastic's terraform provider. Describing it as an afterthought is perhaps a little harsh, but probably fair at least as it relates to the Fleet team itself, which is made up of primarily JavaScript engineers working in Kibana. Simply put there's no expertise or experience with terraform on the Fleet team. This has obviously led us to this situation where changes made on the Fleet team are not made in a way that considers the terraform provider, which we're actively seeking to correct internally.

For now, there's no real clear workaround to deal with this right now that I'm aware of. As I mentioned above, though, I am by no means an expert or even a novice terraform user. There may be some way to hack around using the Fleet API directly and custom scripting within your terraform code, but I wouldn't pretend to know how to approach that as an option.

If the fundamental problem here is that the API input into the Fleet API does not match the eventual output, then I'm afraid that's not something we'd entertain "fixing" about this secrets API - it is an intentional choice to no echo out sensitive policy data like credentials once they've been persisted on a policy. It would defeat the purpose of secrets entirely if the API echoed out the credentials after saving just for the sake of avoiding drift in terraform. This needs to be fixed in the provider itself.

We'll continue to discuss our options for getting a change into the provider to work with the Fleet policy secrets, and in the meantime we are absolutely open to ideas or proofs of concept from the community like @daemitus has suggested above.