Shadow Drift - Plan Modifiers generic.MultiSet and UseStateForUnknown Triggers False Positive Drift

[wellsiau-aws]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• The resources and data sources in this provider are generated from the CloudFormation schema, so they can only support the actions that the underlying schema supports. For this reason submitted bugs should be limited to defects in the generation and runtime code of the provider. Customizing behavior of the resource, or noting a gap in behavior are not valid bugs and should be submitted as enhancements to AWS via the CloudFormation Open Coverage Roadmap.

Terraform CLI and Terraform AWS Cloud Control Provider Version

Affected Resource(s)

• awscc_s3_bucket
• various other resources with similar schema in their attributes (see detail below)

Summary

The AWSCC provider experiences "shadow drift" where Terraform detects configuration drift even when no actual changes exist. This issue specifically manifests when resources have both diff suppression plan modifiers (like generic.Multiset()) and computed attributes using UseStateForUnknown() plan modifiers operating simultaneously. This issue is triggered when other plan modifiers such as generic.Multiset() was called due to upstream API returning slighly different attribute value that still semantically similar.

Problem Description

Framework Behavior

The following is the expected framework behavior and we dont plan to change it:

1. When attributes are marked as Optional: true, Computed: true, the framework automatically sets them to "unknown" during planning.
2. UseStateForUnknown() only copies state values when they are non-null/known. When state values are null, the modifier does nothing, leaving the attribute as unknown.

API interaction

During resource creation, the upstream CCAPI mostly will send null when optional attribute was not specified, such null value are also stored in the Terraform statefile.

Trigger point

AWS Tags is one of the most common attribute across all resources. We learned from several test that CCAPI GetResource might return tags in different order than what was declared in Terraform config. We have generic.Multiset() plan modifiers to address this issue in isolation.

The Interaction Problem

The shadow drift issue is not caused by UseStateForUnknown() plan modifiers in isolation, but rather by their interaction with other diff suppression mechanisms:

1. Trigger Condition: Shadow drift only becomes visible when resources have attributes with diff suppression plan modifiers (e.g., generic.Multiset() for tag ordering).
2. Framework Cascade Effect: When generic.Multiset() is called to suppress legitimate drift, the framework simultaneously processes other computed attributes with UseStateForUnknown() modifiers.
3. Even though generic.Multiset() successfully suppresses drift for its specific attribute, the presence of other "unknown" computed attributes (left unknown due to the plan modifier sees null value in the state) causes Terraform Core to still detect overall resource drift.

Why This Interaction Happened

1. In isolation: UseStateForUnknown() modifiers alone don't cause visible drift because Terraform Core can handle null-to-null comparisons (this is my assumption)
2. With diff suppression: When other plan modifiers like generic.Multiset() are actively suppressing drift, the framework's evaluation of all attributes reveals the "unknown" computed attributes, triggering false positive drift detection.

Affected Scenarios

This issue specifically affects resources that have both:

1. Attributes with diff suppression plan modifiers (generic.Multiset(), semantic equality checks, etc.),
2. Multiple optional + computed attributes with UseStateForUnknown() plan modifiers where state values are null.

Proposed Solution

Custom Plan Modifier Implementation

Replace UseStateForUnknown() with custom implementations that explicitly handle the null-to-null case:

1. IF request config is null (because attribute is not configured in the config), AND
2. IF request state is null (because API did not return default values and state file recorded it as null),
3. THEN set response to null
   This will suppress drift, because all values returned to the Terraform core (as null instead of unknown).

Consideration

The following items need more research:

1. What would happen to resource where the upstream CCAPI returned default values while default values is not declared in schema.
2. How to handle this in the emitter.

References

• #0000

[ewbankkit]

Relates?

• UseStateForUnknown doesn't preserve known null prior state values terraform-plugin-framework#1117
• all: Adjust UseStateForUnknown plan modifiers to preserve known null state values terraform-plugin-framework#1161
• Unexpected Behavior With Plan Modifier (UseStateForUnknown) Prior State Under Lists and Sets terraform-plugin-framework#709
• Consider MatchElementStateForUnknown() Plan Modifiers terraform-plugin-framework#717
• SetNestedAttributes plan does not like single nested compute null values terraform-plugin-framework#302

[wellsiau-aws]

@ewbankkit - yes I think if we could use the upstream fix terraform-plugin-framework #1161 it should achieve the same behavior.

[jar-b]

#2738 included an update of github.com/hashicorp/terraform-plugin-framework to 1.15.1. This version includes a bug fix to preserve known null values in state when the UseStateForUnknown plan modifier is used.

We expect this change to resolve the behavior described here, and will retroactively add an entry to the changelog for the v1.51.0 AWSCC provider release.