MIST-163263 Acceptance test validation by jvander-JNPR · Pull Request #174 · Juniper/terraform-provider-mist

jvander-JNPR · 2026-01-13T19:08:43Z

MIST-163263 Acceptance test validation

Field Coverage Tracking System

Problem: Terraform provider tests need comprehensive field coverage validation, but tracking was manual. This automates discovery of untested schema fields.

Solution: Three-component system using reflection to extract schema fields, normalize test paths, and generate coverage reports.

Core Data Structures

type FieldCoverageTracker struct {
    ResourceName             string
    SchemaFields             map[string]*FieldInfo
    MapNormalizationPaths    map[string]bool         // MapAttribute & MapNestedAttribute
    NormalizedFields         map[string]struct{}     // Deduplicated tested fields
    UnknownFields            map[string]struct{}     // Test paths not matching schema
    SchemaExtractionFailures []string
}

type FieldInfo struct {
    Path           string
    Field          string
    Parent         string
    SchemaAttr     schema.Attribute
    Required       bool
    Optional       bool
    Computed       bool
    MapContainsKey bool             // Synthetic field for MapAttribute {key}
    IsTested       bool
}

Path Normalization Algorithm

Converts concrete test paths to canonical schema paths:

Skip array indices: privileges.0.role → privileges.role, dns_servers.# → dns_servers
Check schema existence: Keep segments that complete known paths
Replace map keys:
- networks.guest.vlan_id → networks.{key}.vlan_id
- port_usages.100.description → port_usages.{key}.description
Mark unknown: Fields not matching schema → UnknownFields

isNumericOrPunctuation() distinguishes context:

List context: "0", "123" → skip as array indices
Map context: "100", "10.0.0.0/8" → replace with {key}

Handles: IPv4/IPv6 CIDR, numeric map keys, nested maps, unlimited depth, unicode, empty paths.

Integration Flow

Setup: FieldCoverageTrackerWithSchema(resourceName, schemaAttributes) extracts schema via reflection. Returns nil when DISABLE_MIST_FIELD_COVERAGE_TRACKER set.
Interception: newTestChecks(path, tracker) → testChecks.append() intercepts TestCheckResourceAttr* calls → tracker.MarkFieldAsTested(fieldPath)
Reporting: tracker.FieldCoverageReport(t) generates JSON to stdout

type CoverageReport struct {
    ResourceName                string
    TestedFieldsCnt             int
    UntestedFieldsCnt           int
    UntestedFields              []string
    UnknownFieldsCnt            int
    UnknownFields               []string
    SchemaExtractionFailuresCnt int
    SchemaExtractionFailures    []string
}

Report filters (isTestableField()):

Computed-only: Computed=true AND Optional=false → excluded
Containers: SingleNestedAttribute, MapNestedAttribute, MapAttribute → excluded
Exception: MapAttribute.{key} → testable
Alphabetically sorted

Schema Extraction

Reflection-based extraction (no public API in terraform-plugin-framework):

extractFields(): Recursive tree walk
extractFieldMetadata(): Reflects Required, Optional, Computed fields
get*NestedAttributes(): Navigate NestedObject.Attributes via reflection

Supported types: String/Bool/Int64/Float64/Number/List/Set/Map/SingleNested/ListNested/SetNested/MapNested

Synthetic fields: Creates {key} entry for MapAttribute paths.

Failures tracked: When reflection can't access nested structures.

Test Suite

TestNormalizeFieldPath (17 scenarios): Array indices, map keys, nested maps, context-aware handling
TestMarkFieldAsTested (6 scenarios): Normalization + schema lookup integration
TestIsNumericOrPunctuation (12 scenarios): CIDR vs field names
TestExtractAllSchemaFields: All attribute types, computed fields, parent tracking, metadata extraction

Isolation: Direct NewFieldCoverageTracker() + extractFields() calls bypass env var.

Key Design Decisions

Reflection required: No public API for nested attributes in terraform-plugin-framework
Unified map tracking: Single MapNormalizationPaths for both MapAttribute and MapNestedAttribute
Context-aware normalization: Numeric strings handled differently in maps vs lists
Container exclusion: Structural nodes excluded; only children testable (except MapAttribute.{key})
Three-phase normalization: Skip indices → check schema → replace map keys
Stdout reporting: Indented JSON with trailing newline
Nil-safe: Returns nil when disabled; call sites check if tracker != nil
Env var gated: DISABLE_MIST_FIELD_COVERAGE_TRACKER disables tracking

Usage

func TestOrgNetworkTemplateModel(t *testing.T) {
    tracker := validators.FieldCoverageTrackerWithSchema(
        "org_networktemplate",
        resource_org_networktemplate.OrgNetworktemplateResourceSchema(context.Background()).Attributes,
    )

    for tName, tCase := range testCases {
        t.Run(tName, func(t *testing.T) {
            checks := tCase.testChecks(t, resourceType, tName, tracker)
            // ... resource.Test() ...
        })
    }
    if tracker != nil {
        tracker.FieldCoverageReport(t)
    }
}

func (o *OrgNetworkTemplateModel) testChecks(..., tracker *validators.FieldCoverageTracker) testChecks {
    checks := newTestChecks(PrefixProviderName(rType) + "." + tName, tracker)
    checks.append(t, "TestCheckResourceAttr", "name", o.Name)
    // ...
    return checks
}

Enabled by default unless DISABLE_MIST_FIELD_COVERAGE_TRACKER set.

100% coverage requirement: untested_fields_count and unknown_fields_count must be 0.

Copilot