helper/resource: Add TF_ACC_LOG, TF_LOG_CORE, and TF_LOG_PROVIDER environment variable handling for Terraform versions 0.15 and later (#993)

Reference: #992

This enables provider developers to have full control over Terraform CLI logging environment variables with the acceptance testing framework in `helper/resource`.

In Terraform CLI 0.15 and later, the logging environment variables work as the following:

| Environment Variable | Description |
| - | - |
| `TF_LOG` | Global logging level for core and providers. Overrides `TF_LOG_CORE` (but not `TF_LOG_PROVIDER`) if set. |
| `TF_LOG_CORE` |  Logging level for core functionality. No effect if `TF_LOG` is set. |
| `TF_LOG_PROVIDER` | Logging level for provider logging sent over plugin file descriptors. |

To execute Terraform CLI during acceptance testing, it uses `terraform-exec` for all interactions. `terraform-exec` has certain environment variable handling rules associated with it, to prevent issues with reading stdout/stderr output from calling Terraform CLI since it needs to be able to parse that information. It was recently updated to support setting `TF_LOG`, `TF_LOG_CORE`, and `TF_LOG_PROVIDER` environment variables when calling Terraform, in addition to the prior support for setting `TF_LOG_PATH`, which defaults to always setting `TF_LOG` to `TRACE` if unset.

This enhancement allows the acceptance testing framework to accept the `TF_ACC_LOG` environment variable to appropriately set the `TF_LOG` environment variable in `terraform-exec`, if Terraform is version 0.15 or later. We could not change the behavior of the existing `TF_LOG` environment variable as it would cause undesirable side effects with the existing testing framework handling for the Go standard library `log` package. The `TF_ACC_LOG` naming follows the convention of `TF_ACC_LOG_PATH` setting `TF_LOG_PATH`. The `TF_LOG` value defaults to `TRACE`, similar to before, when `TF_ACC_LOG_PATH` or `TF_LOG_PATH_MASK` environment variables are set.

This also enables the passthrough of `TF_LOG_CORE` and `TF_LOG_PROVIDER` environment variables to `terraform-exec`. This is required because `terraform-exec` will otherwise default `TF_LOG` to `TRACE` if `TF_LOG_PATH` is used, meaning that without this, provider developers could not disable Terraform core logging. This enhancement opts to not prefix the variables with `TF_ACC_` as there's no prior behaviors to protect and the differing environment variable names for acceptance testing is already confusing enough for everyone. They could still be prefixed for consistency with `TF_ACC_LOG` though, if desired.

To summarize the state of logging environment variable with the acceptance testing framework after this change:

| Environment Variable | Description |
| - | - |
| `TF_ACC_LOG` | Controls the Terraform CLI `TF_LOG` environment variable |
| `TF_ACC_LOG_PATH` | Controls the Terraform CLI `TF_LOG_PATH` environment variable and where terraform-plugin-log SDK/provider loggers will write logs |
| `TF_LOG` | Controls the Go standard library `log` package log level; Any level enables terraform-plugin-log SDK/provider loggers at `TRACE` currently (they have to be tuned down individually at the moment) |
| `TF_LOG_CORE` | Controls the Terraform CLI `TF_LOG_CORE` environment variable |
| `TF_LOG_PATH_MASK` | Act similar to `TF_ACC_LOG_PATH`, but replaces a `%s` placeholder in the value with individual test names (`(testing.T).Name()`) |
| `TF_LOG_PROVIDER` | Controls the Terraform CLI `TF_LOG_PROVIDER` environment variable |

Author: bflad

.changelog/993.txt

@@ -0,0 +1,3 @@
+```release-note:enhancement
+helper/resource: Added `TF_ACC_LOG`, `TF_LOG_CORE`, and `TF_LOG_PROVIDER` environment variable handling for Terraform versions 0.15 and later
+```

internal/logging/keys.go

@@ -31,6 +31,15 @@ const (
 	// The TestStep number of the test being executed. Starts at 1.
 	KeyTestStepNumber = "test_step_number"
 
+	// The Terraform CLI logging level (TF_LOG) used for an acceptance test.
+	KeyTestTerraformLogLevel = "test_terraform_log_level"
+
+	// The Terraform CLI logging level (TF_LOG_CORE) used for an acceptance test.
+	KeyTestTerraformLogCoreLevel = "test_terraform_log_core_level"
+
+	// The Terraform CLI logging level (TF_LOG_PROVIDER) used for an acceptance test.
+	KeyTestTerraformLogProviderLevel = "test_terraform_log_provider_level"
+
 	// The path to the Terraform CLI logging file used for an acceptance test.
 	//
 	// This should match where the rest of the acceptance test logs are going

internal/plugintest/environment_variables.go

@@ -10,6 +10,24 @@ const (
 	// CLI installation, if installation is required.
 	EnvTfAccTempDir = "TF_ACC_TEMP_DIR"
 
+	// Environment variable with level to filter Terraform logs during
+	// acceptance testing. This value sets TF_LOG in a safe manner when
+	// executing Terraform CLI commands, which would otherwise interfere
+	// with the testing framework using TF_LOG to set the Go standard library
+	// log package level.
+	//
+	// This value takes precedence over TF_LOG_CORE, due to precedence rules
+	// in the Terraform core code, so it is not possible to set this to a level
+	// and also TF_LOG_CORE=OFF. Use TF_LOG_CORE and TF_LOG_PROVIDER in that
+	// case instead.
+	//
+	// If not set, but TF_ACC_LOG_PATH or TF_LOG_PATH_MASK is set, it defaults
+	// to TRACE. If Terraform CLI is version 0.14 or earlier, it will have no
+	// separate affect from the TF_ACC_LOG_PATH or TF_LOG_PATH_MASK behavior,
+	// as those earlier versions of Terraform are unreliable with the logging
+	// level being outside TRACE.
+	EnvTfAccLog = "TF_ACC_LOG"
+
 	// Environment variable with path to save Terraform logs during acceptance
 	// testing. This value sets TF_LOG_PATH in a safe manner when executing
 	// Terraform CLI commands, which would otherwise be ignored since it could
@@ -18,6 +36,17 @@ const (
 	// If TF_LOG_PATH_MASK is set, it takes precedence over this value.
 	EnvTfAccLogPath = "TF_ACC_LOG_PATH"
 
+	// Environment variable with level to filter Terraform core logs during
+	// acceptance testing. This value sets TF_LOG_CORE separate from
+	// TF_LOG_PROVIDER when calling Terraform.
+	//
+	// This value has no affect when TF_ACC_LOG is set (which sets Terraform's
+	// TF_LOG), due to precedence rules in the Terraform core code. Use
+	// TF_LOG_CORE and TF_LOG_PROVIDER in that case instead.
+	//
+	// If not set, defaults to TF_ACC_LOG behaviors.
+	EnvTfLogCore = "TF_LOG_CORE"
+
 	// Environment variable with path containing the string %s, which is
 	// replaced with the test name, to save separate Terraform logs during
 	// acceptance testing. This value sets TF_LOG_PATH in a safe manner when
@@ -27,6 +56,22 @@ const (
 	// Takes precedence over TF_ACC_LOG_PATH.
 	EnvTfLogPathMask = "TF_LOG_PATH_MASK"
 
+	// Environment variable with level to filter Terraform provider logs during
+	// acceptance testing. This value sets TF_LOG_PROVIDER separate from
+	// TF_LOG_CORE.
+	//
+	// During testing, this only affects external providers whose logging goes
+	// through Terraform. The logging for the provider under test is controlled
+	// by the testing framework as it is running the provider code. Provider
+	// code using the Go standard library log package is controlled by TF_LOG
+	// for historical compatibility.
+	//
+	// This value takes precedence over TF_ACC_LOG for external provider logs,
+	// due to rules in the Terraform core code.
+	//
+	// If not set, defaults to TF_ACC_LOG behaviors.
+	EnvTfLogProvider = "TF_LOG_PROVIDER"
+
 	// Environment variable with acceptance testing Terraform CLI version to
 	// download from releases.hashicorp.com, checksum verify, and install. The
 	// value can be any valid Terraform CLI version, such as 1.1.6, with or

internal/plugintest/helper.go

@@ -139,17 +139,102 @@ func (h *Helper) NewWorkingDir(ctx context.Context, t TestControl) (*WorkingDir,
 		return nil, fmt.Errorf("unable to disable terraform-exec provider verification: %w", err)
 	}
 
+	tfAccLog := os.Getenv(EnvTfAccLog)
+	tfAccLogPath := os.Getenv(EnvTfAccLogPath)
+	tfLogCore := os.Getenv(EnvTfLogCore)
+	tfLogPathMask := os.Getenv(EnvTfLogPathMask)
+	tfLogProvider := os.Getenv(EnvTfLogProvider)
+
+	if tfAccLog != "" && tfLogCore != "" {
+		err = fmt.Errorf(
+			"Invalid environment variable configuration. Cannot set both TF_ACC_LOG and TF_LOG_CORE. " +
+				"Use TF_LOG_CORE and TF_LOG_PROVIDER to separately control the Terraform CLI logging subsystems. " +
+				"To control the Go standard library log package for the provider under test, use TF_LOG.",
+		)
+		logging.HelperResourceError(ctx, err.Error())
+		return nil, err
+	}
+
+	if tfAccLog != "" {
+		logging.HelperResourceTrace(
+			ctx,
+			fmt.Sprintf("Setting terraform-exec log level via %s environment variable, if Terraform CLI is version 0.15 or later", EnvTfAccLog),
+			map[string]interface{}{logging.KeyTestTerraformLogLevel: tfAccLog},
+		)
+
+		err := tf.SetLog(tfAccLog)
+
+		if err != nil {
+			if !errors.As(err, new(*tfexec.ErrVersionMismatch)) {
+				logging.HelperResourceError(
+					ctx,
+					"Unable to set terraform-exec log level",
+					map[string]interface{}{logging.KeyError: err.Error()},
+				)
+				return nil, fmt.Errorf("unable to set terraform-exec log level (%s): %w", tfAccLog, err)
+			}
+
+			logging.HelperResourceWarn(
+				ctx,
+				fmt.Sprintf("Unable to set terraform-exec log level via %s environment variable, as Terraform CLI is version 0.14 or earlier. It will default to TRACE.", EnvTfAccLog),
+				map[string]interface{}{logging.KeyTestTerraformLogLevel: "TRACE"},
+			)
+		}
+	}
+
+	if tfLogCore != "" {
+		logging.HelperResourceTrace(
+			ctx,
+			fmt.Sprintf("Setting terraform-exec core log level via %s environment variable, if Terraform CLI is version 0.15 or later", EnvTfLogCore),
+			map[string]interface{}{
+				logging.KeyTestTerraformLogCoreLevel: tfLogCore,
+			},
+		)
+
+		err := tf.SetLogCore(tfLogCore)
+
+		if err != nil {
+			logging.HelperResourceError(
+				ctx,
+				"Unable to set terraform-exec core log level",
+				map[string]interface{}{logging.KeyError: err.Error()},
+			)
+			return nil, fmt.Errorf("unable to set terraform-exec core log level (%s): %w", tfLogCore, err)
+		}
+	}
+
+	if tfLogProvider != "" {
+		logging.HelperResourceTrace(
+			ctx,
+			fmt.Sprintf("Setting terraform-exec provider log level via %s environment variable, if Terraform CLI is version 0.15 or later", EnvTfLogProvider),
+			map[string]interface{}{
+				logging.KeyTestTerraformLogCoreLevel: tfLogProvider,
+			},
+		)
+
+		err := tf.SetLogProvider(tfLogProvider)
+
+		if err != nil {
+			logging.HelperResourceError(
+				ctx,
+				"Unable to set terraform-exec provider log level",
+				map[string]interface{}{logging.KeyError: err.Error()},
+			)
+			return nil, fmt.Errorf("unable to set terraform-exec provider log level (%s): %w", tfLogProvider, err)
+		}
+	}
+
 	var logPath, logPathEnvVar string
 
-	if tfAccLogPath := os.Getenv(EnvTfAccLogPath); tfAccLogPath != "" {
+	if tfAccLogPath != "" {
 		logPath = tfAccLogPath
 		logPathEnvVar = EnvTfAccLogPath
 	}
 
 	// Similar to helper/logging.LogOutput() and
 	// terraform-plugin-log/tfsdklog.RegisterTestSink(), the TF_LOG_PATH_MASK
 	// environment variable should take precedence over TF_ACC_LOG_PATH.
-	if tfLogPathMask := os.Getenv(EnvTfLogPathMask); tfLogPathMask != "" {
+	if tfLogPathMask != "" {
 		// Escape special characters which may appear if we have subtests
 		testName := strings.Replace(t.Name(), "/", "__", -1)
 		logPath = fmt.Sprintf(tfLogPathMask, testName)

website/docs/plugin/sdkv2/testing/acceptance-tests/index.mdx

@@ -238,14 +238,34 @@ A number of environment variables are available to control aspects of acceptance
 | Environment Variable Name | Default | Description |
 |---------------------------|---------|-------------|
 | `TF_ACC` | N/A | Set to any value to enable acceptance testing via the [`helper/resource.ParallelTest()`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#ParallelTest) and [`helper/resource.Test()`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#Test) functions. |
-| `TF_ACC_LOG_PATH` | N/A | Set a path for Terraform logs during testing. Refer to `TF_LOG_PATH_MASK` to configure individual log files per test. |
 | `TF_ACC_PROVIDER_HOST`: | `registry.terraform.io` | Set the hostname of the provider under test, such as `example.com` in the `example.com/myorg/myprovider` provider source address. This is only required if any [`TestStep.Config`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#TestStep.Config) specifies a provider source address, such as in the [`terraform` configuration block `required_providers` attribute](https://www.terraform.io/language/settings#specifying-provider-requirements). |
 | `TF_ACC_PROVIDER_NAMESPACE` | `hashicorp` | Set the namespace of the provider under test, such as `myorg` in the `registry.terraform.io/myorg/myprovider` provider source address. This is only required if any [`TestStep.Config`](https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource#TestStep.Config) specifies a provider source address, such as in the [`terraform` configuration block `required_providers` attribute](https://www.terraform.io/language/settings#specifying-provider-requirements). |
 | `TF_ACC_STATE_LINEAGE` | N/A | Set to `1` to enable state lineage debug logs, which are normally suppressed during acceptance testing. |
 | `TF_ACC_TEMP_DIR` | Operating system specific via [`os.TempDir()`](https://pkg.go.dev/os#TempDir) | Set a temporary directory used for testing files and installing Terraform CLI, if installation is required. |
 | `TF_ACC_TERRAFORM_PATH` | N/A | Set the path to a Terraform CLI binary on the local filesystem to be used during testing. It must be executable. If not found and `TF_ACC_TERRAFORM_VERSION` is not set, an error is returned. |
 | `TF_ACC_TERRAFORM_VERSION` | N/A | Set the exact version of Terraform CLI to automatically install into `TF_ACC_TEMP_DIR`. For example, `1.1.6` or `v1.0.11`. |
-| `TF_LOG_PATH_MASK` | N/A | Set a file path containing the string `%s`, which is replaced with the test name, to write a separate log file per test. Refer to `TF_ACC_LOG_PATH` to configure a single log file for all tests. |
+
+### Logging Environment Variables
+
+A number of environment variables available to control logging aspects during acceptance test execution. Some of these modify or replace the production behaviors defined in [managing provider log output](/plugin/log/managing) and [debugging Terraform](/internals/debugging).
+
+#### Logging Levels
+
+| Environment Variable Name | Default | Description |
+|---------------------------|---------|-------------|
+| `TF_ACC_LOG` | N/A | Set the `TF_LOG` environment variable used by Terraform CLI while testing. If set, overrides `TF_LOG_CORE`. Use `TF_LOG_CORE` and `TF_LOG_PROVIDER` to configure separate levels for Terraform CLI logging. |
+| `TF_LOG` | N/A | Set the log level for the Go standard library `log` package. If set to any level, sets the `TRACE` log level for any SDK and provider logs written by [`terraform-plugin-log`](/plugin/log/writing). Use the `TF_LOG_SDK*` and `TF_LOG_PROVIDER_*` environment variables described in [managing log output](/plugin/log/managing) to decrease or disable SDK and provider logs written by [`terraform-plugin-log`](/plugin/log/writing). Use `TF_ACC_LOG`, `TF_LOG_CORE`, or `TF_LOG_PROVIDER` environment variables to set the logging levels used by Terraform CLI while testing. |
+| `TF_LOG_CORE` | `TF_ACC_LOG` | Set the `TF_LOG_CORE` environment variable used by Terraform CLI logging of graph operations and other core functionality while testing. If `TF_ACC_LOG` is set, this setting has no effect. Use `TF_LOG_PROVIDER` to configure a separate level for Terraform CLI logging of external providers while testing (e.g. defined by the `TestCase` or `TestStep` type `ExternalProviders` field). |
+| `TF_LOG_PROVIDER` | `TF_ACC_LOG` | Set the `TF_LOG_PROVIDER` environment variable used by Terraform CLI logging of external providers while testing (e.g. defined by the `TestCase` or `TestStep` type `ExternalProviders` field). If set, overrides `TF_ACC_LOG`. Use `TF_LOG_CORE` to configure a separate level for Terraform CLI logging of graph operations and other core functionality while testing. |
+
+#### Logging Output
+
+By default, there is no logging output when running the `go test` command. Use one of the below environment variables to output logs to the local filesystem or use the `go test` command `-v` (verbose) flag to view logging without writing file(s).
+
+| Environment Variable Name | Default | Description |
+|---------------------------|---------|-------------|
+| `TF_ACC_LOG_PATH` | N/A | Set a file path for all logs during testing. Use `TF_LOG_PATH_MASK` to configure individual log files per test. |
+| `TF_LOG_PATH_MASK` | N/A | Set a file path containing the string `%s`, which is replaced with the test name, to write a separate log file per test. Use `TF_ACC_LOG_PATH` to configure a single log file for all tests. |
 
 ## Troubleshooting