Fix JSON format ignored in some logging subsystems

[uristernik]

Fixes #37433

Target Release

1.14.x

Rollback Plan

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

This actually fixes what is described in the docs https://developer.hashicorp.com/terraform/internals/debugging

• This change is user-facing and I added a changelog entry.
• This change is not user-facing.

[hashicorp-cla-app]

All committers have signed the CLA.

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

_{Have you signed the CLA already but the status is still pending? Recheck it.}

[crw]

Thanks for this submission, I will raise it in triage.

[uristernik]

@crw any chance we can expedite this? can I help somehow?

[crw]

Hi @uristernik, this PR is unlikely (but not impossible) to be reviewed due to making an already complicated situation more complicated. For reference here is an old RFC for provider logging: hashicorp/terraform-plugin-sdk#695, and here is a relevant comment on the PR that implemented TF_LOG=JSON: #27850

my assumption was that we wouldn't make any support guarantees around this at all except that JSON was a valid log level that would output structured logs at a TRACE level. This would make it hard for provider developers to use jq or custom tooling to parse logs, but that wasn't really the intended user experience.

This seems to be the intended implementation. Combining the log format with the log level did not help matters.

That said, I am going to try to get a final answer as to whether this PR would be considered or not, as it may not make things worse, just different.

[uristernik]

@crw Thanks for taking the time to review this and for considering the request.

To elaborate on our use case: we’ve built a CLI tool similar to the one the PR author mentioned. This tool parses logs and extracts insights, and provider logs are particularly valuable for us since we manage fairly large workspaces. At this scale, we often run into API throttling issues, and with this tool we can profile requests and optimize them more effectively.

Currently, we’re working with log files of around ~800Mi. By leveraging provider logs, we can cut that size down significantly, which makes storage more efficient and profiling much faster.

From my perspective, this PR doesn’t alter existing behavior - it simply extends the expected behavior to all logging subsystems, which I believe was the original intent. This comment seems to support that interpretation: #35298 (comment)

Terraform effectively treats the log level json as meaning “trace but in JSON format”. You can set TF_LOG_CORE=json to get just the Terraform Core logs as “trace but in JSON format”.

Thanks again, LMK what do you think 🙏

[uristernik]

gentle ping @crw 🙏

[crw]

Hi @uristernik, apologies, I do not have an update. I raised this comment to the team, there was a bit of a confluence of interest due to the coincidental revival of this thread: https://discuss.hashicorp.com/t/tf-log-provider-name-does-not-work/39963. However, given the relative close timing to HashiConf deadlines, I would be surprised if this gets picked up for review in the near future. I think it is worth keeping open as this logging conundrum may be revisited later, but I leave it in your hands if you want to keep this open. Thanks!