[BUG]: Unexpected errors and warning with RemoteConfig

[bazilmarkov]

Tracer Version(s)

2.6.0

Go Version(s)

1.25.5

Bug Report

I've been experiencing an issue for a while now, and I can't find a clear root cause. Our fleet of go applications is encountering a consistent rate of errors

Datadog Tracer v2.6.0 ERROR: Error sending stats payload: Post "http://<ip_addr>:8126/v0.6/stats": EOF

despite using different versions of go and dd-trace-go. I've conducted multiple rounds of investigation and debugging at both the application and agent levels, but nothing seems suspicious.

However, in debug mode, I observed attempts to send remote config requests to the agent. Since we have remote config disabled on the agent, I see a number of these errors occurring every five seconds.

Datadog Tracer v2.6.0 DEBUG: remoteconfig: http request error: response status code is not 200 (OK) but Not Found

This behavior is somewhat expected, and a separate variable DD_REMOTE_CONFIGURATION_ENABLED is available to disable the remote config on the client. But after I disabled it, I noticed that there were no longer any errors related to v0.6/stats.

Instead, I started seeing new errors and warnings related to remote config and appsec, which we haven't explicitly enabled anywhere.

WARN: Remote config startup error: could not subscribe to at least one remote config product: remote config client not started; %!w(<nil>)

ERROR: appsec: could not start because of an unexpected error: remote config client not started

No security activities will be collected. Please contact support at https://docs.datadoghq.com/help/ for help.

So my questions are:

1. Is there any chance that these v0.6/stats errors are somehow related to the remote config? I can't find any other factors that could correlate, as these errors are occurring across different agent/dd-trace-go/k8s versions.
2. When I set DD_APPSEC_ENABLED: false explicitly, the appsec-related errors are gone, but from what I see, it should already be false by default (I'm not setting up this variable on the app side). The question is: is it correct behavior to return errors when both appsec and remoteconfig are disabled?
3. The same applies to the remoteconfig warnings - do we need to send them if remoteconfig is deliberately disabled?

Reproduction Code

Set up DD_REMOTE_CONFIGURATION_ENABLED: false

Error Logs

No response

Go Env Output

No response

[darccio]

@bazilmarkov Thanks for reaching out. What's your datadog-agent version?

[bazilmarkov]

Currently, we have 7.74.0 - not the latest, but relatively fresh

[bazilmarkov]

Also, Error sending stats payload I'm experiencing seems to be somehow related to the v2 migration of a library - after updating the dd-trace-go in one of the services from v1.63.0 to v2.6.0, it's there.