Datadog endpoint for logs should be unbound from metrics/trace host.

[bradq]

Note:
If you have a feature request, you should contact support so the request can be properly tracked.

Is your feature request related to a problem? Please describe.

I've started playing with CI integration in Datadog. Because Datadog accepts neither stats nor process data in their raw form, we force this plugin to send data to an agent for processing. In the current implementation, this demands I setup a custom TCP "logger" (which I do understand to be closer to a proxy)

The agent's TCP connector feature is understandably brittle, given its lack of broad use. It seems to be somewhat of an orphan in configuration. Indeed, after nearly a month's time, support clarified that though com.datadoghq.ad.logs may largely be configured as expected when running within Docker (using envvars, labels, annotations)...the TCP listener in practice behaves like a check and requires YAML onboard the container to work properly. Although I'm hoping to simply sidestep the need to build a custom container, bind mount, or the like, I'm curious why the configuration for logging is tightly bound to metric collection. There's obvious benefits for consolidating tagging, but that's all I can come up with.

The TCP submission method does not use encryption and is considered by Datadog to be a backup method, and the proxy does seem even less reliable. It's particularly painful given my experience that when the pipe reaches an error state (presumably when Jenkins is abruptly terminated or another malformed request is sent...or some flavor of un-ACKed RST is sent from one end, the agent appears to do nothing to correct the matter. Exascerbating that is that this plugin writes a SEVERE level log after each send error, spamming Jenkins logs 1:1 with any offending job, such that I end up with quite a bit of log bloat on any failure.

I think it's likely that #221 didn't appreciate that the appeal of #74 extended beyond the lack of a local agent. At their core, however, the gathering of CI metrics and logging are independent activities and I'd expect configuration to reflect that if possible.

Describe the solution you'd like
Configuration for CI Metrics remains like present, toggled by a boolean with either unified host/port configuration or preferably (someone's bound to end up with an edge case) separate host/ports for each.

• Logging submission is offered its own host configuration, and is selectable between HTTP and TCP submission

Describe alternatives you've considered

• Not collecting logs at all
• Monitoring Datadog and Jenkins alike to keep the TCP flow connection healthy
• Life as a lumberjack.

Additional context
During my either quixotic or dramatically ill-informed attempt to get the TCP listener to be generated promptly, I've run across a fairly wide swath of troubled souls who had some flavor of trouble setting this up as such. Plenty of unreliable TCP connections, a few issues getting the custom log uptake to start properly, and the occasional local networking frustration. This approach would retain CI functionality while allowing a user to sidestep what I believe is a needless binding of the two configurations.

[shris-github]

Thank you for your note. We will look deeper into our current process for metrics and logs ingestion, but not something we can change in the short term. I would also like to take the opportunity to clarify a few points here

TCP is a backup method
In the context of Jenkins plugin, there is no easy alternative to collect logs. TCP is the next best alternative to when the Agent cannot tail logs from a file. So while TCP is considered a backup, here it makes sense for a TCP connection.

TCP does not use encryption
Yes, while the TCP connection between the Jenkins plugin and the Agent is unencrypted, the Agent itself further processes these logs and sends them over an encrypted HTTPS connection to Datadog.

Plugin generates spammy Jenkins logs upon connection error
Thank you for your feedback. We will take a look to improve the logging and have a more graceful failure logging

[bradq]

In the context of Jenkins plugin, there is no easy alternative to collect logs. TCP is the next best alternative to when the Agent cannot tail logs from a file.

I find this confusing, which is probably my own lack of familiarity. However, forwarding logs over HTTPS to Datadog, directly, which is the default setting, suits my purposes and works consistently. I don't note much complexity in either the writer or the client it invokes. Why is it not possible to offer a configuration option to use the agent for trace data and the like, while sending logs over HTTP? There's multiple other tickets/issues indicating issues with the lack of per-service configuration that this would solve, and outwardly the approach seems simple.

Yes, while the TCP connection between the Jenkins plugin and the Agent is unencrypted, the Agent itself further processes these logs and sends them over an encrypted HTTPS connection to Datadog.

Ah, my misunderstanding, then. "TCP log shipment" is a bit of an overloaded term, and the TCP connector feature isn't terribly well-documented. I certainly don't have much concern about encrypting Jenkins -> agent traffic, but assumed the TCP connector feature was opening a direct port-forward to Datadog's receivers.

TCP is a backup method

This is my primary issue with the approach as-implemented. While I agree it's the agreed-upon solution for the agent to ship in this context, it's not a well-implemented, or well-documented feature. It took me nearly a month to convince DD support that a TCP listener was even a feature that could be implemented by an agent container. Even as such, implementing it using the Docker agent requires a rather un-Dockery bind mount of a configuration YAML file, or otherwise a custom container. I'd consider both considerable drawbacks over simply sending the logs over HTTP. I'm not sure what utility the agent is providing in this context past ensuring consistent tagging.

[github-actions]

Thanks for your contribution!

This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.

If you would like this issue to remain open:

1. Verify that you can still reproduce the issue in the latest version of this project.

2. Comment that the issue is still reproducible and include updated details requested in the issue template.

[bradq]

Yes, nothing has not changed to improve the agent configuration. I can continue to replicate ongoing issues with the TCP connector. Documentation has not been improved. I continue to contemplate life as a lumberjack.

[shris-github]

Hi, I would like to apologize for my tardiness. I have spoken to my team, and we will add a feature request to decouple metrics and logs ingest. We are also currently working on the documentation and should have it ready within a couple of weeks at max.

Regarding the log level - can you share a sample log message. We would like to understand if the message here pertains to failure to send a log line or a connection failure. TIA!

[yzhan289]

Hi @bradq, just bumping this in case you missed @shris-github's message!