Collecting AWS cloudwatch logs

[jaslkaur29]

I am trying to use the awscloudwatch receiver and print them to console for now using logging or debug exporter, this is the config that I am using. I have given full Cloudwatch permissions to IAM role. I can see that OTEL is able to connect with log group but unable to print logs from all the streams. How should I proceed further?

receivers:
  awscloudwatch:
    logs:
      groups:
        autodiscover:
          limit: 100
          prefix: /aws/events/networkmanagerloggroup
      poll_interval: 10s
    region: us-west-2
exporters:
  logging:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      exporters: [logging]
  telemetry:
    logs:
      level: "debug"

These are the logs that I am receiving from my OTEL pod (It does not print any stream related logging)

2024-02-19T18:57:01.806Z debug [email protected]/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: "arn:aws:logs:us-west-2:494665075828:log-group:/aws/events/networkmanagerloggroup:",\n CreationTime: 1706123061129,\n LogGroupClass: "STANDARD",\n LogGroupName: "/aws/events/networkmanagerloggroup",\n MetricFilterCount: 0,\n StoredBytes: 33447\n}"}
2024-02-19T18:57:11.773Z debug [email protected]/logs.go:287 attempting to discover log groups. {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "limit": 10}
2024-02-19T18:57:11.804Z debug [email protected]/logs.go:323 discovered log group {"kind": "receiver", "name": "awscloudwatch", "data_type": "logs", "log group": "{\n Arn: "arn:aws:logs:us-west-2:494665075828:log-group:/aws/events/networkmanagerloggroup:",\n CreationTime: 1706123061129,\n LogGroupClass: "STANDARD",\n LogGroupName: "/aws/events/networkmanagerloggroup",\n MetricFilterCount: 0,\n StoredBytes: 33447\n}"}

[jaslkaur29]

Can you please help with this? @djaglowski @schmikei

[schmikei]

Hmm here's how I'm reading your configuration: retrieve all log streams under discovered log groups with a limit of 100 in us-west-2 with prefix /aws/events/networkmanagerloggroup every 10 seconds.

If that's the intention I think your config looks right! Not sure how helpful I can be without knowing more of your environment information. I can try and emulate an aws-cli filter-log-events for you to see if it highlights anything https://docs.aws.amazon.com/cli/latest/reference/logs/filter-log-events.html

I'm just double checking but could you validate that logs are being sent while the collector lifetime is running? The current config will only start looking at logs that are ingested by CloudWatch 10 seconds from component start and going forward from the last known event time. One thing in the log I noticed was that for the StoredBytes for the discovered cloudwatch log group is not changing (33447 = 33447). So I imagine that the receiver is not perceiving new events in that log group.

I can try to help as best as I can but I think live logs going into Cloudwatch may be what is causing them to not be emitted by the logging exporter. Hopefully that was some helpful perspective, but without knowing more details I think thats about as much insight as I can give!

[jaslkaur29]

The intention is correct, I want data from all the log streams present in a specific log group

Sure, I can provide the environment details:

• The collector is running as a Kubernetes pod on an EC2 Linux instance
• It's using the base image otel/opentelemetry-collector-contrib:0.93.0
  Do you need any more details?

I checked that the logs were not being sent while the collector lifetime was running, is it required?

[schmikei]

I checked that the logs were not being sent while the collector lifetime was running, is it required?

Yes the current design of the receiver is not intended to be used to migrate historical data, it is mostly concerned with ingesting live data within the collector's lifecycle.

You could set the poll interval to a larger value to capture historical data, but this is not the intended use/design of the receiver and would not be recommended in general. This approach will most likely be unsupported in the near future as well,

[jaslkaur29]

This helps, thank you so much!