[aws_cloudtrail_otel] Content pack of EDOT Cloud Forwarder for AWS - CloudTrail Logs

[mykola-elastic]

Content pack for EDOT Cloud Forwarder for AWS - CloudTrail Logs - Dashboard

Redo of AWS CloudTrail dashboard from AWS integration but using data from https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding/awslogsencodingextension#cloudtrail-log-record-fields

Proposed commit message

See title.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices
• Add auto-install via discovery.datasets

Screenshots

Screenshot of AWS CloudTrail dashboard from AWS integration (for comparison)

Screenshot of the dashboard for content pack

[mykola-elastic]

The EDOT Cloud Forwarder page doesn't yet contain anything about CloudTrail Logs

[mykola-elastic]

ping @ishleenk17 @daniela-elastic marked this one ready for review. In the PR description there are screenshots of the existing dashboard for Cloudtrail logs (from AWS integration), and the one I made with ESQL using data from aws.cloudtrail.otel data_stream for this content pack

[mykola-elastic]

@daniela-elastic @ishleenk17 I have a few questions:

• Is it OK for pies/donuts to display all the data without putting the rest to "Others" category?
• Most of the events come with User ID set to null (see the panel on the bottom right). Should I display them or remove from the table?
• Who else can I invite for review?

[ishleenk17]

• Is it OK for pies/donuts to display all the data without putting the rest to "Others" category?

Is this because there is lesser data to categorize ? Is there is enough data tp categorize, we shoul dmaybe do the top 10 and put rest to others.

• Most of the events come with User ID set to null (see the panel on the bottom right). Should I display them or remove from the table?

If user ID's are NULL mostly, I would assume its not of much relevance to the customer/user ?

• Who else can I invite for review?

Added @ShourieG from Security team.

[ishleenk17]

Use the word " OpenTelemetry" in the Title of the dashboard.
What are the titlees we have used till now for other dashbaords ? I think we should be consistent in how we name them

[mykola-elastic]

In the dashboard titles:

• system_otel uses OTel
• nginx_otel, mysql_otel, iis_otel, and aws elb content pack use OTEL

[mykola-elastic]

Is this because there is lesser data to categorize ? Is there is enough data tp categorize, we shoul dmaybe do the top 10 and put rest to others.

I haven't figured out a way yet how to do LIMIT while having the Others category to display the sum of those which didn't get into the "limit".
There are no configuration options on pie chart to do this, this needs to be done in ES|QL with some complex logic it seems

UPDATE: Confirmed, there is no way to do that in ES|QL at the moment - to show top 10 results and group the rest as "Others" in Kibana pie chart

[daniela-elastic]

Should we exclude "from Amazon S3" in case in the future we enable other ways to get the logs? CC @ishleenk17

[ishleenk17]

@daniela-elastic : Since currently we are supporting only ECF, I think its fine to mention this way.
when we update it with other flows, we will be updating our README's as well

[daniela-elastic]

How do we plan to update this if in the future ECF supports more than just serverless? CC @ishleenk17

[ishleenk17]

When ECF supports more than serverless they will do that with a particular new stack version.
We will also update our kibana version then and along with that the README as well

[daniela-elastic]

Do we have a process to ensure that we don't miss this and then proactively update the README?

[ishleenk17]

UPDATE: Confirmed, there is no way to do that in ES|QL at the moment - to show top 10 results and group the rest as "Others" in Kibana pie chart

So, its an ESQL issue. In KQL it would be possible right ?

[mykola-elastic]

So, its an ESQL issue. In KQL it would be possible right ?

Yes, it is possible to do using Lens (screenshot below).
Though I suggest sticking with ES|QL and fix it when such feature appears. The package version is still 0.1.0 and maybe being consistent is better in this case. What do you think?

[ishleenk17]

So, its an ESQL issue. In KQL it would be possible right ?

Yes, it is possible to do using Lens (screenshot below). Though I suggest sticking with ES|QL and fix it when such feature appears. The package version is still 0.1.0 and maybe being consistent is better in this case. What do you think?

If we don't limit it, its looking cluttered and TMI for the customer.
I would let @daniela-elastic take the final call on this.

[mykola-elastic]

@daniela-elastic ESQL (top) vs LENS (bottom):

Note: in the "User agents" the labels for ESQL panel are short and nice as I used REPLACE with regex to remove clutter. In Lens I have to use labels (user agents) as they are in the data

[daniela-elastic]

@daniela-elastic ESQL (top) vs LENS (bottom):

Note: in the "User agents" the labels for ESQL panel are short and nice as I used REPLACE with regex to remove clutter. In Lens I have to use labels (user agents) as they are in the data

If we already have a workaround for ESQL suggest keeping this and submitting a request to platform / Kibana team to show top 10 results and group the rest as "Others" in Kibana pie chart (submit a ticket and also add to kibana wishlist spreadsheet)

[mykola-elastic]

Opened issue in Kibana elastic/kibana#239279 (and added to the wishlist)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ad22e61

History

• 💚 Build #32748 succeeded 467c6eb
• 💔 Build #32747 failed 2b9c97d
• 💚 Build #32705 succeeded aca5d11
• 💚 Build #32699 succeeded d7931e0

cc @mykola-elastic