ESF incorrectly sets region for VPC flow logs from multi-region S3 bucket

[zmoog]

• Version: 1.14.0

Context

I am creating an issue for a user who reported it using a private channel.

All VPC flow log data, regardless of originating AWS region, is stored in a single bucket in the us-east-1 region.

In this scenario, the user consolidates VPC logs from multiple regions in a single S3 bucket and uses ESF to ingest all VPC logs from the S3 bucket.

ESF creates documents with the region of the S3 bucket instead of the region of the S3 object.

Current behavior

My understanding of the region problem is that the user:

• Is sending all the VPC flow logs in a single S3 bucket; for example, a bucket named my-vpcflow-logs hosted on the region eu-north-1.
• Each VPC Flow log file is stored in an object key like the following:

AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz

• When ESF processes the AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz object, it creates a document with the following fields:

{
  "cloud": {
    "provider": "aws",
    "region": "eu-north-1",
    "account": {
      "id": "123"
    }
  }
}

Expected behavior

• However, since the AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz comes from the eu-central-1 region, the user expects the document to have the following content:

{
  "cloud": {
    "provider": "aws",
    "region": "eu-central-1",
    "account": {
      "id": "123"
    }
  }
}

Notes

In the current version, ESF uses the region from the notification S3 published in the SQS queue.