Logs from S3 bucket in another Account

[adiospeds]

Hello All,

I have logs going into an s3 bucket in an AWS account X. And CLO is installed in AWS Account Y. If I goto CLO > Application log > Create Pipeline > S3 .. it only allow me to select s3 bucket from same account where CLO is installed (account Y) only even thought Account X is enrolled. Is there a known way I can achieve sending of those logs from S3 account to opensearch via CLO ?

Regions(us-east-1) is same for all S3, CLO and Opensearch. (While I have tried this, the Marketplace App doesn't allow me to write those logs to another account Y so we cant go that route as of now) Please requesting some guidance here if I am missing something here wrt CLO on how this can be achieved. ?

[jangidms]

Hi @adiospeds

Currently, CLO doesn't natively support cross-account ingestion for Application logs with S3 as source.
Below is one of the workaround you can try -

1. Create an Application Log pipeline with any bucket from CLO deployed account as source
2. Modify EventBridge rule:
   • Locate the EventBridge rule in the pipeline stack
   • Update the rule pattern to reference the bucket from your source account
3. Update IAM permissions:
   • Add sts:AssumeRole permission to the LogProcessor lambda role for Member Account Role (found in SubAccount stack output as MemberAccountRoleARN)
4. Update LogProcessor environment variables:
   • Set LOG_SOURCE_ACCOUNT_ASSUME_ROLE to the MemberAccountRoleARN value
   • Set LOG_SOURCE_ACCOUNT_ID to your source account ID
5. Modify LogProcessor Lambda code:.
   • In event/event_parser.py file, change: if log_type in ("Lambda", "RDS") or IS_APP_PIPELINE: to if log_type in ("Lambda", "RDS"):

After implementing these changes, the pipeline will ingest logs from your source account bucket.

[adiospeds]

Thanks @jangidms - This is helpful. I will try this out and post here.

[adiospeds]

Hi @jangidms .. I was trying this today. I did not find the Eventbridge rule.
I see that that the temp bucket which I had used from CLO deployed account has an event notification rule which send the events to an SQS queue and on that sqs queue there is a lambda trigger that seems to be triggering the lambda processor. Can you help me find this eventbridge rule.

[adiospeds]

Also when I tried to change the bucket name parameter in CF stack I get the following error

Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (AccessDenied) when calling the GetBucketNotificationConfiguration operation: User: arn:aws:sts::xxxx:assumed-role/CL-AppPipe-ea1f853f-BucketNotificationsHandler050a0-M5xqIlp0oygi/CL-AppPipe-ea1f853f-BucketNotificationsHandler050a-CXxnHjCMUIbl is not authorized to perform: s3:GetBucketNotification on resource: "arn:aws:s3:::xxxx-logs-xxxxxxxxxxx-ue1" because no resource-based policy allows the s3:GetBucketNotification action.

[jangidms]

It seems like you are using older version of the solution which uses SQS to trigger log processor lambda. You can try above steps after upgrading to latest version of the solution (v2.4.0)

[adiospeds]

Sure - I will try this today .. I am running Template version v2.2.2
Since I am upgrading for the first time (and it isn't mentioned in the doc here ) - just want to understand -

Does this upgrade modify any EXISITING lambda's or ISM policies within the Opensearch cluster ?