Solution Upgrade: Upgrading from v2.2.2 causes CloudFormation rollback (CRInitConfig validation errors)

[yanapoz]

Describe the bug

Attempting to upgrade the Centralized Logging with OpenSearch AWS Solution from v2.2.2 to a newer version consistently rolls back during the stack update. The failure occurs in the Lambda-backed custom resource CRInitConfig, which throws validation errors for missing fields.

Expected behavior

Upgrade is successful.

Current Behavior

The resource CRCRLambdaDFCBCB5E is in a UPDATE_FAILED state
This AWS::CloudFormation::CustomResource resource is in a UPDATE_FAILED state.

Received response status [FAILED] from custom resource. Message returned: Error: 3 validation errors for OpenSearchDomain domainInfo field required (type=value_error.missing) importMethod field required (type=value_error.missing) resources field required (type=value_error.missing) Logs: /aws/lambda/centralized-logging-stack-v2-CRInitConfig61C535DA-hFDu9QONvkgP at invokeUserFunction (/var/task/framework.js:2:6) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async onEvent (/var/task/framework.js:1:369) at async Runtime.handler (/var/task/cfn-response.js:1:1837) (RequestId: 43a7e065-356b-4316-8981-a536dbd38c9f)

│ Error: waiting for CloudFormation Stack (centralized-logging-stack-v2) update: stack status (UPDATE_ROLLBACK_COMPLETE): Resource update cancelled │ Resource update cancelled │ Resource update cancelled │ Resource update cancelled │ Received response status [FAILED] from custom resource. Message returned: Error: 3 validation errors for OpenSearchDomain │ domainInfo │ field required (type=value_error.missing) │ importMethod │ field required (type=value_error.missing) │ resources │ field required (type=value_error.missing) │ │ Logs: /aws/lambda/centralized-logging-stack-v2-CRInitConfig61C535DA-hFDu9QONvkgP │ │ at invokeUserFunction (/var/task/framework.js:2:6) │ at process.processTicksAndRejections (node:internal/process/task_queues:105:5) │ at async onEvent (/var/task/framework.js:1:369) │ at async Runtime.handler (/var/task/cfn-response.js:1:1837) (RequestId: 43a7e065-356b-4316-8981-a536dbd38c9f)

To Reproduce

CLO Stack is deployed with v2.2.2
Upgrade the Stack using a template version v2.3.0 or higher.

Possible Solution

No response

Additional context

No response

Solution Version

v2.2.2

AWS Region. e.g., us-east-1

us-east-1

Other information / Screenshots

No response

[jangidms]

Hi @yanapoz
Thanks for reporting this issue.
We'll review this and revert back

[jangidms]

Hi @yanapoz

The upgrade is failing due to schema validation errors in the OpenSearch domain record from OpenSearchDomain DynamoDB table. The existing domain record is missing required fields: domainInfo, importMethod, and resources.
Could you confirm how this domain was imported ?

Resolution:

• If no active pipelines are using this domain, remove the domain record from the Centralized Logging console and DynamoDB table. You can re-import the Domain after upgrade.
• If you have active pipeline using this domain, manually update the DynamoDB record to add the missing fields

[yanapoz]

Hi @jangidms
I manually added the missing fields to the DynamoDB, and it worked! Thanks!

[yanapoz]

I can't create any pipelines. Could you clarify what should go in these fields? Maybe I'm missing something.

Fail to connect to selected OpenSearch domain due to: Lack of VPC Peering Connection between solution's VPC and domain's VPC.,Domain's VPC route table misses rule to route traffic to solution's VPC.,Solution's VPC route table misses rule to route traffic to domain's VPC. Please update OpenSearch domain setting or select another domain.

[jangidms]

A VPC peering connection is required between the solution VPC and OpenSearch VPC. The solution automatically creates this connection when importing domains using Automatic mode.
For this existing domain, you can follow the VPC peering instructions mentioned in the Implementation guide.

[yanapoz]

@jangidms It's already in place. After adding the fields to DynamoDB, the connection to domain fails.

[jangidms]

@yanapoz Please verify the VPC route tables. From the error, it seems the route is missing.

[jangidms]

Hi @yanapoz
Are you still experiencing this issue?