Skip to content

Conversation

@brianf-aws
Copy link
Contributor

@brianf-aws brianf-aws commented Oct 22, 2025

Description

Upon making a CVE fix #4298 . which involved bumping netty, there was a netty exception.

? ERROR][o.o.m.e.a.a.MLAgentExecutor] [integTest-0] Failed to run conversational agent
?  org.opensearch.OpenSearchStatusException: Error communicating with remote model: java.lang.IllegalStateException: unexpected message type: LastHttpContent$1, state: 0
?  	at org.opensearch.ml.engine.algorithms.remote.MLSdkAsyncHttpResponseHandler.onError(MLSdkAsyncHttpResponseHandler.java:108) [opensearch-ml-algorithms-2.19.4.0-SNAPSHOT.jar:?]
?  	at software.amazon.awssd

There exists a cherry pick which bumped netty on mainline but required code changes
#4175 . The issue here is that the version catalog in that mainline commit was not synced from core 2.19.4 . Making the change non-trivial

image

Reviewer objectives

  • To ensure at least 1 CI passes that no errors are related to netty
  • To review commits from use mainline versions.aws via hardcode and above
  • To note that the lack of version catalog is because core's 2.19.4 version catalog is 11 months old!

Related Issues

Resolves the snapshot PR #4143

Next steps

  • backport the Core version catalog to 2.19.4 [OS core]
  • Update 2.19.4 to have proper version catalog [ML-Commons]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@brianf-aws brianf-aws changed the title 2.19.4 CVE fix Address Netty failure at Agent Execute runtime Oct 22, 2025
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 22, 2025 23:51 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 22, 2025 23:51 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 22, 2025 23:51 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 22, 2025 23:51 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 22, 2025 23:51 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 22, 2025 23:51 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 23, 2025 00:04 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 23, 2025 00:04 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 23, 2025 00:04 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 23, 2025 00:04 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 23, 2025 00:04 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 23, 2025 00:04 — with GitHub Actions Waiting
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 23, 2025 00:06 — with GitHub Actions Failure
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 23, 2025 00:06 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 23, 2025 00:06 — with GitHub Actions Failure
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 23, 2025 00:06 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 23, 2025 00:06 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 23, 2025 21:55 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 23, 2025 21:55 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 23, 2025 21:55 — with GitHub Actions Failure
@brianf-aws brianf-aws changed the title Address Netty failure at Agent Execute runtime fix CVE-2025-55163, CVE-2025-48924, Oct 23, 2025
@brianf-aws brianf-aws changed the title fix CVE-2025-55163, CVE-2025-48924, fix CVE-2025-55163, CVE-2025-48924, CVE-2025-58057 Oct 23, 2025
@ylwu-amzn
Copy link
Collaborator

RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_ERROR
    REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests 'org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent' -Dtests.seed=E7A4785AECC5E01F -Dtests.security.manager=false -Dtests.locale=zh-HK -Dtests.timezone=America/Argentina/Salta -Druntime.java=17

RestConnectorToolIT > testConnectorToolInFlowAgent FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://[::1]:33257], URI [/_plugins/_ml/agents/9WMdE5oB5K9gs9y6yzRh/_execute], status line [HTTP/1.1 404 Not Found]
    {"status":404,"error":{"type":"OpenSearchStatusException","reason":"System Error","details":"Error from remote service: {\"message\":\"This model version has reached the end of its life. Please refer to the AWS documentation for more details.\"}"}}

@brianf-aws seems we need to user another model in the IT.

@brianf-aws
Copy link
Contributor Author

RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_ERROR
    REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests 'org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent' -Dtests.seed=E7A4785AECC5E01F -Dtests.security.manager=false -Dtests.locale=zh-HK -Dtests.timezone=America/Argentina/Salta -Druntime.java=17

RestConnectorToolIT > testConnectorToolInFlowAgent FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://[::1]:33257], URI [/_plugins/_ml/agents/9WMdE5oB5K9gs9y6yzRh/_execute], status line [HTTP/1.1 404 Not Found]
    {"status":404,"error":{"type":"OpenSearchStatusException","reason":"System Error","details":"Error from remote service: {\"message\":\"This model version has reached the end of its life. Please refer to the AWS documentation for more details.\"}"}}

@brianf-aws seems we need to user another model in the IT.

Also blocked by remote metadata SDK tracking change with this opensearch-project/opensearch-remote-metadata-sdk#276

@dhrubo-os
Copy link
Collaborator

RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_ERROR
    REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests 'org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent' -Dtests.seed=E7A4785AECC5E01F -Dtests.security.manager=false -Dtests.locale=zh-HK -Dtests.timezone=America/Argentina/Salta -Druntime.java=17

RestConnectorToolIT > testConnectorToolInFlowAgent FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://[::1]:33257], URI [/_plugins/_ml/agents/9WMdE5oB5K9gs9y6yzRh/_execute], status line [HTTP/1.1 404 Not Found]
    {"status":404,"error":{"type":"OpenSearchStatusException","reason":"System Error","details":"Error from remote service: {\"message\":\"This model version has reached the end of its life. Please refer to the AWS documentation for more details.\"}"}}

@brianf-aws seems we need to user another model in the IT.

Also blocked by remote metadata SDK tracking change with this opensearch-project/opensearch-remote-metadata-sdk#276

I think somebody solved it already, we just need to find that PR and backport.

@brianf-aws
Copy link
Contributor Author

Will apply the netty version catalog , and find the missing PR

@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:18 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:18 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:18 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:18 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:18 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:18 — with GitHub Actions Waiting
opensearch-ci-bot and others added 8 commits October 23, 2025 21:37
Signed-off-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Brian Flores <[email protected]>
* address commons-lang3 CVE-2025-48924

Signed-off-by: Brian Flores <[email protected]>

* pin netty to 4.2.5.Final version address  CVE-2025-55163

Signed-off-by: Brian Flores <[email protected]>

* force all subProjects to use updated common-lang3 version

Signed-off-by: Brian Flores <[email protected]>

---------

Signed-off-by: Brian Flores <[email protected]>
…earch-project#4175)

* Move HttpClientFactory to common to expose to other componenets

Signed-off-by: zane-neo <[email protected]>

* optimize code for better maintainability

Signed-off-by: zane-neo <[email protected]>

* Optimize code and increase UT coverage

Signed-off-by: zane-neo <[email protected]>

* Address comments

Signed-off-by: zane-neo <[email protected]>

* Use amazon aws version from opensearch core

Signed-off-by: zane-neo <[email protected]>

* address comments

Signed-off-by: zane-neo <[email protected]>

---------

Signed-off-by: zane-neo <[email protected]>
Signed-off-by: Brian Flores <[email protected]>
Signed-off-by: Brian Flores <[email protected]>
Signed-off-by: Brian Flores <[email protected]>
Signed-off-by: Brian Flores <[email protected]>
* fix model it by replace claude v1/v2

Signed-off-by: xinyual <[email protected]>

* remove useless change

Signed-off-by: xinyual <[email protected]>

---------

Signed-off-by: xinyual <[email protected]>
Signed-off-by: Brian Flores <[email protected]>
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:39 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:39 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:39 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:39 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:39 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 24, 2025 04:39 — with GitHub Actions Waiting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants