Skip to content

Conversation

brianf-aws
Copy link
Contributor

@brianf-aws brianf-aws commented Oct 10, 2025

Description

Fix CVEs CVE-2025-55163, CVE-2025-48924

Addresses #4143

./gradlew test 

BUILD SUCCESSFUL in 2m 51s
47 actionable tasks: 47 executed

 find plugin/build/distributions/opensearch-ml-2.19.4.0-SNAPSHOT -name "*netty-codec-http*" -type f 


plugin/build/distributions/opensearch-ml-2.19.4.0-SNAPSHOT/netty-codec-http2-4.2.4.Final.jar
plugin/build/distributions/opensearch-ml-2.19.4.0-SNAPSHOT/netty-codec-http-4.2.4.Final.jar

 find plugin/build/distributions/opensearch-ml-2.19.4.0-SNAPSHOT -name "*commons-lang3*" -type f

plugin/build/distributions/opensearch-ml-2.19.4.0-SNAPSHOT/commons-lang3-3.18.0.jar

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:00 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:00 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:00 — with GitHub Actions Failure
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:00 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:00 — with GitHub Actions Failure
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:00 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:16 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:16 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:16 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:16 — with GitHub Actions Failure
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:16 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 11, 2025 00:16 — with GitHub Actions Failure
@dbwiddis
Copy link
Member

These shouldn't be necessary if you use the version catalog.

I just went through and updated all the OpenSearch CVE versions, between opensearch-project/OpenSearch#19155 and https://github.com/opensearch-project/OpenSearch/pulls?q=is%3Aopen+is%3Apr+author%3Adbwiddis+label%3ACVE most should be handled.

@brianf-aws
Copy link
Contributor Author

Taking a look again, was dragged into other tasks

@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 21, 2025 23:40 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 21, 2025 23:40 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 21, 2025 23:40 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 21, 2025 23:40 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 21, 2025 23:40 — with GitHub Actions Waiting
@brianf-aws brianf-aws requested a deployment to ml-commons-cicd-env-require-approval October 21, 2025 23:40 — with GitHub Actions Waiting
@brianf-aws
Copy link
Contributor Author

brianf-aws commented Oct 22, 2025

The mend security port is saying commlang3.version != 3.18.0

 ./gradlew :opensearch-ml-plugin:dependencies --configuration runtimeClasspath | grep commons-lang3

|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    +--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0

and if I look at the snapshot unzipped I don't see the commons-lang3 as a different version

% pwd
/Users/iflorbri/IdeaProjects/ml-commons/plugin/build/distributions/opensearch-ml-2.19.4.0-SNAPSHOT
opensearch-ml-2.19.4.0-SNAPSHOT % find ./ -name "*commons-lang3*" -type f                                                        

./commons-lang3-3.18.0.jar

@gaiksaya can you give any advice? Do you think its a problem with security Mend?

@gaiksaya
Copy link
Member

@gaiksaya can you give any advice? Do you think its a problem with security Mend?

Looks like the dependency is marked transitive in mend failure: https://github.com/opensearch-project/ml-commons/pull/4298/checks?check_run_id=53328647583
You can try to see by ./gradlew dependencies | grep commons-lang3

@brianf-aws
Copy link
Contributor Author

Thanks @gaiksaya

I see now, will make appropriate edits

 for project in opensearch-ml-algorithms opensearch-ml-client opensearch-ml-common opensearch-ml-memory opensearch-ml-plugin opensearch-ml-search-processors opensearch-ml-spi; do
  echo "=== $project ==="
  ./gradlew :$project:dependencies | grep commons-lang3
done

=== opensearch-ml-algorithms ===
|    \--- org.apache.commons:commons-lang3:3.13.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.13.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.13.0
|    \--- org.apache.commons:commons-lang3:3.13.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.13.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.13.0
|    \--- org.apache.commons:commons-lang3:3.13.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.13.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.13.0
|    \--- org.apache.commons:commons-lang3:3.13.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.13.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.13.0
=== opensearch-ml-client ===
=== opensearch-ml-common ===
|    \--- org.apache.commons:commons-lang3:3.12.0
|    \--- org.apache.commons:commons-lang3:3.12.0
|    \--- org.apache.commons:commons-lang3:3.12.0
=== opensearch-ml-memory ===
=== opensearch-ml-plugin ===
+--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0 (n)
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    +--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    +--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
=== opensearch-ml-search-processors ===
+--- org.apache.commons:commons-lang3:3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
+--- org.apache.commons:commons-lang3:3.18.0 (n)
+--- org.apache.commons:commons-lang3:3.18.0
     \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
=== opensearch-ml-spi ===

@brianf-aws
Copy link
Contributor Author

Current commit changes show common lang dependencies to use lang3 3.18.0

 for project in opensearch-ml-algorithms opensearch-ml-client opensearch-ml-common opensearch-ml-memory opensearch-ml-plugin opensearch-ml-search-processors opensearch-ml-spi; do
  echo "=== $project ==="
  ./gradlew :$project:dependencies | grep commons-lang3
done
=== opensearch-ml-algorithms ===
|    \--- org.apache.commons:commons-lang3:3.13.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.13.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.13.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.13.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
=== opensearch-ml-client ===
=== opensearch-ml-common ===
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
=== opensearch-ml-memory ===
=== opensearch-ml-plugin ===
+--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0 (n)
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    +--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    |    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
|    +--- org.apache.commons:commons-lang3:3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
=== opensearch-ml-search-processors ===
+--- org.apache.commons:commons-lang3:3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
+--- org.apache.commons:commons-lang3:3.18.0 (n)
+--- org.apache.commons:commons-lang3:3.18.0
     \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
+--- org.apache.commons:commons-lang3:3.18.0
|    \--- org.apache.commons:commons-lang3:3.12.0 -> 3.18.0
=== opensearch-ml-spi ===

will monitor the mend

@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 22, 2025 21:40 — with GitHub Actions Failure
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 22, 2025 21:40 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 22, 2025 21:40 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 22, 2025 21:40 — with GitHub Actions Error
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 22, 2025 21:40 — with GitHub Actions Failure
@brianf-aws brianf-aws had a problem deploying to ml-commons-cicd-env-require-approval October 22, 2025 21:40 — with GitHub Actions Error
@brianf-aws
Copy link
Contributor Author

[Mend Security Check] has passed now

@gaiksaya
Copy link
Member

will monitor the mend

Looks like it went through @brianf-aws 🎉

@brianf-aws
Copy link
Contributor Author

Observing multiple errors on the Linux 11 build.

We should merge this first and then drill down on the build problems

@brianf-aws
Copy link
Contributor Author

Errors present in the logs may be correlated with the CVE change

 ERROR][o.o.m.e.a.a.MLChatAgentRunner] [integTest-0] Failed to run chat agent
?  org.opensearch.OpenSearchStatusException: Error communicating with remote model: java.lang.IllegalStateException: unexpected message type: LastHttpContent$1, state: 0
?  	at org.opensearch.ml.engine.algorithms.remote.MLSdkAsyncHttpResponseHandler.onError(MLSdkAsyncHttpResponseHandler.java:108) [opensearch-ml-algorithms-2.19.4.0-SNAPSHOT.jar:?]
?  	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:308) [netty-nio-client-2.30.18.jar:?]
?  	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$writeRequest$13(NettyRequestExecutor.java:262) [netty-nio-client-2.30.18.jar:?]
?  	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:603) [netty-common-4.2.5.Final.jar:4.2.5.Final]
?  	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:570) [netty-common-4.2.5.Final.jar:4.2.5.Final]
?  	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:505) [netty-common-4.2.5.Final.jar:4.2.5.Final]
?  	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:649) [netty-common-4.2.5.Final.jar:4.2.5.Final]
?  	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:642) [netty-common-4.2.5.Final.jar:4.2.5.Final]
?  	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:131) [netty-common-4.2.5.Final.jar:4.2.5.Final]
?  	at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) [netty-common-4.2.5.Final.jar:4.2.5.Final]
?  	at io.netty.channel.AbstractChannelHandlerContext.notifyOutboundHandlerException(AbstractChannelHandlerContext.java:875) [netty-transport-4.2.5.Final.jar:4.2.5.Final]

@mingshl
Copy link
Collaborator

mingshl commented Oct 22, 2025

merge to unblock, would you reproduce the issue issue locally?

@mingshl
Copy link
Collaborator

mingshl commented Oct 22, 2025

RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_ERROR
    REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests 'org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent' -Dtests.seed=49FCB213B6777104 -Dtests.security.manager=false -Dtests.locale=bg-BG -Dtests.timezone=Asia/Baku -Druntime.java=11


RestConnectorToolIT > testConnectorToolInFlowAgent FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:36543/], URI [/_plugins/_ml/agents/KYrrDZoBQ7uGzlOjmZkQ/_execute], status line [HTTP/1.1 404 Not Found]
    {"status":404,"error":{"type":"OpenSearchStatusException","reason":"System Error","details":"Error from remote service: {\"message\":\"This model version has reached the end of its life. Please refer to the AWS documentation for more details.\"}"}}
        at __randomizedtesting.SeedInfo.seed([49FCB213B6777104:6C57830229BC50CC]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:371)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:186)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:159)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:148)
        at app//org.opensearch.ml.rest.RestBaseAgentToolsIT.executeAgent(RestBaseAgentToolsIT.java:83)
        at app//org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent(RestConnectorToolIT.java:139)

RestConnectorToolIT STANDARD_ERROR
    NOTE: leaving temporary files on disk at: /__w/ml-commons/ml-commons/plugin/build/testrun/integTest/temp/org.opensearch.ml.rest.RestConnectorToolIT_49FCB213B6777104-001

    NOTE: test params are: codec=Lucene912, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=bg-BG, timezone=Asia/Baku

    NOTE: Linux 6.11.0-1018-azure amd64/Azul Systems, Inc. 11.0.29 (64-bit)/cpus=4,threads=1,free=463068672,total=536870912

    NOTE: All tests run in this JVM: [MLModelAutoReDeployerIT, RestBedRockInferenceIT, RestConnectorToolIT]

  1> [2025-10-23T01:54:54,553][INFO ][o.o.m.r.RestConnectorToolIT] [testConnectorToolInFlowAgent_WrongAction] before test
  1> [2025-10-23T01:54:54,556][INFO ][o.o.m.r.RestConnectorToolIT] [testConnectorToolInFlowAgent_WrongAction] initializing REST clients against [http://[::1]:37717, http://127.0.0.1:36543]/
  1> [2025-10-23T01:55:15,052][INFO ][o.o.m.r.RestConnectorToolIT] [testConnectorToolInFlowAgent_WrongAction] after test
  1> [2025-10-23T01:55:15,061][INFO ][o.o.m.r.RestConnectorToolIT] [testConnectorToolInFlowAgent] before test
  1> [2025-10-23T01:55:35,748][INFO ][o.o.m.r.RestConnectorToolIT] [testConnectorToolInFlowAgent] after test

REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests 'org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent' -Dtests.seed=49FCB213B6777104 -Dtests.security.manager=false -Dtests.locale=bg-BG -Dtests.timezone=Asia/Baku -Druntime.java=11

Suite: Test class org.opensearch.ml.rest.RestConnectorToolIT
  2> X 23, 2025 1:55:14 AM org.opensearch.client.RestClient logResponse
  2> WARNING: request [DELETE http://127.0.0.1:36543/.plugins-ml-agent] returned 1 warnings: [299 OpenSearch-2.19.4-SNAPSHOT-404107e4c6db6b883f067e465fd66b954cc0bf49 "this request accesses system indices: [.plugins-ml-agent], but in a future major version, direct access to system indices will be prevented by default"]
  2> REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests 'org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent' -Dtests.seed=49FCB213B6777104 -Dtests.security.manager=false -Dtests.locale=bg-BG -Dtests.timezone=Asia/Baku -Druntime.java=11
  2> org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:36543/], URI [/_plugins/_ml/agents/KYrrDZoBQ7uGzlOjmZkQ/_execute], status line [HTTP/1.1 404 Not Found]
    {"status":404,"error":{"type":"OpenSearchStatusException","reason":"System Error","details":"Error from remote service: {\"message\":\"This model version has reached the end of its life. Please refer to the AWS documentation for more details.\"}"}}
        at __randomizedtesting.SeedInfo.seed([49FCB213B6777104:6C57830229BC50CC]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:479)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:371)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:186)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:159)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:148)
        at app//org.opensearch.ml.rest.RestBaseAgentToolsIT.executeAgent(RestBaseAgentToolsIT.java:83)
        at app//org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent(RestConnectorToolIT.java:139)
  2> NOTE: leaving temporary files on disk at: /__w/ml-commons/ml-commons/plugin/build/testrun/integTest/temp/org.opensearch.ml.rest.RestConnectorToolIT_49FCB213B6777104-001
  2> NOTE: test params are: codec=Lucene912, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=bg-BG, timezone=Asia/Baku
  2> NOTE: Linux 6.11.0-1018-azure amd64/Azul Systems, Inc. 11.0.29 (64-bit)/cpus=4,threads=1,free=463068672,total=536870912

@mingshl mingshl merged commit 7a07243 into opensearch-project:create-pull-request/2.19.4-SNAPSHOT Oct 22, 2025
6 of 12 checks passed
@brianf-aws
Copy link
Contributor Author

@mingshl I'm certain the Netty bump caused a change on the code to not work. Currently trying to apply a previous commit from mainline but facing issues as the versions drift from mainline and 2.19

#4175

image

@brianf-aws
Copy link
Contributor Author

merge to unblock, would you reproduce the issue issue locally?

Its a IT that uses an agent. Will need to use API keys to observe the test. It will fail in the snapshot PR

#4143

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants