Test ML model server #120270

jan-elastic · 2025-01-16T11:05:38Z

Some integration tests attempt to download ML models of >100MB from ml-models.elastic.co. This may fail for various reasons, leading to these tests being muted.

In order to fix this, this PR spins up a simple HTTP server on localhost, which serves tiny versions of these models, and uses that server in the integration tests.

In the process, two bugs are also fixed:

downloading models that are smaller than a few MB
dynamically changing the model repo URL

Closes: #113950 #113983 #114023 #114239 #114913 #115361 #116140 #116142

elasticsearchmachine · 2025-01-20T11:08:43Z

Pinging @elastic/ml-core (Team:ML)

jan-elastic · 2025-01-20T15:41:12Z

x-pack/plugin/inference/qa/inference-service-tests/build.gradle

 apply plugin: 'elasticsearch.internal-java-rest-test'

 dependencies {
+  javaRestTestImplementation project(path: xpackModule('core'))


Need to get XPackSettings.ML_NATIVE_CODE_PLATFORMS into the model server

wwang500

Code looks good, but I will let Dave give the final LGTM as my knowledge about integration test is limited here.

Just one question:

are those unmuted integration tests only running in locally, right? If those tests will run against a remote cluster, like in MKI or ECH, they will fail I guess.

jan-elastic · 2025-01-21T08:05:47Z

are those unmuted integration tests only running in locally, right? If those tests will run against a remote cluster, like in MKI or ECH, they will fail I guess.

Yes, this is running locally or on Bulidkite, not vs remote clusters etc.

davidkyle

LGTM

...ice-tests/src/javaRestTest/java/org/elasticsearch/xpack/inference/InferenceBaseRestTest.java

...e-loader/src/main/java/org/elasticsearch/xpack/ml/packageloader/action/ModelLoaderUtils.java

elasticsearchmachine · 2025-01-22T08:56:31Z

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts
❌	8.16	Commit could not be cherrypicked due to conflicts
❌	8.17	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 120270

* Fix model downloading for very small models. * Test MlModelServer * Tiny ELSER * unmute TextEmbeddingCrudIT and DefaultEndPointsIT * update ELSER * Improve MlModelServer * tiny E5 * more logging * improved E5 model * tiny reranker * scan for ports * [CI] Auto commit changes from spotless * Serve default models when optimized model is requested * @ClassRule * polish code * Respect dynamic setting ML model repo * fix metadata for optimized models * improve logging --------- Co-authored-by: elasticsearchmachine <[email protected]>

* Test ML model server (#120270) * Fix model downloading for very small models. * Test MlModelServer * Tiny ELSER * unmute TextEmbeddingCrudIT and DefaultEndPointsIT * update ELSER * Improve MlModelServer * tiny E5 * more logging * improved E5 model * tiny reranker * scan for ports * [CI] Auto commit changes from spotless * Serve default models when optimized model is requested * @ClassRule * polish code * Respect dynamic setting ML model repo * fix metadata for optimized models * improve logging --------- Co-authored-by: elasticsearchmachine <[email protected]> * backport HttpHeaderParser --------- Co-authored-by: elasticsearchmachine <[email protected]>

* Test ML model server (#120270) * Fix model downloading for very small models. * Test MlModelServer * Tiny ELSER * unmute TextEmbeddingCrudIT and DefaultEndPointsIT * update ELSER * Improve MlModelServer * tiny E5 * more logging * improved E5 model * tiny reranker * scan for ports * [CI] Auto commit changes from spotless * Serve default models when optimized model is requested * @ClassRule * polish code * Respect dynamic setting ML model repo * fix metadata for optimized models * improve logging --------- Co-authored-by: elasticsearchmachine <[email protected]> * backport HttpHeaderParser * Fix stripping platform --------- Co-authored-by: elasticsearchmachine <[email protected]>

jan-elastic added >test-failure Triaged test failures from CI :ml Machine learning Team:ML Meta label for the ML team v9.0.0 v8.18.0 labels Jan 16, 2025

jan-elastic requested a review from davidkyle January 16, 2025 11:05

jan-elastic marked this pull request as draft January 16, 2025 11:05

jan-elastic force-pushed the test-ml-model-server branch 5 times, most recently from af8f5d2 to 6f54f4f Compare January 20, 2025 11:05

jan-elastic marked this pull request as ready for review January 20, 2025 11:08

elasticsearchmachine added the needs:risk Requires assignment of a risk label (low, medium, blocker) label Jan 20, 2025

jan-elastic requested a review from wwang500 January 20, 2025 14:23

jan-elastic commented Jan 20, 2025

View reviewed changes

wwang500 reviewed Jan 21, 2025

View reviewed changes

davidkyle added >test Issues or PRs that are addressing/adding tests and removed needs:risk Requires assignment of a risk label (low, medium, blocker) >test-failure Triaged test failures from CI labels Jan 21, 2025

davidkyle approved these changes Jan 21, 2025

View reviewed changes

...ice-tests/src/javaRestTest/java/org/elasticsearch/xpack/inference/InferenceBaseRestTest.java Outdated Show resolved Hide resolved

...e-loader/src/main/java/org/elasticsearch/xpack/ml/packageloader/action/ModelLoaderUtils.java Show resolved Hide resolved

jan-elastic added 7 commits January 22, 2025 08:47

Fix model downloading for very small models.

0651731

Test MlModelServer

b1497fb

Tiny ELSER

572ceb0

unmute TextEmbeddingCrudIT and DefaultEndPointsIT

1c4dd60

update ELSER

2c82aae

Improve MlModelServer

07e9046

tiny E5

bdf776b

jan-elastic and others added 10 commits January 22, 2025 08:47

improved E5 model

2408654

tiny reranker

d612dd2

scan for ports

e092b7f

[CI] Auto commit changes from spotless

49d97dd

Serve default models when optimized model is requested

16e1424

@ClassRule

08517f2

polish code

3cc62d3

Respect dynamic setting ML model repo

10cb533

fix metadata for optimized models

ca60b55

improve logging

559b99f

jan-elastic force-pushed the test-ml-model-server branch from bd71425 to 559b99f Compare January 22, 2025 07:53

jan-elastic added auto-backport Automatically create backport pull requests when merged v8.16.3 v8.17.2 labels Jan 22, 2025

jan-elastic merged commit 6fd99c6 into main Jan 22, 2025
17 checks passed

jan-elastic deleted the test-ml-model-server branch January 22, 2025 08:55

elasticsearchmachine added the backport pending label Jan 22, 2025

jan-elastic mentioned this pull request Jan 22, 2025

[8.x] Test ML model server (#120270) #120586

Merged

jan-elastic mentioned this pull request Jan 22, 2025

[8.17] Test ML model server (#120270) #120588

Merged

jan-elastic mentioned this pull request Jan 22, 2025

[8.16] Test ML model server (#120270) #120589

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Test ML model server #120270

Test ML model server #120270

Uh oh!

jan-elastic commented Jan 16, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jan 20, 2025

Uh oh!

jan-elastic Jan 20, 2025

Uh oh!

wwang500 left a comment

Uh oh!

jan-elastic commented Jan 21, 2025

Uh oh!

davidkyle left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Test ML model server #120270

Test ML model server #120270

Uh oh!

Conversation

jan-elastic commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 20, 2025

Uh oh!

jan-elastic Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

wwang500 left a comment

Choose a reason for hiding this comment

Uh oh!

jan-elastic commented Jan 21, 2025

Uh oh!

davidkyle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 22, 2025

💔 Backport failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jan-elastic commented Jan 16, 2025 •

edited

Loading