Skip to content

[ML] Partial fix for deployment disappearing #137216

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jonathan-buttner wants to merge 3 commits into from
Closed

[ML] Partial fix for deployment disappearing #137216

jonathan-buttner wants to merge 3 commits into from

Conversation

@jonathan-buttner
Copy link
Contributor

@jonathan-buttner jonathan-buttner commented Oct 27, 2025 •
edited
Loading

WIP

This PR is just to show the changes I made to be able to test the issue here: #137134

To make the reproduction faster I temporarily changed the code to allow the times to be shorter:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.ml.trained_models.adaptive_allocations.scale_to_zero_time": "10s",
    "xpack.ml.trained_models.adaptive_allocations.scale_up_cooldown_time": "10s",
    "logger.org.elasticsearch.xpack.ml.inference.assignment": "DEBUG"
  }
}

Then we can follow the steps in the issue to reproduce, which are:

  1. Create deployment via creating inference endpoint
PUT _inference/rerank/mytest-old
{
    "service": "elasticsearch",
    "service_settings": {
        "num_threads": 1,
        "model_id": ".rerank-v1",
        "adaptive_allocations": {
            "enabled": true,
            "min_number_of_allocations": 0,
            "max_number_of_allocations": 2
        }
    }
}
  1. Wait for mytest-old to scale to zero ~10 seconds
GET _ml/trained_models/_stats
  1. Create a new deployment via inference endpoint, mytest-old should still exist, but it will have an allocation which is not intended.
PUT _inference/rerank/mytest-new3
{
    "service": "elasticsearch",
    "service_settings": {
        "num_threads": 1,
        "model_id": ".rerank-v1",
        "adaptive_allocations": {
            "enabled": true,
            "min_number_of_allocations": 0,
            "max_number_of_allocations": 2
        }
    }
}

GET _ml/trained_models/_stats

@jonathan-buttner jonathan-buttner added >bug :ml Machine learning Team:ML Meta label for the ML team v9.3.0 labels Oct 27, 2025
@jonathan-buttner jonathan-buttner mentioned this pull request Oct 27, 2025
Closed
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Oct 27, 2025

Hi @jonathan-buttner, I've created a changelog YAML for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

>bug :ml Machine learning Team:ML Meta label for the ML team v9.3.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jonathan-buttner @elasticsearchmachine