Allow timeout during trained model download process

[dan-rubinstein]

Description

We currently allow users to provide a timeout during inference endpoint creation and when performing an inference request. When creating an endpoint requiring a trained model deployment to be started or performing an inference request to a default endpoint that does not have a trained model deployment started we will download the model before starting a deployment if it has not been previously downloaded. During this download process, we do not currently timeout if the user's requested timeout is exceeded and instead download the model fully and then timeout during the model deployment starting process. This change fixes this poor experience and allows the system to timeout during the model download. If this timeout occurs, we should still retain the experience that the model will be downloaded and a trained model deployment will be started in the background so the user does not have to take any further action for the process to complete.

Testing

• Tested that locally creating an ElasticsearchInternalService endpoint with a small timeout (1 second) will throw the ModelDeploymentTimeoutException and will complete the download/deployment start asynchronously.
• Tested that calling inference on a default endpoint with no model downloaded/no trained model deployment started has the same experience as the test above.
• Should we have some QA tests or IT tests for this?
  • Discussed with Wei and he will be working on QA tests for this as part of this issue

[elasticsearchmachine]

Hi @dan-rubinstein, I've created a changelog YAML for you.

[dan-rubinstein]

@elasticmachine merge upstream

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[dan-rubinstein]

@elasticmachine merge upstream