Skip to content

Commit e57befa

Browse files
shreyas-goenkashreyas-goenka
authored
Granular updates for model serving endpoints (#3988)
## Changes This PR implements granular updates for model serving endpoints. Instead of always updating the entire endpoint configuration, the CLI now only sends updates for the specific fields that have changed. This are: - AI Gateway configuration - Route configuration - Email notifications - Tags ## Why Model serving endpoint updates are expensive operations. By sending only the changed fields, we reduce the scope of updates and improve deployment performance. We also don't have guarentees that these API calls are safe to do in parallel. This matches the TF implementation: https://github.com/databricks/terraform-provider-databricks/blob/b0a2a1c6a1688498fd6a00c64003ef4948da21e8/serving/resource_model_serving.go#L366 ## Tests Added comprehensive acceptance tests for various update scenarios: - AI Gateway updates - Route config updates - Email notification updates - Tag updates - Combined updates (multiple fields at once) --- **Note:** This PR depends on #3995 and should be merged after that PR is merged and this branch is rebased. **Note:** These tests are local only because model serving endpoints take a long time (~30 minutes) to spin up and can be flaky. We can confirm though that the TF and DABs behavior matches.
1 parent 312bd27 commit e57befa

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+1348
-19
lines changed

‎acceptance/bundle/resources/model_serving_endpoints/running-endpoint/output.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,10 @@ Deployment complete!
3333
"creator": "[USERNAME]"
3434
}
3535

36+
=== verify there's no persistent drift
37+
>>> [CLI] bundle plan
38+
Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
39+
3640
>>> [CLI] bundle destroy --auto-approve
3741
The following resources will be deleted:
3842
delete resources.model_serving_endpoints.my_endpoint

‎acceptance/bundle/resources/model_serving_endpoints/running-endpoint/script‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,8 @@ trace $CLI bundle deploy
1414
trace print_requests.py //serving-endpoints
1515

1616
endpoint_name=$($CLI bundle summary -o json | jq -r '.resources.model_serving_endpoints.my_endpoint.name')
17+
endpoint_id=$($CLI bundle summary -o json | jq -r '.resources.model_serving_endpoints.my_endpoint.id')
1718
trace $CLI serving-endpoints get $endpoint_name | jq '{name, creator}'
19+
20+
title "verify there's no persistent drift"
21+
trace $CLI bundle plan

‎acceptance/bundle/resources/model_serving_endpoints/update/ai-gateway/databricks.yml.tmpl‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
bundle:
2+
name: test-mse-update-ai-gateway-$UNIQUE_NAME
3+
4+
workspace:
5+
root_path: ~/.bundle/$UNIQUE_NAME
6+
7+
resources:
8+
model_serving_endpoints:
9+
test_endpoint:
10+
name: test-endpoint-$UNIQUE_NAME
11+
config:
12+
served_entities:
13+
- name: prod
14+
external_model:
15+
name: gpt-4o-mini
16+
provider: openai
17+
task: llm/v1/chat
18+
openai_config:
19+
openai_api_key: "{{secrets/test-scope/openai-key}}"
20+
ai_gateway:
21+
inference_table_config:
22+
catalog_name: "first-inference-catalog"

‎acceptance/bundle/resources/model_serving_endpoints/update/ai-gateway/out.plan.direct.json‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
{
2+
"plan": {
3+
"resources.model_serving_endpoints.test_endpoint": {
4+
"action": "update",
5+
"new_state": {
6+
"value": {
7+
"ai_gateway": {
8+
"inference_table_config": {
9+
"catalog_name": "second-inference-catalog"
10+
}
11+
},
12+
"config": {
13+
"served_entities": [
14+
{
15+
"external_model": {
16+
"name": "gpt-4o-mini",
17+
"openai_config": {
18+
"openai_api_key": "{{secrets/test-scope/openai-key}}"
19+
},
20+
"provider": "openai",
21+
"task": "llm/v1/chat"
22+
},
23+
"name": "prod"
24+
}
25+
]
26+
},
27+
"name": "[ENDPOINT_ID]"
28+
}
29+
},
30+
"remote_state": {
31+
"endpoint_details": {
32+
"ai_gateway": {
33+
"inference_table_config": {
34+
"catalog_name": "first-inference-catalog"
35+
}
36+
},
37+
"config": {
38+
"served_entities": [
39+
{
40+
"external_model": {
41+
"name": "gpt-4o-mini",
42+
"openai_config": {
43+
"openai_api_key": "{{secrets/test-scope/openai-key}}"
44+
},
45+
"provider": "openai",
46+
"task": "llm/v1/chat"
47+
},
48+
"name": "prod"
49+
}
50+
]
51+
},
52+
"creator": "[USERNAME]",
53+
"id": "[UUID]",
54+
"name": "[ENDPOINT_ID]",
55+
"state": {
56+
"config_update": "NOT_UPDATING"
57+
}
58+
},
59+
"endpoint_id": "[UUID]"
60+
},
61+
"changes": {
62+
"local": {
63+
"ai_gateway.inference_table_config.catalog_name": {
64+
"action": "update"
65+
}
66+
}
67+
}
68+
}
69+
}
70+
}

‎acceptance/bundle/resources/model_serving_endpoints/update/ai-gateway/out.plan.terraform.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"plan": {
3+
"resources.model_serving_endpoints.test_endpoint": {
4+
"action": "update"
5+
}
6+
}
7+
}

‎acceptance/bundle/resources/model_serving_endpoints/update/ai-gateway/out.test.toml‎

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/resources/model_serving_endpoints/update/ai-gateway/output.txt‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
2+
>>> [CLI] bundle deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
4+
Deploying resources...
5+
Updating deployment state...
6+
Deployment complete!
7+
8+
>>> [CLI] serving-endpoints get [ENDPOINT_ID]
9+
{
10+
"inference_table_config": {
11+
"catalog_name": "first-inference-catalog"
12+
}
13+
}
14+
15+
>>> update_file.py databricks.yml catalog_name: "first-inference-catalog" catalog_name: "second-inference-catalog"
16+
17+
>>> [CLI] bundle plan -o json
18+
19+
>>> [CLI] bundle deploy
20+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
21+
Deploying resources...
22+
Updating deployment state...
23+
Deployment complete!
24+
25+
>>> print_requests.py //serving-endpoints
26+
{
27+
"method": "POST",
28+
"path": "/api/2.0/serving-endpoints",
29+
"body": {
30+
"ai_gateway": {
31+
"inference_table_config": {
32+
"catalog_name": "first-inference-catalog"
33+
}
34+
},
35+
"config": {
36+
"served_entities": [
37+
{
38+
"external_model": {
39+
"name": "gpt-4o-mini",
40+
"openai_config": {
41+
"openai_api_key": "{{secrets/test-scope/openai-key}}"
42+
},
43+
"provider": "openai",
44+
"task": "llm/v1/chat"
45+
},
46+
"name": "prod"
47+
}
48+
]
49+
},
50+
"name": "[ENDPOINT_ID]"
51+
}
52+
}
53+
{
54+
"method": "PUT",
55+
"path": "/api/2.0/serving-endpoints/[ENDPOINT_ID]/ai-gateway",
56+
"body": {
57+
"inference_table_config": {
58+
"catalog_name": "second-inference-catalog"
59+
}
60+
}
61+
}
62+
63+
>>> [CLI] serving-endpoints get [ENDPOINT_ID]
64+
{
65+
"inference_table_config": {
66+
"catalog_name": "second-inference-catalog"
67+
}
68+
}
69+
70+
>>> [CLI] bundle destroy --auto-approve
71+
The following resources will be deleted:
72+
delete resources.model_serving_endpoints.test_endpoint
73+
74+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]
75+
76+
Deleting files...
77+
Destroy complete!

‎acceptance/bundle/resources/model_serving_endpoints/update/ai-gateway/script‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
envsubst < databricks.yml.tmpl > databricks.yml
3+
4+
cleanup() {
5+
trace $CLI bundle destroy --auto-approve
6+
rm -f out.requests.txt
7+
}
8+
trap cleanup EXIT
9+
10+
trace $CLI bundle deploy
11+
12+
ENDPOINT_ID=$($CLI bundle summary -o json | jq -r '.resources.model_serving_endpoints.test_endpoint.id')
13+
echo "$ENDPOINT_ID:ENDPOINT_ID" >> ACC_REPLS
14+
trace $CLI serving-endpoints get "${ENDPOINT_ID}" | jq '.ai_gateway'
15+
16+
trace update_file.py databricks.yml 'catalog_name: "first-inference-catalog"' 'catalog_name: "second-inference-catalog"'
17+
18+
trace $CLI bundle plan -o json > out.plan.$DATABRICKS_BUNDLE_ENGINE.json
19+
trace $CLI bundle deploy
20+
21+
trace print_requests.py //serving-endpoints
22+
23+
trace $CLI serving-endpoints get "${ENDPOINT_ID}" | jq '.ai_gateway'

‎acceptance/bundle/resources/model_serving_endpoints/update/both_gateway_and_tags/databricks.yml.tmpl‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
bundle:
2+
name: test-mse-update-ai-gateway-$UNIQUE_NAME
3+
4+
workspace:
5+
root_path: ~/.bundle/$UNIQUE_NAME
6+
7+
resources:
8+
model_serving_endpoints:
9+
test_endpoint:
10+
name: test-endpoint-$UNIQUE_NAME
11+
config:
12+
served_entities:
13+
- name: prod
14+
external_model:
15+
name: gpt-4o-mini
16+
provider: openai
17+
task: llm/v1/chat
18+
openai_config:
19+
openai_api_key: "{{secrets/test-scope/openai-key}}"
20+
ai_gateway:
21+
inference_table_config:
22+
catalog_name: "first-inference-catalog"
23+
24+
tags:
25+
- key: team
26+
value: my-team-one

‎acceptance/bundle/resources/model_serving_endpoints/update/both_gateway_and_tags/out.plan.direct.json‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
{
2+
"plan": {
3+
"resources.model_serving_endpoints.test_endpoint": {
4+
"action": "update",
5+
"new_state": {
6+
"value": {
7+
"ai_gateway": {
8+
"inference_table_config": {
9+
"catalog_name": "second-inference-catalog"
10+
}
11+
},
12+
"config": {
13+
"served_entities": [
14+
{
15+
"external_model": {
16+
"name": "gpt-4o-mini",
17+
"openai_config": {
18+
"openai_api_key": "{{secrets/test-scope/openai-key}}"
19+
},
20+
"provider": "openai",
21+
"task": "llm/v1/chat"
22+
},
23+
"name": "prod"
24+
}
25+
]
26+
},
27+
"name": "[ENDPOINT_ID]",
28+
"tags": [
29+
{
30+
"key": "team",
31+
"value": "my-team-two"
32+
}
33+
]
34+
}
35+
},
36+
"remote_state": {
37+
"endpoint_details": {
38+
"ai_gateway": {
39+
"inference_table_config": {
40+
"catalog_name": "first-inference-catalog"
41+
}
42+
},
43+
"config": {
44+
"served_entities": [
45+
{
46+
"external_model": {
47+
"name": "gpt-4o-mini",
48+
"openai_config": {
49+
"openai_api_key": "{{secrets/test-scope/openai-key}}"
50+
},
51+
"provider": "openai",
52+
"task": "llm/v1/chat"
53+
},
54+
"name": "prod"
55+
}
56+
]
57+
},
58+
"creator": "[USERNAME]",
59+
"id": "[UUID]",
60+
"name": "[ENDPOINT_ID]",
61+
"state": {
62+
"config_update": "NOT_UPDATING"
63+
},
64+
"tags": [
65+
{
66+
"key": "team",
67+
"value": "my-team-one"
68+
}
69+
]
70+
},
71+
"endpoint_id": "[UUID]"
72+
},
73+
"changes": {
74+
"local": {
75+
"ai_gateway.inference_table_config.catalog_name": {
76+
"action": "update"
77+
},
78+
"tags[0].value": {
79+
"action": "update"
80+
}
81+
}
82+
}
83+
}
84+
}
85+
}

0 commit comments

Comments
 (0)