Gracefully exit when terminated mid deploy / destroy #3758

shreyas-goenka · 2025-10-14T11:43:41Z

Changes

Adds graceful signal handling (SIGINT, SIGTERM, SIGHUP, SIGQUIT) during bundle deployment and destruction. When interrupted, the CLI waits for the current operation to complete, releases the deployment lock, and exits with the appropriate exit code.

Makes lock release idempotent to handle race conditions when multiple cleanup paths attempt to release the lock.

Tests

Added acceptance test for signal during lock release
Added unit tests for lock release and unlock idempotency

bundle/phases/deploy.go

eng-dev-ecosystem-bot · 2025-10-14T12:04:55Z

Run: 19141414460

	Env	🪲BUG	❌FAIL	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip
🪲	aws linux	2			7	1	1	347	596
🪲	aws windows	2			2	1	1	353	595
🪲	aws-ucws linux	2				1	1	477	491
🪲	aws-ucws windows	2				1	1	478	490
🪲	azure linux	2			4	1	1	350	595
🪲	azure windows	2				1	1	355	594
🪲	azure-ucws linux	2				1	1	473	490
🪲	azure-ucws windows	2				1	1	474	489
🪲	gcp linux	2	21	1			1	332	597
🪲	gcp windows	2	26	1			1	328	596

49 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	🟨K	🟨K
TestAccept/bundle/deploy/files/no-snapshot-sync	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/destroy/jobs-and-pipeline/DATABRICKS_BUNDLE_ENGINE=direct	✅p	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p
TestAccept/bundle/destroy/jobs-and-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p
TestAccept/bundle/local_state_staleness	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/local_state_staleness/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/clusters/deploy/data_security_mode	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
TestAccept/bundle/resources/experiments/basic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/experiments/basic/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/pipelines/allow-duplicate-names	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/pipelines/allow-duplicate-names/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/pipelines/auto-approve	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/pipelines/lakeflow-pipeline	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/pipelines/lakeflow-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/resources/pipelines/update	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/resources/pipelines/update/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/pipelines/update/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/run/app-with-job	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
TestAccept/bundle/templates/default-python/combinations/classic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=no	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=no/PY=yes	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=yes/PY=no	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=no/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=no	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.10	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.11	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.12	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.10	🔄f	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.11	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestImportDir	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F	✅p
TestLock	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B
TestLockUnlockWithoutAllowsLockFileNotExist	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B

acceptance/bundle/deploy/signal-cleanup/script

shreyas-goenka · 2025-11-06T15:37:25Z

bundle/deploy/lock/release.go

 )

-type Goal string
+type release struct{}


We don't need goal anymore since lock.Release is idempotent.

shreyas-goenka temporarily deployed to test-trigger-is October 14, 2025 11:43 — with GitHub Actions Inactive

pietern reviewed Oct 14, 2025

View reviewed changes

bundle/phases/deploy.go Show resolved Hide resolved

denik reviewed Oct 14, 2025

View reviewed changes

acceptance/bundle/deploy/signal-cleanup/script Show resolved Hide resolved

shreyas-goenka added 2 commits November 5, 2025 22:15

Gracefully exit when terminated mid deploy / destroy

48bd034

address comments

3023134

shreyas-goenka force-pushed the catch-signals branch from 2db9670 to 3023134 Compare November 6, 2025 11:56

shreyas-goenka temporarily deployed to test-trigger-is November 6, 2025 11:56 — with GitHub Actions Inactive

update with latest

9d2fb04

shreyas-goenka temporarily deployed to test-trigger-is November 6, 2025 14:20 — with GitHub Actions Inactive

shreyas-goenka added 2 commits November 6, 2025 16:15

Merge remote-tracking branch 'origin' into catch-signals

5610096

better test

cec6a88

shreyas-goenka temporarily deployed to test-trigger-is November 6, 2025 15:28 — with GitHub Actions Inactive

update tests

5932884

shreyas-goenka temporarily deployed to test-trigger-is November 6, 2025 15:36 — with GitHub Actions Inactive

shreyas-goenka commented Nov 6, 2025

View reviewed changes

shreyas-goenka requested review from denik and pietern November 6, 2025 15:44

shreyas-goenka marked this pull request as ready for review November 6, 2025 15:44

shreyas-goenka requested review from andrewnester and anton-107 as code owners November 6, 2025 15:44

minor rename

42f03eb

shreyas-goenka temporarily deployed to test-trigger-is November 6, 2025 15:48 — with GitHub Actions Inactive

shreyas-goenka marked this pull request as draft November 6, 2025 15:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Gracefully exit when terminated mid deploy / destroy #3758

Gracefully exit when terminated mid deploy / destroy #3758

Uh oh!

shreyas-goenka commented Oct 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Oct 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

shreyas-goenka Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Gracefully exit when terminated mid deploy / destroy #3758

Are you sure you want to change the base?

Gracefully exit when terminated mid deploy / destroy #3758

Uh oh!

Conversation

shreyas-goenka commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

shreyas-goenka Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shreyas-goenka commented Oct 14, 2025 •

edited

Loading

eng-dev-ecosystem-bot commented Oct 14, 2025 •

edited

Loading