Skip to content

Store both public and private client keys in secret scopes#3775

Merged
ilia-db merged 19 commits intomainfrom
ssh-tunnel-integ-tests
Nov 10, 2025
Merged

Store both public and private client keys in secret scopes#3775
ilia-db merged 19 commits intomainfrom
ssh-tunnel-integ-tests

Conversation

@ilia-db
Copy link
Contributor

@ilia-db ilia-db commented Oct 16, 2025

Changes

Before we were passing a client key name (a secret name that contains the actual client public key) to the ssh server job. After it's started, the ssh server was only accepting one client with one specific key. This is a problem when you try to ssh to the same cluster under the same databricks user from multiple different client machines (that have different ssh keys).

Our integration tests can't work consistently because of this, and we've already got one customer report about this problem.

The main change here is that now we don't always generate key-pairs on the client, but check if the pair already exists in the secret scope and use it. And if not, we generate the pair and save both keys to the scope (and before we were only saving public key there)

@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Oct 16, 2025

Run: 19230233378

Env 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip
💚​ aws linux 1 1 356 601
💚​ aws windows 1 1 357 600
💚​ aws-ucws linux 1 1 491 492
💚​ aws-ucws windows 1 1 492 491
💚​ azure linux 1 1 356 600
💚​ azure windows 1 1 357 599
💚​ azure-ucws linux 1 1 487 491
💚​ azure-ucws windows 1 1 488 490
🔄​ gcp linux 3 1 353 602
🔄​ gcp windows 3 1 1 353 601
7 failing tests:
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 🔄​f 💚​R
TestAccept/bundle/deployment/bind/job/generate-and-bind ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
TestAccept/bundle/resources/jobs/check-metadata ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
TestAccept/bundle/resources/jobs/check-metadata/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
TestAccept/bundle/run/app-with-job 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
TestAccept/bundle/templates/default-python/integration_classic ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.10 ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f

@ilia-db ilia-db temporarily deployed to test-trigger-is October 16, 2025 12:11 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 16, 2025 12:43 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 17, 2025 08:41 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 17, 2025 13:51 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 17, 2025 13:53 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 20, 2025 10:52 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 21, 2025 09:34 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 21, 2025 09:46 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 21, 2025 09:51 — with GitHub Actions Inactive
@ilia-db ilia-db force-pushed the ssh-tunnel-integ-tests branch from 9cc7630 to da13a01 Compare October 21, 2025 14:13
@ilia-db ilia-db temporarily deployed to test-trigger-is October 21, 2025 14:13 — with GitHub Actions Inactive
@ilia-db ilia-db changed the title WIP check what's missing for ssh integ tests Cloud only acceptance tests for ssh tunnel Oct 23, 2025
@ilia-db ilia-db temporarily deployed to test-trigger-is October 23, 2025 11:05 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 24, 2025 09:39 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 24, 2025 11:00 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 24, 2025 13:11 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 24, 2025 14:18 — with GitHub Actions Inactive
@ilia-db ilia-db force-pushed the ssh-tunnel-integ-tests branch from dcbfd5b to d95fc1d Compare October 27, 2025 12:20
@ilia-db ilia-db temporarily deployed to test-trigger-is October 27, 2025 12:20 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is October 27, 2025 12:27 — with GitHub Actions Inactive
@ilia-db ilia-db changed the title Cloud only acceptance tests for ssh tunnel Pass public key name in every client ssh connection Oct 27, 2025
@ilia-db ilia-db force-pushed the ssh-tunnel-integ-tests branch from d307bfa to a988e53 Compare October 27, 2025 12:42
@ilia-db ilia-db temporarily deployed to test-trigger-is October 27, 2025 12:42 — with GitHub Actions Inactive
"github.com/databricks/databricks-sdk-go"
)

type AuthorizedKeysManager struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding a comment documenting the lifecycle of the keys and when they are added

But this time also store private client key in the secrets scope.
@ilia-db ilia-db temporarily deployed to test-trigger-is October 31, 2025 10:18 — with GitHub Actions Inactive
@ilia-db ilia-db temporarily deployed to test-trigger-is November 6, 2025 12:49 — with GitHub Actions Inactive
@ilia-db ilia-db added this pull request to the merge queue Nov 10, 2025
Merged via the queue into main with commit 6d9bfa3 Nov 10, 2025
13 checks passed
@ilia-db ilia-db deleted the ssh-tunnel-integ-tests branch November 10, 2025 12:32
github-merge-queue bot pushed a commit that referenced this pull request Nov 14, 2025
## Changes
Based on #3775

Make a simple connection with "echo 'Hello'" and check that databricks
ssh connect output has it.

SSH tunnel requires linux releases to be present, so it can upload them
to the workspace for the ssh server to use. Added the build step
directly to the acceptance_test, similar to how we build the CLI itself
there. Initially I've modified our runner to install goreleaser and
running make snapshot-release, but it takes over 10m on linux machines.
Current build step only adds ~1m to the test time.
@ilia-db ilia-db changed the title Pass public key name in every client ssh connection Store both public and private client keys in secret scopes Nov 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants