debug migrating to Kokoro instances - pared down PR by menahyouyeah · Pull Request #9991 · GoogleContainerTools/skaffold

menahyouyeah · 2026-02-25T14:32:09Z

Description
debugging Kokoro instance migration

gemini-code-assist · 2026-02-25T14:34:27Z

Summary of Changes

Hello @menahyouyeah, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on debugging and resolving issues encountered during the migration to new Kokoro instances. It addresses build tool incompatibilities by adjusting environment variables and temporarily skips several integration tests that are currently not functional in the new environment, primarily due to differences in Docker daemon behavior or general incompatibility.

Highlights

Jib/Maven/Gradle Configuration Fix: Modified the Kokoro presubmit script to set MAVEN_OPTS and GRADLE_USER_HOME to /tmp. This resolves an issue where Jib would crash because a non-existent Maven settings file (~/.m2/settings.xml) was mistakenly created as a folder by Docker in the clean Kokoro environment.
Conditional Integration Test Skipping: Implemented conditional skips for several integration tests (TestBuildDependenciesCache, TestRunWithDockerAndBuildArgs, TestMultiPlatformWithRun, TestMultiplatformWithDevAndDebug, TestMultiplatformWithDeploy, TestRenderWithGCBRepositoryRemoteDependency). These skips are applied in hybrid environments or due to general incompatibility with the new Kokoro instances and Docker daemon API limitations, allowing the migration debugging to proceed without blocking on these specific failures.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

hack/kokoro/presubmit.sh
- Exported MAVEN_OPTS to set user.home to /tmp.
- Exported GRADLE_USER_HOME to /tmp/.gradle.
- Added comments explaining the rationale for these changes related to Jib and Kokoro's clean environment.
integration/build_dependencies_test.go
- Added a conditional t.Skip statement for TestBuildDependenciesCache when GKE_CLUSTER_NAME is 'presubmit-hybrid', citing issues with image storage in BuildKit cache.
integration/build_test.go
- Added a conditional t.Skip statement for TestRunWithDockerAndBuildArgs when GKE_CLUSTER_NAME is 'presubmit-hybrid', citing issues with image storage in BuildKit cache.
integration/multiplatform_test.go
- Added a conditional t.Skip statement for TestMultiPlatformWithRun when isRunningInHybridCluster is true, citing Docker daemon API limitations during Kokoro migration.
- Added a conditional t.Skip statement for TestMultiplatformWithDevAndDebug when isRunningInHybridCluster is true, citing Docker daemon API limitations during Kokoro migration.
- Added a conditional t.Skip statement for TestMultiplatformWithDeploy when isRunningInHybridCluster is true, citing Docker daemon API limitations during Kokoro migration.
integration/remote_config_dependency_test.go
- Added an unconditional t.Skip statement for TestRenderWithGCBRepositoryRemoteDependency, stating it 'does not work on new Kokoro instances'.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request addresses issues related to migrating to Kokoro instances by preventing Jib crashes, skipping tests in hybrid environments, and skipping tests that don't work on new Kokoro instances. The changes involve modifying shell scripts and Go test files to accommodate the new environment.

gemini-code-assist · 2026-02-25T14:37:06Z

integration/remote_config_dependency_test.go

 )

 func TestRenderWithGCBRepositoryRemoteDependency(t *testing.T) {
+	t.Skip("Skipping these tests - does not work on new Kokoro instances")


It's important to document why these tests are being skipped. Elaborate on the reasons why they don't work on the new Kokoro instances. This will help future developers understand the issue and potentially fix it.

gemini-code-assist · 2026-02-25T14:37:06Z

hack/kokoro/presubmit.sh

+    # Prevent Jib (Maven/Gradle) from crashing on Kokoro.
+    # Kokoro is a "clean" environment and doesn't have a Maven settings file (~/.m2/settings.xml).
+    # When Skaffold tries to sync that non-existent file into a Docker container, Docker 
+    # mistakenly creates a FOLDER named 'settings.xml' instead. Jib then crashes because 
+    # it can't read a folder as a configuration file. 
+    # Pointing home to /tmp avoids this file-vs-folder conflict.


These comments are helpful in explaining the context and the problem being solved. However, consider adding a link to the specific Jib issue or documentation that describes this behavior for better traceability and understanding.

gemini-code-assist · 2026-02-25T14:37:06Z

integration/build_dependencies_test.go

+	// Skip in hybrid environment
+	if os.Getenv("GKE_CLUSTER_NAME") == "presubmit-hybrid" {
+		t.Skip("Skipping test in hybrid environment: docker-container driver stores images in BuildKit cache, not local daemon")


This skip condition is specific to the hybrid environment. It would be beneficial to add a comment explaining why the test is skipped in this environment, referencing the docker-container driver and BuildKit cache issue mentioned in the skip message.

gemini-code-assist · 2026-02-25T14:37:07Z

integration/build_test.go

+	// Skip in hybrid environment
+	if os.Getenv("GKE_CLUSTER_NAME") == "presubmit-hybrid" {
+		t.Skip("Skipping test in hybrid environment: docker-container driver stores images in BuildKit cache, not local daemon")


Similar to the other test skips, adding a comment that elaborates on why this test is skipped in the hybrid environment would improve clarity.

gemini-code-assist · 2026-02-25T14:37:07Z

integration/multiplatform_test.go

+	if isRunningInHybridCluster == hybridClusterName {
+		t.Skip("Skipping hybrid tests during Kokoro migration due to Docker daemon API limitations.")


It's good to skip tests that are known to fail in certain environments. Adding a comment that explains the Docker daemon API limitations would provide more context.

gemini-code-assist · 2026-02-25T14:37:07Z

integration/multiplatform_test.go

+	if isRunningInHybridCluster {
+		t.Skip("Skipping hybrid tests during Kokoro migration due to Docker daemon API limitations.")


Same comment as above, adding a comment that explains the Docker daemon API limitations would provide more context.

gemini-code-assist · 2026-02-25T14:37:07Z

integration/multiplatform_test.go

+	if isRunningInHybridCluster {
+		t.Skip("Skipping hybrid tests during Kokoro migration due to Docker daemon API limitations.")


Same comment as above, adding a comment that explains the Docker daemon API limitations would provide more context.

debug migrating to Kokoro instances - pared down PR

e98055e

menahyouyeah requested a review from a team as a code owner February 25, 2026 14:32

pull-request-size bot added the size/S label Feb 25, 2026

gemini-code-assist bot reviewed Feb 25, 2026

View reviewed changes

menahyouyeah added 2 commits February 25, 2026 14:48

switch to new project in presubmit.sh

c1e9e3d

pared down to just one test

ff8dc03

pull-request-size bot added size/M and removed size/S labels Feb 25, 2026

menahyouyeah added 5 commits February 25, 2026 17:53

tag skaffold-builder image with commit and push to AR

ba0f37a

use buildx with --load

30c6406

jib errors

3333cd9

remove wrong project

ec559e9

jib updates

df24723

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

debug migrating to Kokoro instances - pared down PR#9991

debug migrating to Kokoro instances - pared down PR#9991
menahyouyeah wants to merge 8 commits intoGoogleContainerTools:mainfrom
menahyouyeah:debug-kokoro

menahyouyeah commented Feb 25, 2026

Uh oh!

gemini-code-assist bot commented Feb 25, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Feb 25, 2026

Uh oh!

gemini-code-assist bot Feb 25, 2026

Uh oh!

gemini-code-assist bot Feb 25, 2026

Uh oh!

gemini-code-assist bot Feb 25, 2026

Uh oh!

gemini-code-assist bot Feb 25, 2026

Uh oh!

gemini-code-assist bot Feb 25, 2026

Uh oh!

gemini-code-assist bot Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		if isRunningInHybridCluster == hybridClusterName {
		t.Skip("Skipping hybrid tests during Kokoro migration due to Docker daemon API limitations.")

		if isRunningInHybridCluster {
		t.Skip("Skipping hybrid tests during Kokoro migration due to Docker daemon API limitations.")

Conversation

menahyouyeah commented Feb 25, 2026

Uh oh!

gemini-code-assist bot commented Feb 25, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant