Skip to content

cherry pick fix retry scenario for query to table materialization (#36912) #36916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
damccorm merged 1 commit into from
Nov 26, 2025
+24 −0

fix retry scenario for query to table materialization (#36912)

9a5af9b
Select commit
Loading
Failed to load commit list.
Sign in for the full log view
Merged

cherry pick fix retry scenario for query to table materialization (#36912) #36916

fix retry scenario for query to table materialization (#36912)
9a5af9b
Select commit
Loading
Failed to load commit list.
GitHub Actions / Test Results failed Nov 26, 2025 in 0s

2 fail, 230 skipped, 2 029 pass in 2h 55m 3s

2 261 tests   2 029 ✅  2h 55m 3s ⏱️
  223 suites    230 💤
  223 files        2 ❌

Results for commit 9a5af9b.

Annotations

Check warning on line 0 in org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

testReadFromBigQueryWithExceptionHandling (org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest) failed 

sdks/java/io/google-cloud-platform/build/test-results/test/TEST-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest.xml [took 0s]
Raw output 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: Tried to get a dataset project-id:temp_dataset_beam_bq_job_QUERY_bigqueryiodynamicreadtest0testreadfrombigquerywithexceptionhandlingrunner11262005456ecf446f_0a1b6df9cfa5470781b506b18a13a0bb, but no such dataset was set
	at app//org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:377)
	at app//org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:345)
	at app//org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
	at app//org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
	at app//org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
	at app//org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:442)
	at app//org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:381)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest.testReadFromBigQueryWithExceptionHandling(BigQueryIODynamicReadTest.java:784)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
	at app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at app//org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at app//org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:369)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest$1$1.evaluate(BigQueryIODynamicReadTest.java:174)
	at app//org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
	at app//org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
	at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at app//org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at app//org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at app//org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at app//org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at app//org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at app//org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at app//org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at app//org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:54)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:53)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:92)
	at com.sun.proxy.$Proxy6.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:183)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:132)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:122)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:72)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.io.IOException: Tried to get a dataset project-id:temp_dataset_beam_bq_job_QUERY_bigqueryiodynamicreadtest0testreadfrombigquerywithexceptionhandlingrunner11262005456ecf446f_0a1b6df9cfa5470781b506b18a13a0bb, but no such dataset was set
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.throwNotFound(FakeDatasetService.java:816)
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTableImpl(FakeDatasetService.java:264)
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:252)
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:243)
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:237)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageQuerySource.removeDestinationIfExists(BigQueryStorageQuerySource.java:208)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$DynamicRead$CreateBoundedSourceForTable.processElement(BigQueryIO.java:983)
Caused by: com.google.api.client.http.HttpResponseException
	at app//com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.throwNotFound(FakeDatasetService.java:816)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTableImpl(FakeDatasetService.java:264)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:252)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:243)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:237)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageQuerySource.removeDestinationIfExists(BigQueryStorageQuerySource.java:208)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$DynamicRead$CreateBoundedSourceForTable.processElement(BigQueryIO.java:983)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$DynamicRead$CreateBoundedSourceForTable$DoFnInvoker.invokeProcessElement(Unknown Source)
	at app//org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:214)
	at app//org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:191)
	at app//org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:79)
	at app//org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:244)
	at app//org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:54)
	at app//org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165)
	at app//org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
	at [email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at [email protected]/java.lang.Thread.run(Thread.java:829)

Check warning on line 0 in org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest

See this annotation in the file changed.

@github-actions github-actions / Test Results

testReadQueryFromBigQueryIO (org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest) failed 

sdks/java/io/google-cloud-platform/build/test-results/test/TEST-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest.xml [took 0s]
Raw output 
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: Tried to get a dataset project-id:temp_dataset_beam_bq_job_QUERY_bigqueryiodynamicreadtest0testreadqueryfrombigqueryiorunner112620054610081c61_6b4c331314404405ae6074a50c4b7d7b, but no such dataset was set
	at app//org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:377)
	at app//org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:345)
	at app//org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
	at app//org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
	at app//org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
	at app//org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:442)
	at app//org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:381)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest.testReadQueryFromBigQueryIO(BigQueryIODynamicReadTest.java:755)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
	at app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at app//org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at app//org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at app//org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:369)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryIODynamicReadTest$1$1.evaluate(BigQueryIODynamicReadTest.java:174)
	at app//org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
	at app//org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
	at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at app//org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at app//org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at app//org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at app//org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at app//org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at app//org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at app//org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at app//org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at app//org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:54)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:53)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:92)
	at com.sun.proxy.$Proxy6.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:183)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:132)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:122)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:72)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.io.IOException: Tried to get a dataset project-id:temp_dataset_beam_bq_job_QUERY_bigqueryiodynamicreadtest0testreadqueryfrombigqueryiorunner112620054610081c61_6b4c331314404405ae6074a50c4b7d7b, but no such dataset was set
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.throwNotFound(FakeDatasetService.java:816)
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTableImpl(FakeDatasetService.java:264)
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:252)
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:243)
	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:237)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageQuerySource.removeDestinationIfExists(BigQueryStorageQuerySource.java:208)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$DynamicRead$CreateBoundedSourceForTable.processElement(BigQueryIO.java:983)
Caused by: com.google.api.client.http.HttpResponseException
	at app//com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.throwNotFound(FakeDatasetService.java:816)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTableImpl(FakeDatasetService.java:264)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:252)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:243)
	at app//org.apache.beam.sdk.io.gcp.testing.FakeDatasetService.getTable(FakeDatasetService.java:237)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageQuerySource.removeDestinationIfExists(BigQueryStorageQuerySource.java:208)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$DynamicRead$CreateBoundedSourceForTable.processElement(BigQueryIO.java:983)
	at app//org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$DynamicRead$CreateBoundedSourceForTable$DoFnInvoker.invokeProcessElement(Unknown Source)
	at app//org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:214)
	at app//org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:191)
	at app//org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:79)
	at app//org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:244)
	at app//org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:54)
	at app//org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165)
	at app//org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
	at [email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at [email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at [email protected]/java.lang.Thread.run(Thread.java:829)
View more details on GitHub Actions