[Test] fix(bigquery): handle field named "f" in tableRowFromMessage #36397

liferoad · 2025-10-06T01:38:30Z

Use setF method when field name is "f" to avoid IllegalArgumentException with internal fields. Add test case to verify the fix.

Addresses #33531

Stack Trace:

Error message from worker: java.lang.IllegalArgumentException: Can not set
java.util.List field com.google.api.services.bigquery.model.TableRow.f to
java.lang.Double

java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)

java.base/jdk.internal.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)

java.base/jdk.internal.reflect.UnsafeObjectFieldAccessorImpl.set(UnsafeObjectFieldAccessorImpl.java:81)

java.base/java.lang.reflect.Field.set(Field.java:780)

com.google.api.client.util.FieldInfo.setFieldValue(FieldInfo.java:281)

com.google.api.client.util.FieldInfo.setValue(FieldInfo.java:237)

com.google.api.client.util.GenericData.put(GenericData.java:98)

org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.tableRowFromMessage(TableRowToStorageApiProto.java:1115)

org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.jsonValueFromMessageValue(TableRowToStorageApiProto.java:1140)

org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.tableRowFromMessage(TableRowToStorageApiProto.java:1117)

org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.jsonValueFromMessageValue(TableRowToStorageApiProto.java:1140)

org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.tableRowFromMessage(TableRowToStorageApiProto.java:1117)

org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.tableRowFromMessage(TableRowToStorageApiProto.java:1098)

org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn$DestinationState.flush(StorageApiWriteUnshardedRecords.java:655)

org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn.flushAll(StorageApiWriteUnshardedRecords.java:1042)

org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords$WriteRecordsDoFn.finishBundle(StorageApiWriteUnshardedRecords.java:1207)

Tested:

./gradlew :sdks:java:io:google-cloud-platform:integrationTest --tests "*BigQueryNestedFFieldIT*" --info

./gradlew :sdks:java:io:google-cloud-platform:test --tests "org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest" --info

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Use setF method when field name is "f" to avoid IllegalArgumentException with internal fields. Add test case to verify the fix.

gemini-code-assist · 2025-10-06T01:38:43Z

Summary of Changes

Hello @liferoad, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a specific issue within the BigQuery I/O connector where a field named "f" in a TableRow could cause an IllegalArgumentException during conversion from a Protobuf message. The fix ensures that such fields are correctly processed by utilizing the setF method, preventing data conversion errors and improving the robustness of BigQuery integration. A dedicated test case has been added to verify the proper functionality.

Highlights

Fix for IllegalArgumentException: Addressed an issue where using a field named "f" in BigQuery TableRow conversion could lead to an IllegalArgumentException due to conflicts with internal fields.
Special Handling for 'f' Field: The tableRowFromMessage method now explicitly checks for fields named "f" and uses the setF method of TableRow to correctly handle its value, wrapping scalar values in TableCell lists as required by the BigQuery API.
New Test Case: A new unit test testTableRowFromMessageWithFieldNamedF has been added to validate the fix and ensure proper conversion of fields named "f" from DynamicMessage to TableRow.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2025-10-06T03:42:04Z

Assigning reviewers:

R: @kennknowles for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Abacn

we probably cannot use "setF" unless we know the order of fields at this moment. We either need to use setF for all field values, or wrap field name "f" to something like "__f" that workaround with TableRow limit (since this method is used for DLQ, it might be fine to change field name here)

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java

Add integration test verifying BigQuery Storage API write with nested structures containing 'f' field. Tests fix for IllegalArgumentException when setting List field to Double in nested TableRow structures.

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java

Add PAssert to validate failed Storage API inserts in BigQueryNestedFFieldIT test Modify test to generate multiple test cases with different payload sizes Remove redundant test case for multiple nested structures

...cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryNestedFFieldIT.java

liferoad · 2025-10-07T18:17:16Z

R: @baeminbo

github-actions · 2025-10-07T18:18:26Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

Consolidate field processing in TableRowToStorageApiProto to always use setF with TableCells. This removes special handling for field 'f' and ensures consistent field ordering based on descriptor

Introduce a new flag to enable enhanced table row conversion in BigQuery Storage API writes. This provides better handling of nested fields and complex data types, particularly for cases involving fields named 'f'. The enhanced conversion avoids special handling of 'f' fields that could cause conflicts in nested structures. The change maintains backward compatibility by defaulting to the legacy behavior. When enabled, the new conversion logic processes fields in descriptor order and uses the F list format consistently. This addresses issues with nested structures containing 'f' fields while preserving existing functionality for other cases. Added integration tests to verify the enhanced conversion behavior with nested 'f' fields and backward compatibility tests to ensure existing pipelines continue to work as expected.

Abacn · 2025-10-08T18:36:10Z

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

+     *     behavior
+     * @return the updated Write transform
+     */
+    public Write<T> withEnhancedTableRowConversion(boolean useEnhancedTableRowConversion) {


users on new Beam version could still trigger the bug if not opt-in this PTransform configuration. Also, this is only used in DLQ in storage_write_api write methods. Personally prefer over #36425 in terms of fix, and the new tests added in this PR is valuable and could check in after fix merged

Sounds good. Thanks!

liferoad · 2025-10-10T12:41:59Z

#36425 continues the work.

fix(bigquery): handle field named "f" in TableRow conversion

7be50bf

Use setF method when field name is "f" to avoid IllegalArgumentException with internal fields. Add test case to verify the fix.

liferoad requested a review from reuvenlax October 6, 2025 01:38

github-actions bot added java io gcp labels Oct 6, 2025

liferoad requested a review from Abacn October 6, 2025 01:38

github-actions bot added the Next Action: Reviewers label Oct 6, 2025

Abacn reviewed Oct 6, 2025

View reviewed changes

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java Outdated Show resolved Hide resolved

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java Outdated Show resolved Hide resolved

liferoad added 2 commits October 7, 2025 09:38

Merge branch 'master' into fix-tableRowFromMessage-setF

5eddc3c

test(bigquery): add integration test for nested 'f' field handling

224294c

Add integration test verifying BigQuery Storage API write with nested structures containing 'f' field. Tests fix for IllegalArgumentException when setting List field to Double in nested TableRow structures.

reuvenlax reviewed Oct 7, 2025

View reviewed changes

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java Show resolved Hide resolved

test(BigQueryNestedFFieldIT): add validation for failed BigQuery inserts

310ad4e

Add PAssert to validate failed Storage API inserts in BigQueryNestedFFieldIT test Modify test to generate multiple test cases with different payload sizes Remove redundant test case for multiple nested structures

liferoad commented Oct 7, 2025

View reviewed changes

...cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryNestedFFieldIT.java Outdated Show resolved Hide resolved

liferoad added 4 commits October 7, 2025 14:27

spotless

cb69c74

Merge branch 'master' into fix-tableRowFromMessage-setF

d1bef95

fixed the test

5535a9f

refactor(bigquery): simplify table row conversion logic in Storage API

6cf8394

Consolidate field processing in TableRowToStorageApiProto to always use setF with TableCells. This removes special handling for field 'f' and ensures consistent field ordering based on descriptor

liferoad requested review from Abacn and reuvenlax October 7, 2025 23:21

liferoad changed the title ~~fix(bigquery): handle field named "f" in tableRowFromMessage~~ [Test] fix(bigquery): handle field named "f" in tableRowFromMessage Oct 8, 2025

fixed tests

a922d3c

Abacn reviewed Oct 8, 2025

View reviewed changes

liferoad closed this Oct 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Test] fix(bigquery): handle field named "f" in tableRowFromMessage #36397

[Test] fix(bigquery): handle field named "f" in tableRowFromMessage #36397

Uh oh!

liferoad commented Oct 6, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Oct 6, 2025

Uh oh!

github-actions bot commented Oct 6, 2025

Uh oh!

Abacn left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liferoad commented Oct 7, 2025

Uh oh!

github-actions bot commented Oct 7, 2025

Uh oh!

Abacn Oct 8, 2025

Uh oh!

liferoad Oct 8, 2025

Uh oh!

liferoad commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Test] fix(bigquery): handle field named "f" in tableRowFromMessage #36397

[Test] fix(bigquery): handle field named "f" in tableRowFromMessage #36397

Uh oh!

Conversation

liferoad commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Oct 6, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions bot commented Oct 6, 2025

Uh oh!

Abacn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liferoad commented Oct 7, 2025

Uh oh!

github-actions bot commented Oct 7, 2025

Uh oh!

Abacn Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

liferoad Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

liferoad commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

liferoad commented Oct 6, 2025 •

edited

Loading