[Bug][Go SDK] Native BigQuery IO should require registrations for passed in types. #26298
Description
What happened?
On distributed runners, the type must be registered to ensure it's serialized correctly. This can cause problems for BigQueryIO's "Null" type wrappers, which will lose their type identity going through beam serialization.
The user side solution is to register their type with beam.
func init() {
beam.RegisterType(reflect.TypeOf((*BigQueryEntity)(nil)).Elem())
}
Or to use the register package on a downstream DoFn, which will register input and output types from the DoFn's ProcessElement method.
func init() {
register.DoFn1x1[BigQueryEntity, Outut]((*myDoFn)(nil))
}
However it can be easy to forget to do this. So we can have the BigqueryIO do this at pipeline construction, to make the behavior clear and avoid user issues at pipeline execution time.
By adding to the internal mustInferSchema function, and have it get the key and look up the type with the beam runtime functions [runtime.TypeKey and runtime.LookupType. ](https://github.com/apache/beam/blob/sdks/v2.46.0/sdks/go/pkg/beam/core/runtime/types.go#L50
Note, that due to current limitations on looking up arbitrary types in Go, type registration must happen before beam.Init is called, so we can't simply register types there. We need to fail with a clear error message and instructions.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner