Schema mis alignment from Spark DF to BQ table during write

[bixi-hemant]

Hello,
As a part of Version upgrade from 0.23.1 to 0.43.1, I encountered an issue while creating a new table with the connector.

My Spark final DF has schema :

|-- ad_format_ids: array (nullable = true)
| |-- element: integer (containsNull = true)
|-- ad_format_names: array (nullable = true)
| |-- element: string (containsNull = true)

and we are expecting to get schema in our BQ table as like :

This is working fine with 0.23.1, but as soon as I moved to 0.43.1 the schema during BQ write change and its getting flatten like :

We have same config for both V as :

  writer
      .format("bigquery")
      .mode(SaveMode.Overwrite)
      .option("writeMethod", "indirect")
      .option("table", s"$dataset.$table") // db.table_name
      .option("datePartition", partition) // 2026021005
      .option("partitionField", partitionField) // hour_timestamp
      .option("partitionType", partitionType) // HOUR
      .option("allowFieldAddition", allowFieldAddition) // true
      .option("allowFieldRelaxation", allowFieldRelaxation) // true
      .option("clusteredFields", clusteredFields.mkString(","))

NOTE : The intermediate file is not mentioned explicitly so its parquet by default in both versions.
Is the newer version NO more supporting to have the array schema?

[davidrabinowitz]

Add the following option: .option("enableListInference", "false").

[bixi-hemant]

Hello @davidrabinowitz ,
Thank you for the response. By default it's false, Do we need to define this explicitly?

[davidrabinowitz]

You (or someone else) may set it to true somewhere, let's rule that out

[bixi-hemant]

Thanks, I will try to use this one, but it would be strange because with/without this option this is working fine with older version (0.23.1). Is this something changes in newer version?

[davidrabinowitz]

We added that long ago in order to avoid the array->list->element imposed by the intermediate parquet. Also, upgrading such an old version is in many cases problematic. I'd suggest to upgrade at least once every 3 months, and try to keep with the latest version as much as possible

[bixi-hemant]

Hi @davidrabinowitz,
I tried adding this option .option("enableListInference", "false"). , but still having same issue, Now my final options during write is having :

allowfieldrelaxation = 'true'
datepartition = '2026020512'
partitionfield = 'hour_timestamp'
partitiontype = 'HOUR'
enablelistinference = 'false'
clusteredfields = 'id, name, id2, name2'
writemethod = 'indirect'
allowfieldaddition = 'true'
table = 'db.table_name'

Also , i have double check the data inside table and none of the rows has any data :

Is there anything we still missing regarding config?
Thanks!

[davidrabinowitz]

I would need more information regarding your app. Can you please open an issue via the Google Cloud console or your account manager? then we will be able to communicate more securely